www.Tobold.com
Tobold@GMail.com
A blog with my thoughts regarding games I am playing and other stuff in life. Please read my Terms of Service
RSS Feed
Thursday, September 06, 2007
What are beta tests for?
Bloggers read other people's blogs. And unsurprisingly, subjects that are discussed one one blog thus often jump to the next blog. Although this custom is described as "circle jerking", I think it isn't all bad. As long as every blogger also makes some posts with his own ideas, and the same subjects don't circle forever until they are stale, crosslinking to other blogs creates the fabled "blogosphere". This weeks subject in the MMO blogosphere is beta tests, as discussed at Kill Ten Rats, based on a rant at Wife Aggro.
Pvthudson from Wife Aggro thinks that people aren't playing betas right, and should stop complaining about what's wrong with the game on that game's beta forums. He says: "The beta is not for you to form a guild, get all your friends in, camp mobs, farm money and then complain that the game isn’t complete yet." Well, I'd say that if you would form a guild, get all your friends in, camp mobs and farm money in the real game, then this is exactly what you should do in the beta as well. It is good if some beta testers try out exotic stuff to find obscure bugs. But if the large majority of beta testers plays the beta exactly as if it was the real thing, that is very helpful for the developers too.
Finding bugs and exploits and reporting them can be very rewarding. In one of the betas I'm currently in I found an exploit, reported it, and saw it fixed in the patch notes a week later. I also had non-canned "thank you for finding this bug, good catch" e-mails from devs. Things like this really make you feel you are contributing to the game improving towards release.
But people just playing and then complaining that "the combat system sucks" on the beta forums are important too. Age of Conan was delayed by half a year based on exactly that sort of comment. Whether a feature is too complicated, working well, or just plain boring is not something you're likely to find in a bug report. Even a heated flame war on the forums with some people hating feature X and other defending it can give ideas to developers on how to improve it. Because if the flame war doesn't happen on the beta forums, it will sure break out later on the real forums, and then it's much harder to change things.
Open betas, or the kind of betas you can get into by just having a Fileplanet subscription, aren't even intended for everybody to write bug reports. If everybody did that, the devs would get thousands of bug reports on the same bug, with a huge workload to sift through all of them. Open betas are a joint project of the hardware guys and the marketing department, the guys developing game mechanics do barely figure at that point. "Open beta" just sounds more attractive than "free trial", because everybody is aware that it won't last, and you better get in before the release. So the hardware guys get their stress test, and the marketing guys get their free advertising. I think the way betas are going now is working as intended, there is no need to admonish players that they are doing something wrong in the betas.
The only advice I would give game companies is to keep the closed beta phase restricted to a small number of players, and to keep the open beta phase short. One of the most frequent problems of new MMORPGs is that due to having to release the game on a schedule and not having unlimited amounts of cash, games often are released with a lack of content. The last thing you want is a large number of beta players already playing through all of your content and being bored of your game before it is even released. You want to give the players of the open beta just enough of a hook to make them buy the game, not enough time to finish it for free.
- posted by Tobold @ 8:31 AM Permanent Link
Comments:
<< Home
Newer› ‹Older
Any developer that thinks beta testers can be trusted is delusional. In fact, I'm sure that none actually do.
Too many beta testers want a head start and so they horde info - including exploits.
And why would a potential gold farmer reveal secrets to making easy cash. It's inevitable that money making exploits are rarely reported.
Sad but true. Far too many mmorpg players are scum.
Too many beta testers want a head start and so they horde info - including exploits.
And why would a potential gold farmer reveal secrets to making easy cash. It's inevitable that money making exploits are rarely reported.
Sad but true. Far too many mmorpg players are scum.
# posted by 
: 6/9/07 09:46
: 6/9/07 09:46
http://ve3d.ign.com/articles/news/34146/Tabula-Rasa-Launch-Date-October-19
Speaking of beta test, Tabula Rasa is about to have it's NDA lifted.
Well, this won't be pretty )
Speaking of beta test, Tabula Rasa is about to have it's NDA lifted.
Well, this won't be pretty )
# posted by : 6/9/07 09:54
: 6/9/07 09:54
'Far too many mmorpg players are scum'
Well I'd personally prefer to hang out with scum than over-opinionated bores who have very little to add to a discussion despite their pretensions :)
Necra
Well I'd personally prefer to hang out with scum than over-opinionated bores who have very little to add to a discussion despite their pretensions :)
Necra
# posted by : 6/9/07 10:46
: 6/9/07 10:46
And you manage to fall into both categories - congratulations )
Why can't mmorpg developers hire professional beta testers, just like the rest of the industry does.
Why can't mmorpg developers hire professional beta testers, just like the rest of the industry does.
# posted by : 6/9/07 11:57
: 6/9/07 11:57
Anyone seeing the irony of me saying that even flame wars can be useful, and two people promptly starting one in the comment section? :)
I do think that game companies hire some professional testers for finding bugs. Another reason why it isn't the end of the world if the people in open beta aren't busily searching and reporting bugs. They've just been invited for the numbers, for stress testing (and marketing).
I do think that game companies hire some professional testers for finding bugs. Another reason why it isn't the end of the world if the people in open beta aren't busily searching and reporting bugs. They've just been invited for the numbers, for stress testing (and marketing).
don;t forget just for gameplay :)
professional testers are 'testers'
beta-players are doing actually things they find fun. if they never 'touch' certain designs, objects, quests or arena's after the initial week they know it's a stupid design altho inhouse they thought it was great!
professional testers are 'testers'
beta-players are doing actually things they find fun. if they never 'touch' certain designs, objects, quests or arena's after the initial week they know it's a stupid design altho inhouse they thought it was great!
In public beta testing, there are still professional testers playing the game.
In every beta I've been a part of the chain's been like this.
1.) Build made and looked at by dev team.
2.) Build sent to QA (Quality Assurance) for further testing.
3.) If QA says it's good to go, it then proceeds to public beta testers for further MASS testing.
Number 3 is what we are, the gaming public. We're there to play the game, give our feedback on the systems, and find what bugs we can. In public beta phases, most developers will TELL their testers to just play the game and be sure to report their feedback.
I'm not saying there's not a problem with people who think it's just free game time, never offering their input. But this problem is often created by the fact that we're asked to test it in the same way that we'd actually play the game.
I'm in one beta right now, where bugs are not so glaring and so most people take to testing the PvP system or crafting and report on that. And you know what? It's our findings combined with dev feedback that are shaping the game.
That right there, dev feedback, is essential in any beta. There are several betas I'm lucky enough to be a part of. I'll give 3 examples.
1 has decent developer communication. 1 has outstanding dev communication. 1 has little or none. The one that has little or no feedback from the devs is the one that is in the worst shape. People all over the boards are trying to tell the company to push the game back, that it's slowly become less and less fun with each patch... and yet it's still forging onward towards release.
Do I have a point? Not really a concrete one. I'm just giving some examples from someone who's been in a decent amount of betas.
/endramble
In every beta I've been a part of the chain's been like this.
1.) Build made and looked at by dev team.
2.) Build sent to QA (Quality Assurance) for further testing.
3.) If QA says it's good to go, it then proceeds to public beta testers for further MASS testing.
Number 3 is what we are, the gaming public. We're there to play the game, give our feedback on the systems, and find what bugs we can. In public beta phases, most developers will TELL their testers to just play the game and be sure to report their feedback.
I'm not saying there's not a problem with people who think it's just free game time, never offering their input. But this problem is often created by the fact that we're asked to test it in the same way that we'd actually play the game.
I'm in one beta right now, where bugs are not so glaring and so most people take to testing the PvP system or crafting and report on that. And you know what? It's our findings combined with dev feedback that are shaping the game.
That right there, dev feedback, is essential in any beta. There are several betas I'm lucky enough to be a part of. I'll give 3 examples.
1 has decent developer communication. 1 has outstanding dev communication. 1 has little or none. The one that has little or no feedback from the devs is the one that is in the worst shape. People all over the boards are trying to tell the company to push the game back, that it's slowly become less and less fun with each patch... and yet it's still forging onward towards release.
Do I have a point? Not really a concrete one. I'm just giving some examples from someone who's been in a decent amount of betas.
/endramble
I've been in a number of closed betas and the one of the main problems is a lack of coordination by the devs.
If the devs want something to be tested, they should direct the testers towards it. This rarely happens.
Another problem is the phased nature of most betas. They take in a bunch of new testers who start at the bottom and go through the same content that has been tested to death already.
But the most important prolem is a lack of communication. In order to properly test AND give useful feedback, testers need to know about which bugs are already known and slated to be fixed, which parts of the game are due to be reworked or changed etc.
None of the betas I've participated in had anything resembling a bug tracker for testers. Or at least a maintained list of known bugs and planned changes.
That's why you have testers reporting the same shit over and over again and testers being generally clueless about how to effectively test the things that need to be tested.
If MMO devs want people to test the game instead of using beta as a free preview they should start to treat them like testers. Throwing them into the game world and telling them to give uncoordinated feedback in a forum isn't enough.
If the devs want something to be tested, they should direct the testers towards it. This rarely happens.
Another problem is the phased nature of most betas. They take in a bunch of new testers who start at the bottom and go through the same content that has been tested to death already.
But the most important prolem is a lack of communication. In order to properly test AND give useful feedback, testers need to know about which bugs are already known and slated to be fixed, which parts of the game are due to be reworked or changed etc.
None of the betas I've participated in had anything resembling a bug tracker for testers. Or at least a maintained list of known bugs and planned changes.
That's why you have testers reporting the same shit over and over again and testers being generally clueless about how to effectively test the things that need to be tested.
If MMO devs want people to test the game instead of using beta as a free preview they should start to treat them like testers. Throwing them into the game world and telling them to give uncoordinated feedback in a forum isn't enough.
# posted by : 6/9/07 16:59
: 6/9/07 16:59
The only advice I would give game companies is to keep the closed beta phase restricted to a small number of players, and to keep the open beta phase short
Several games have been a disaster because they announced the release date they could not back out of for distribution reasons and then had their open beta phase only to find the game was not up to handling the larger numbers than had with “small number” closed beta. This was the problem of the notorious Anarchy Online release. A short open beta pahse is fine long as is not the week before release. Do it a single week or weekend 2 months from release. You risk showing a bugger product to the masses but help insure a much better release. This is another example where WoW got it right.
Several games have been a disaster because they announced the release date they could not back out of for distribution reasons and then had their open beta phase only to find the game was not up to handling the larger numbers than had with “small number” closed beta. This was the problem of the notorious Anarchy Online release. A short open beta pahse is fine long as is not the week before release. Do it a single week or weekend 2 months from release. You risk showing a bugger product to the masses but help insure a much better release. This is another example where WoW got it right.
# posted by : 6/9/07 17:48
: 6/9/07 17:48
I'm with krankykraut. The many closed betas I have been in are terrible about letting us know what is needed.
I've generally been in towards the end of the closed beta but there's never a known bug list or a "we're working on this" list so it's discouraging submitting bugs I am certain have been submitted 50 times already (since I'm seeing the low level content that everyone else has done).
I'm okay with just being another warm body to add to server stress, but I'd love to actually help test and report bugs -- it's just hard to do that, even in most closed betas.
I've generally been in towards the end of the closed beta but there's never a known bug list or a "we're working on this" list so it's discouraging submitting bugs I am certain have been submitted 50 times already (since I'm seeing the low level content that everyone else has done).
I'm okay with just being another warm body to add to server stress, but I'd love to actually help test and report bugs -- it's just hard to do that, even in most closed betas.
I've said it before, some of the best Beta Tests I have been in were the finitely controlled tests where every player who logged on that night knew exactly where and what was being looked at and needed feedback.
I had a miserable time in Vanguard because the game was huge and I had no idea where to start to help. How can you not feel that the developers care little about your feedback?
What i think pvthudson was saying though, was that the closed betas are beginning to look more like the open betas. They may in fact be playing areas of the game that should be looked at, but if they are spending so much time bitching in /ooc, do you think they are doing the same in /bug?
I had a miserable time in Vanguard because the game was huge and I had no idea where to start to help. How can you not feel that the developers care little about your feedback?
What i think pvthudson was saying though, was that the closed betas are beginning to look more like the open betas. They may in fact be playing areas of the game that should be looked at, but if they are spending so much time bitching in /ooc, do you think they are doing the same in /bug?
Actually, if you reread his post, his complaint isn't that they are complaining that it's not working in the official forums. It's that they are complaining it's not working in /ooc or /general (or whatever). To quote his post:
"Guess what people, there is a HUGE thread on the beta forums from CuppaJo stating what is still broken, what will be looked at, and what will be added to the beta in the upcoming weeks."
The problem is (to PvtHudson) that people _aren't_ reading the forums and complaining about things which are either on the "to be fixed" list or obviously not complete, such as quests unable to be completed because they are at a placeholder.
His complaint seems to be not people who are playing the beta like they would be playing a live game, but people who are playing the beta and bitching because it _is_ a beta. That type of bitching would be fine and expected if the game was live, but it's not. Complaining because a game isn't complete when it's in beta is stupid and painful and I agree with PvtHudson on that aspect.
==========
Another thing that gets me is people judging the value of a game based upon Closed Beta reports. This is related to PvtHudson's post, in so much as people treating an incomplete game like it's the final product.
I've seen posts by people on unofficial forums with indirect reports on how "Warhammer Online is bad, and it's *such and such* way." They haven't even introduced every class, but people are already claiming it's not worth buying.
Another example was a friend of mine was offered a slot in the LotRO Alpha and after she played it for 2 hours, she deemed the game as bad and not worth buying. Alpha... and she expected it to give a final opinion of the game's total value. I wanted to bang my head against a wall.
Don't get me wrong. It's fine to play a game in beta to try it out, but you need to realize that anything short of an Open Beta/Free Trial period is not necessarily a good indicator of the game's final form.
Post a Comment
"Guess what people, there is a HUGE thread on the beta forums from CuppaJo stating what is still broken, what will be looked at, and what will be added to the beta in the upcoming weeks."
The problem is (to PvtHudson) that people _aren't_ reading the forums and complaining about things which are either on the "to be fixed" list or obviously not complete, such as quests unable to be completed because they are at a placeholder.
His complaint seems to be not people who are playing the beta like they would be playing a live game, but people who are playing the beta and bitching because it _is_ a beta. That type of bitching would be fine and expected if the game was live, but it's not. Complaining because a game isn't complete when it's in beta is stupid and painful and I agree with PvtHudson on that aspect.
==========
Another thing that gets me is people judging the value of a game based upon Closed Beta reports. This is related to PvtHudson's post, in so much as people treating an incomplete game like it's the final product.
I've seen posts by people on unofficial forums with indirect reports on how "Warhammer Online is bad, and it's *such and such* way." They haven't even introduced every class, but people are already claiming it's not worth buying.
Another example was a friend of mine was offered a slot in the LotRO Alpha and after she played it for 2 hours, she deemed the game as bad and not worth buying. Alpha... and she expected it to give a final opinion of the game's total value. I wanted to bang my head against a wall.
Don't get me wrong. It's fine to play a game in beta to try it out, but you need to realize that anything short of an Open Beta/Free Trial period is not necessarily a good indicator of the game's final form.
<< Home
