Tobold's Blog
Friday, December 07, 2007
 
The many meanings of beta

Recently Richard Garriot revived one of the dead horses of MMORPG discussion by stating that it was a mistake to invite so many people to the Tabula Rasa beta. Cameron from Random Battle has an excellect article on beta testing up. To a certain extent I do agree with him: *If* MMORPGs would have the option for a free trial from day 1 of the release, beta tests could be a lot smaller and shorter.

There are several games I never bought because I disliked them in the beta, Vanguard, Tabula Rasa, and Hellgate London for example just for this year. And while I do like Pirates of the Burning Sea, I'm currently not participating in the open beta, because I already played too much of it in the closed beta, and rather wait for the release version instead of experiencing another character wipe. And I only played the PotBS beta since August, other players were in that beta for even longer, I heard the first beta test started in 2005. In my opinion beta tests should be a lot shorter, otherwise you risk that the beta testers are burned out before the game is even released. And if they are burned out when the NDA is dropped, that has a negative effect on the reviews as well.

On the other hand stress test weekends are a bit too short to give players a good impression of a game. Often the very fact that it is a stress test results in lots of server problems, which aren't necessarily there in the real game. And for more complicated game features a weekend isn't sufficient to experience them. For example on the PotBS stress test weekends, the economy features couldn't really be explored, because it takes days to build something up.

The PotBS devs are a pleasure to read on the official forums, because they are so disarmingly honest. About the open PotBS beta (STARTS TODAY!) they quite honestly said that it was more of a marketing device than for quality control and testing. Games having an open beta of two to four weeks just before release can serve both for marketing and for still finding some bugs. It is not because not every beta tester reports every bug that a beta test is useless. Some people take beta testing more serious than others. I must admit that I was very pleased when I reported a possible exploit in the PotBS beta which then ended up gettting fixed in the next patch, or when I got reply mails from the PotBS devs saying "good catch". But even somebody not reporting a single bug can serve as tester, because the simple fact that he is playing causes an effect on the game. Some exploits are caught not because the people who find them report them, but because players observe other players using them and complain. Or if a player manages to "break" the economy, that should become pretty obvious without anyone having to write a bug report. I would even go so far as to say that if all beta testers were just looking for bugs instead of trying to play the game, many faults would be missed. So I would have to say that it is more the length of a beta that I see as problematic than the fact that there is an open beta in which not everybody is an eager bug reporter.

But the bad press of Vanguard, Tabula Rasa, Hellgate London, and to some extent even Pirates of the Burning Sea, is also due to the game companies releasing games when they are just about playable. I read lots of reports that Vanguard is getting better, that Tabula Rasa improved a lot from beta to release, that Hellgate London is slowly adding things that would actually make it interesting to subscribe, or read about all the wonderful things the PotBS devs want to add after release. But sorry, you can't release an unfinished game and then ask of all the game reviewers to not write about it until a year later when the game is better. If you ask your paying customers to pay for a game, you must live with the reviews you get for the version you released. If you need better reviews, do it like Warhammer Online, cancel your beta and delay the game. You only get one chance to make a good first impression.
Comments:
What is this "Hellfire: London" you speak of?

You must have an alt in Hellfire Peninsula right now, eh?

:)

But yes, I agree on all fronts. Be very picky with beta testers, use your apps to weed out the "players" from the testers, and use focus testing throughout the beta and keep beta to a max of 9 months I say from beginning to end. Not 2+ years.

I think Warhammer has the right idea with it's closing, re-opening, and soon to be more focused testing stages.
 
An open beta 2 weeks before launch has nothing to do with testing or finding bugs. When has an mmo been delayed due to issues found during it's open beta? None.
 
What is this "Hellfire: London" you speak of?

Fixed that to Hellgate: London, thanks for pointing it out.

An open beta 2 weeks before launch has nothing to do with testing or finding bugs. When has an mmo been delayed due to issues found during it's open beta? None.

Why would it have to be delayed? Getting the fix in two to four weeks earlier would already be good.
 
Reminds me of when I went on the WoW PTR to see what it was like for myself.
I created a character, logged in, appeared in Ghostlands, and immediately got ganked by half a dozen Horde players. So much for the PTR.
 
Regarding open beta as a marketing event - it worked for me to get me to play WoW. My friend had an open beta account, let me play for a few hours, and that hooked me into buying WoW the first week of release.

However, the bigger problem is that these other MMO's are releasing games before they're ready. Tabula Rasa is the latest example. The game in open beta form wasn't a good game if you're comparing it to a finished product. I was shocked when I heard that they were releasing it in the state that it was in. Basically, I found the general gameplay just wasn't all that much fun - too much of a grind when you get to level 10 or so. Was I looking for bugs? Sure. Was I willing to assume that the "real game" wouldn't be as laggy as it was during stress testing? Yes. Did I assume that they'd improve crafting? Yes. But there were too many issues that were still present that probably won't get fixed or adjusted until months after launch. And why should I buy a game that's not ready?

Actually, that'd be a good future column, IMHO. At what point in time would you be willing to give an MMO a 2nd chance? It seems like a lot of MMO's aren't so good initially, but get "improved" 6-12 months later. But since a lot of us have already made up our minds off of the initial impression, is there a way to get us to change our mind? Plus, since you're "behind" the initial rush, won't that affect your ability to make groups?

Honestly, it'd probably be better for games to give 2 week free trials. Eve-Online sucked me in for a couple of months because of that. Some of the micro-transaction games are getting me to try them for the same reason.
 
...that should read:

"Honestly, it'd probably be better for games to give 2 week free trials instead of turning Open Beta into a marketing event."
 
I don't think length of beta matter at all, what matters is releasing crappy unfinished games. WoW was polished and wonderful in open beta, it didn't matter how long it was or if they even had one. Vanguard would still have sucked no matter how long or short the beta was, etc.

Really sad thing about Pirates is that the open beta (which I just joined) let me see how utterly unplayable the game is due to the molasses slow mouselook and no slider to adjust it. That it is in beta and they haven't fixed this glaring error tells me all I need to know about how clueless they are. Kinda sad, since the game sounds cool.
 
TR got really quite good reviews - it's their *sales* that were hurt by their beta strategy, and that's what RG was talking about.
 
i am seriously amazed every time with an open-beta.
i know for sure Wow-open beta wasn't smooth at all. server-lag et all, but comparing to all the open-beta's i have been in it was just heaven.
but i'm shocked how not finished the games are/were.
even PotBS doesn't feel finished in any way.
 
Post a Comment

<< Home
Newer›  ‹Older

  Powered by Blogger   Free Page Rank Tool