www.Tobold.com
Tobold@GMail.com
A blog with my thoughts regarding games I am playing and other stuff in life. Please read my Terms of Service
RSS Feed
Wednesday, November 05, 2008
You can't test everything in a beta
The beta for Wrath of the Lich King just ended, and I consciously didn't play it all that much, wanting to keep the content for the release version. I mainly leveled a Death Knight from 55 to 58, the main conclusion of which exercise was that it was a great experience until level 57, and not so great to do old Azeroth and Hellfire Peninsula again. And I looked at various professions, mainly inscription and alchemy. I had copied my alchemist warrior to the beta server, and was delighted to find that alchemy gets a new recipe for "research", with a 7-day cooldown. I find this is a great way to learn recipes, much better than having to repeatedly do some dungeon and hope for a 1% rare drop.
But as I didn't do a single Northrend quest, I was still level 70 at the end of the beta, and getting the herbs together for that research recipe wasn't all that easy. I first had to gather Goldclover in the Northrend starting zones until I made skill 420, and then go to zones with mobs level up to 76 to gather the other herbs. And that was where I realized a big difference between a beta and a live version: In the live version I will probably be able to buy those herbs on the AH. Overpriced, for sure, but they will be there. And as I plan to level my priest first, that might be the better option in the live version, except for the Goldclover. On the beta servers there were no herbs on the AH. Basically there was *nothing* on the AH. The player economy didn't exist on the beta servers.
The thing is that players behave differently on a beta server than on a real server, because their motivations are different. Why care about gold on a character who is going to be deleted at the end of the beta? There are various motivations to play in a beta: Wanting to help testing, seeing new stuff before everyone else, preparing for the release version, trying things out with less consequences, having fun. But by definition the long-term goals players often pursue on real servers aren't part of the equation on beta servers. Some fundamental parts of the game on the real servers just aren't important on the beta servers.
Other people already remarked that Warhammer Online was a different game in beta than live. People wanting to experience all of it, and testing the new things, lead to far more players doing public quests or open world RvR. Once the game went live, habits changed, because for the long-term goal of reaching the level cap fastest, grinding scenarios was a better strategy. One could say that without the carrot of the endgame dangling in front of them, players did what was most fun for them. Once the stupid idea of "the real game is the endgame" grabbed them, they optimized the fun out of it, reducing a huge game to a very small number of scenarios.
All this has consequences for beta testing itself. If people on a beta server behave differently than on the live servers, then what the devs can learn from the beta test is necessarily limited. Even if Mythic had added an auction house to the beta earlier (it was only put in last minute), not all that many people would have used it during beta, so we probably still would have had the current sub-par system. In the beta the balance between PvE and PvP, instances and open world, looked fine, it was only after release where it broke down. And Blizzard didn't notice anything wrong with retribution paladins on the beta servers, where very little PvP was played, only to have to nerf them "to the ground" repeatedly after patch 3.0.2.
So, are beta tests useless? Certainly not. They may not be a good indicator of player behavior on the live servers, but with beta testers trying all sorts of crazy stuff they are more likely to find obscure bugs. And for things like stress tests and server stability tests you simply need a large number of beta testers, using just a small team of in-house testers won't suffice. You just need to be aware that a beta test can't test everything, because in a beta the goals of players are different.
- posted by Tobold @ 11:24 AM Permanent Link
Comments:
<< Home
Newer› ‹Older
This problem has bitten Blizzard in the tail end repeatedly; consider the initial version of the Brewfest Dark Iron attacks, or the rollout of the honor as currency system in patch 2.0 (followed by a 33% nerf to honor gain mere days later).
The only thing a dev might be able to do about this that isn't happening these days is to have a persistent test server, where your characters will not be wiped between testing cycles. Still, that's not without its flaws either - such a server will quickly get populated with characters loaded down with exploited gear/levels, which will skew testing in a different direction.
The only thing a dev might be able to do about this that isn't happening these days is to have a persistent test server, where your characters will not be wiped between testing cycles. Still, that's not without its flaws either - such a server will quickly get populated with characters loaded down with exploited gear/levels, which will skew testing in a different direction.
I am so done with beta's. I really think it ruins the game for the player.
We are reporting on issues, looking for issues, that as we go into launch, we are still aware of those issues.
I am surprised at how many of the issues still exists after the launch of any MMO also, and I hate the whole "An MMO needs 6 months" to work out it's kinks.
Well, cool. I will wait 6 months to buy it also.
But, for beta, I refuse, as I want to be able to experience the game as a whole...and I might even start avoiding forums also (forums contributed to a hate program for AoC as an example, and now look at WAR...yuck....)
We are reporting on issues, looking for issues, that as we go into launch, we are still aware of those issues.
I am surprised at how many of the issues still exists after the launch of any MMO also, and I hate the whole "An MMO needs 6 months" to work out it's kinks.
Well, cool. I will wait 6 months to buy it also.
But, for beta, I refuse, as I want to be able to experience the game as a whole...and I might even start avoiding forums also (forums contributed to a hate program for AoC as an example, and now look at WAR...yuck....)
I think that there's another level, here.
WotLK is an expansion to a game whereas WAR is a standalone game. The beta for WotLK means that you have another WoW character (or a "copy" of one) on the live realms. Because of this, I believe that the effect of betas is much stronger. You don't want to earn money or grind reputation or raid the same dungeon over and over again.
However, in WAR, the beta was the only place for your WAR character. So, if you got into it early enough, it felt like a "real" character. You always knew that the character would be deleted eventually, but for a significant amount of time it was yours.
So, in WAR, I felt like I was playing the game as I would normally. In WotLK, if I had been invited to the beta, I would have done some very superficial things, then stopped.
It seems that betas of expansions kill all but short-term goals and betas of new games (if long enough) only kill off long-term goals, leaving short- and medium-term goals intact. Or, if it's not even about goals, it could be that you feel that you own a character in a long beta for a standalone game and you never feel that ownership in the beta for an expansion.
WotLK is an expansion to a game whereas WAR is a standalone game. The beta for WotLK means that you have another WoW character (or a "copy" of one) on the live realms. Because of this, I believe that the effect of betas is much stronger. You don't want to earn money or grind reputation or raid the same dungeon over and over again.
However, in WAR, the beta was the only place for your WAR character. So, if you got into it early enough, it felt like a "real" character. You always knew that the character would be deleted eventually, but for a significant amount of time it was yours.
So, in WAR, I felt like I was playing the game as I would normally. In WotLK, if I had been invited to the beta, I would have done some very superficial things, then stopped.
It seems that betas of expansions kill all but short-term goals and betas of new games (if long enough) only kill off long-term goals, leaving short- and medium-term goals intact. Or, if it's not even about goals, it could be that you feel that you own a character in a long beta for a standalone game and you never feel that ownership in the beta for an expansion.
Oh, I don't know. I think betas are still extremely useful. To use your example, people pointed out Ret's stupid burst PvP damage for MONTHS in beta - it's just that Blizzard did not choose to act on it, until the forums were awash with tears. They could have saved themselves a lot of trouble had they acted sooner - they simply had other priorities, like fleshing out the DK, new content, new areas.
As to WAR, the reason their beta failed so hard is that it wasn't a beta, really. It was more a series of focused tests : "Ok, this week we're going to test Tier3, elf zones." And a week later they wiped everything and oriented focus on something else. Which is why their overall game design suffers - at no point did they test, for example, viability and opportunity cost of character progression for the various progression paths: ORvR, quests, PQs and scenarios. Also changing the leveling xp curve 2 days before launch doesn't help either.
As to WAR, the reason their beta failed so hard is that it wasn't a beta, really. It was more a series of focused tests : "Ok, this week we're going to test Tier3, elf zones." And a week later they wiped everything and oriented focus on something else. Which is why their overall game design suffers - at no point did they test, for example, viability and opportunity cost of character progression for the various progression paths: ORvR, quests, PQs and scenarios. Also changing the leveling xp curve 2 days before launch doesn't help either.
# posted by 
: 5/11/08 19:25
: 5/11/08 19:25
The thing is that players behave differently on a beta server than on a real server, because their motivations are different
I believe this is the root of all the problems WAR is having. Mythic tested everything to make sure it works right, but didn't test whether players would play as they expected. Actually, it may be impossible to do so, not without creating permanent test servers that people could play on and not have their characters wiped. Then they'd have the same motivations as other players.
Post a Comment
I believe this is the root of all the problems WAR is having. Mythic tested everything to make sure it works right, but didn't test whether players would play as they expected. Actually, it may be impossible to do so, not without creating permanent test servers that people could play on and not have their characters wiped. Then they'd have the same motivations as other players.
<< Home
