www.Tobold.com
Tobold@GMail.com
A blog with my thoughts regarding games I am playing and other stuff in life. Please read my Terms of Service
RSS Feed
Wednesday, December 03, 2008
Is the WAR endgame technically possible?
I was reading an interesting article on Keen & Graev's blog about a WAR fortress siege that was extremely laggy and ended up crashing the server when over 200 people turned up. MMORPGs in general get laggy and prone to crashes when there are too many players in the same zone. The WoW world event of opening of the Ahn'qiraij gates was a horrible lagfest, and even nowadays I have significantly more lag in overpopulated places like Dalaran. But generally WoW does a good job of dispersing people, by sending them off in various directions with quests. And especially if you consider the WoW endgame, people will be in groups of 10 or 25 people in a raid instance, a sufficiently small number to not cause big lag problems on half-decent computers. The endgame of WAR on the other hand is about capturing fortresses and cities, with no limit to participation. The more people reach the level cap in WAR (which appears to be a relatively slow process), the more likely it becomes that hundreds of them will turn up for a battle. These large-scale battles are the USP, the unique selling point of WAR. But are they actually technically possible?
The problem appears to be the amount of data your computer needs to process when there are hundreds of people around you. The problem is worse in a MMORPG than in other multiplayer games, because in a MMORPG every player looks different, with the avatar you are seeing on your screen representing his race, class, and equipment. So I was wondering whether to reduce the lag it would be possible to switch to a display mode where you see everyone with much simplified avatars. Like one avatar for every class/race combo, with no display of what color of armor he is wearing or whether he wields an axe or a sword. It's not as if you'd care whether you are being hit by a sword or axe. Yes, if you were fighting several Ironbreakers they'd all look like clones, the only difference between would be the name floating over their heads. But if that reduced lag considerably, and made crash-free fortress battles possible, wouldn't that be a price worth paying?
I remember Final Fantasy XI where the lag problem was solved by simply *not* displaying everyone immediately. You'd walk onto a crowded space in front of the auction house and at first see nobody, until they one after the other popped into view. Obviously that isn't a feasible solution for a PvP game. So if you need to see every other player, friend or foe, in your vicinity, the only technical solution I see is to reduce the amount of data necessary per player. I remember having seen a cartoon once, but can't find it any more, from somebody who buys a new first-person shooter game, swoons for 5 minutes about the spectacular graphics and effects, and then turns the resolution to minimum and all effects off to be able to actually be fast enough to win. Couldn't WAR introduce a simple toggle switch which you use when you start to experience lag and which reduces graphics to an ugly minimum, but allows you to continue playing without (or with less) lag? Pretty is good, playable is better.
- posted by Tobold @ 11:10 AM Permanent Link
Comments:
<< Home
Newer› ‹Older
The thing is without having a clue on the architecture of an MMO is very hard to see where things can be improved.
Different games mean different problems and most important different bottlenecks.
I think most games work like this: all the graphic rendering is made on the client based on information received from the server. That information is probably strings of characters that tell things like position of players, mobs, equipment, stats, movements, etc. That information has to be stored somewhere, so games will use databases in order to properly store and the #&%@load of information that a game has.
So, as long as animators, modellers, etc, you have to have excellent network and database programmers. And these do not grow on trees.
Anyway, I do believe, due to my professional experience, that probably one of the biggest bottlenecks is querying data in a split second in order to send information on 200 players and not sending the said data over the network. And the query for each player would be relatively simple, so i think that even if you had a game of matchstick men you would still have lag.
Either that, or most MMO operators really get cheap servers for their infrastructure...
Different games mean different problems and most important different bottlenecks.
I think most games work like this: all the graphic rendering is made on the client based on information received from the server. That information is probably strings of characters that tell things like position of players, mobs, equipment, stats, movements, etc. That information has to be stored somewhere, so games will use databases in order to properly store and the #&%@load of information that a game has.
So, as long as animators, modellers, etc, you have to have excellent network and database programmers. And these do not grow on trees.
Anyway, I do believe, due to my professional experience, that probably one of the biggest bottlenecks is querying data in a split second in order to send information on 200 players and not sending the said data over the network. And the query for each player would be relatively simple, so i think that even if you had a game of matchstick men you would still have lag.
Either that, or most MMO operators really get cheap servers for their infrastructure...
As someone currently actually working in the industry on a triple-A MMO, perhaps I’m somewhat qualified to comment. You should distinguish between client-side lag and server-side. Many MMOs are plagued with both, but it’s server-side processing that’s really the intractable problem.
When a lot of players gather in one place, there will be some impact on client side graphical performance for some people (those with lesser systems). This is your standard, single-player game frame rate hit – the more going on onscreen, the more work the client has to do to render it and the fewer frames it can render in a given second. As we all know, this is not a problem unique to MMOs, and the solutions are also well known – some MMOs just implement them better than others.
The REAL problem in MMOs with high player densities is that the servers assigned to manage player activity in those particular regions become heavily loaded. Most of the really important gameplay logic in an MMO is handled server side. Most MMO back-end architectures attempt to distribute this load amongst as many machines as possible – usually by region-subdivision. So what people consider a logical server or ‘realm’ in MMO terms is likely to be a cluster, each individual machine handling different regions of the world. This only really works when people are well distributed amongst the various machines (i.e. spread around the region). It’s the reason there’s usually more than one region in WoW for a given level range, for instance.
The more players gather in a single region, the more the load becomes concentrated onto a single physical machine, and the less it is able to keep the game state in sync amongst the various players. So the latency between network packets grows and you get the infamous lag. This is NETWORK lag related to high server load, as opposed to graphical ‘lag’ related to a struggling client machine. The distinction is important.
So far, there has been no silver bullet to the problem of server side load distribution in MMOs. There are many ways to try to mitigate the problem, but the bottom line is that you can’t really scale an MMO the way you can scale a webserver, because while a webserver can treat every person requesting a page as independent (and therefore farm them out and service them across a cluster), every person playing an MMO requests information from the server about the other players, and it’s this coordination of information that cannot easily be distributed. So even so called “unsharded” MMOs like EVE may have 50k concurrent players on a single logical ‘realm’, but you sure as hell won’t get more than a few hundred in a single system.
I don’t see this fundamental problem being solved anytime soon.
When a lot of players gather in one place, there will be some impact on client side graphical performance for some people (those with lesser systems). This is your standard, single-player game frame rate hit – the more going on onscreen, the more work the client has to do to render it and the fewer frames it can render in a given second. As we all know, this is not a problem unique to MMOs, and the solutions are also well known – some MMOs just implement them better than others.
The REAL problem in MMOs with high player densities is that the servers assigned to manage player activity in those particular regions become heavily loaded. Most of the really important gameplay logic in an MMO is handled server side. Most MMO back-end architectures attempt to distribute this load amongst as many machines as possible – usually by region-subdivision. So what people consider a logical server or ‘realm’ in MMO terms is likely to be a cluster, each individual machine handling different regions of the world. This only really works when people are well distributed amongst the various machines (i.e. spread around the region). It’s the reason there’s usually more than one region in WoW for a given level range, for instance.
The more players gather in a single region, the more the load becomes concentrated onto a single physical machine, and the less it is able to keep the game state in sync amongst the various players. So the latency between network packets grows and you get the infamous lag. This is NETWORK lag related to high server load, as opposed to graphical ‘lag’ related to a struggling client machine. The distinction is important.
So far, there has been no silver bullet to the problem of server side load distribution in MMOs. There are many ways to try to mitigate the problem, but the bottom line is that you can’t really scale an MMO the way you can scale a webserver, because while a webserver can treat every person requesting a page as independent (and therefore farm them out and service them across a cluster), every person playing an MMO requests information from the server about the other players, and it’s this coordination of information that cannot easily be distributed. So even so called “unsharded” MMOs like EVE may have 50k concurrent players on a single logical ‘realm’, but you sure as hell won’t get more than a few hundred in a single system.
I don’t see this fundamental problem being solved anytime soon.
Actually.. EvE recently had some successful 1000+-person battles with their new server architecture. However, as I've explained before, having a lot of active participants increases the server and bandwidth requirements exponentially, so it's quite hard to scale beyond that.
Since the StacklessIO/64bit updates EVE's alliances have been fighting regularly with 1000-1200 on-grid with "managable" lag and not too many node crashes. EVE is a special case though, for a variety of design reasons. I don't expect a more conventional avatar-based game could lag as much and still be considered playable.
# posted by 
: 3/12/08 13:52
: 3/12/08 13:52
To use the WoW comparison, I remember the January after the initial launch, attacking (with a group of about 30 Horde) that Alliance town in Ashenvale. It was an awful lag fest, and my toon got stuck.
It eventually got patched. I expect WAR to eventually do the same.
It eventually got patched. I expect WAR to eventually do the same.
# posted by : 3/12/08 14:31
: 3/12/08 14:31
Remember how everyone complained when Funcom limited the sieges to 40+ members per side in Age of Conan? The uproar was hilarious. Yet, it is logical to limit these war's to skirmishes due to the data needed.
Technology is not close enough to make RTS type wars possible...yet (even though Dreamlords tries this, but I am unsure of total players on field..and we know a lot of NPC's are involved...ARGH)
Anyways, AoC did do the toggle, but takes control from you, and turns it on depending on the number of players...the more that appear, the more particles are turned off, etc..the system is smart enough to do it.
This is why I still feel AoC has more potential for the future of MMO large scale combat, thanks to the newer engine that is constantly being worked on.
WAR I do not have much faith in if its simple graphics engine and code can't even handle this type of load.
Technology is not close enough to make RTS type wars possible...yet (even though Dreamlords tries this, but I am unsure of total players on field..and we know a lot of NPC's are involved...ARGH)
Anyways, AoC did do the toggle, but takes control from you, and turns it on depending on the number of players...the more that appear, the more particles are turned off, etc..the system is smart enough to do it.
This is why I still feel AoC has more potential for the future of MMO large scale combat, thanks to the newer engine that is constantly being worked on.
WAR I do not have much faith in if its simple graphics engine and code can't even handle this type of load.
Wow still turns into a lagfest whenever sizable numbers congregate in the same place. Pre-wotlk city sieges were all the rage, and I remember turning up to defend TB against a raid group or two, I guess somewhere around 100 players in total (both sides) and the FPS on my usually very good (ati 4870, 4gb ram, core 2 cpu) computer ground to a halt.
The difference with WAR is that the that game is designed around large scale sieges. With Wow, 100 players in the same place is a real rarity (even on servers with many thousands of players) but in WAR it is the aim of the game. That is something that WAR needs to do it's best to fix if the "endgame" is going to have any future.
The difference with WAR is that the that game is designed around large scale sieges. With Wow, 100 players in the same place is a real rarity (even on servers with many thousands of players) but in WAR it is the aim of the game. That is something that WAR needs to do it's best to fix if the "endgame" is going to have any future.
If I remember correctly the original WAR design was much more highly instanced. Then due to player feedback it was shifted more toward open RvR sometime in mid beta. Perhaps if WAR had been designed around large open RvR encounters from the ground up the performance would be better.
It is difficult to draw parrallels in WoW but I know that I have been in large (200+ players) assaults on capital cities that have crashed the servers. However whatever they have done to Wintergrasp to support that much combat seems to be working well. I play with the graphics settings all the way up and had little to no lag in about a 40v40 matchup last night. (Guessing the numbers as we had a 30 man raid with quite a few solo players and neither side had the tenacity buff.)
It is difficult to draw parrallels in WoW but I know that I have been in large (200+ players) assaults on capital cities that have crashed the servers. However whatever they have done to Wintergrasp to support that much combat seems to be working well. I play with the graphics settings all the way up and had little to no lag in about a 40v40 matchup last night. (Guessing the numbers as we had a 30 man raid with quite a few solo players and neither side had the tenacity buff.)
# posted by : 3/12/08 15:28
: 3/12/08 15:28
Nick wrote: The REAL problem in MMOs with high player densities is that the servers assigned to manage player activity in those particular regions become heavily loaded.
I just want to echo Nick's point. Drawing 200 custom swords for nearby players may slow you down if you have a slow computer, but it doesn't tend to be the real problem. Like Nick says, the real problem tends to be that the game server (the server, not your computer) gets overwhelmed dealing with the data.
Suppose 2 people are fighting and they each cast a spell. The server needs to send four messages: tell person A that A's spell cast worked, tell person B that A's spell cast worked, tell A that B's spell cast worked, and tell B that B's spell cast worked. The server needs to send this message so that player A and B's computer can draw the right spell effects, subtract the right damage from the onscreen health display, etc.
If 3 people are fighting, each person needs to be informed of the other 3 person's spells and actions. That means 9 messages per round of action-taking.
For 4 people, 16 messages.
The number of messages scales as the square of the number of people, and when you reach 200 people, the server is doing 10,000 times as much work as it was doing for the 2-person battle. Eventually you will reach a point where the server can't keep up.
I suspect that are good strategies to help deal with this N-squared scaling problem, but I do not work in the MMORPG industry, and I have no idea what the strategies ma be.
I just want to echo Nick's point. Drawing 200 custom swords for nearby players may slow you down if you have a slow computer, but it doesn't tend to be the real problem. Like Nick says, the real problem tends to be that the game server (the server, not your computer) gets overwhelmed dealing with the data.
Suppose 2 people are fighting and they each cast a spell. The server needs to send four messages: tell person A that A's spell cast worked, tell person B that A's spell cast worked, tell A that B's spell cast worked, and tell B that B's spell cast worked. The server needs to send this message so that player A and B's computer can draw the right spell effects, subtract the right damage from the onscreen health display, etc.
If 3 people are fighting, each person needs to be informed of the other 3 person's spells and actions. That means 9 messages per round of action-taking.
For 4 people, 16 messages.
The number of messages scales as the square of the number of people, and when you reach 200 people, the server is doing 10,000 times as much work as it was doing for the 2-person battle. Eventually you will reach a point where the server can't keep up.
I suspect that are good strategies to help deal with this N-squared scaling problem, but I do not work in the MMORPG industry, and I have no idea what the strategies ma be.
One of BRK's latest endeavours ended up crashing the Argent Dawn server when it couldn't handle a few hundred people attacking and defending Stormwind. If the World of WARcraft servers can't handle more than a few hundred people beating the crap out of each other, I don't like WAR's chances.
What I find odd about this is that such loads are eminently predictable for a game designed for an RvR endgame. It's a touch more forgivable for WOW, where large scale PvP wasn't part of the original design, but if WAR failed in this area, that would be bad design, pure & simple. Didn't they test this stuff before they released?
Just one possible correction - I can't see Keens site from work anymore, but if he's talking about the Fortress attempt on the Maw that went down a few days ago, that was actually 800 or so players according to Mythic's metrics. It represents a rare case but Mythic is already trying to find ways to make it work better, some of which will likely include graphics display changes like you mention.
# posted by : 3/12/08 18:08
: 3/12/08 18:08
Since last patch, lag is a lot less, but the zones still crash.
In principle, that is encouraging, because a crash has to be fixable, whereas lag, as Tobold suggests, might not have been.
In principle, that is encouraging, because a crash has to be fixable, whereas lag, as Tobold suggests, might not have been.
# posted by : 3/12/08 18:12
: 3/12/08 18:12
a dynamic cone-of-vision type visible area can help with reducing lags, for melee player the cone can be shorter / up to his max ranged weapon throw. front cone of vision should be a bit farther than sides / back vision.
this way it will limit the amount of data traffic.
i have compared WAR and WOW traffic during scenario (WAR) and battleground (WOW) and WAR's network traffic is higher
this way it will limit the amount of data traffic.
i have compared WAR and WOW traffic during scenario (WAR) and battleground (WOW) and WAR's network traffic is higher
# posted by : 3/12/08 18:14
: 3/12/08 18:14
Having a FoV limit is all well and good, until:
a) You turn quickly.
b) Someone is stabbing you in the back.
c) Someone is bunnyjumping at the edge. Now you see him, now you don't.
d) You happen to gaze down from your keep into the wide open battlefield.
a) You turn quickly.
b) Someone is stabbing you in the back.
c) Someone is bunnyjumping at the edge. Now you see him, now you don't.
d) You happen to gaze down from your keep into the wide open battlefield.
Previously WAR had too few players to make open RvR work... but now there are too many, you're right. The easiest way to make this concept work would be to have multiple copies of each zone a la Age of Conan, but that doesn't sit right with a lot of people. I'm not sure if it can actually be solved to both be enjoyed by the players conceptually (ie not artificially locking people out of a particular zone or instance) and technically (achieving proper frame rates).
Sorry to double post. Was just reading the comments:
"I play with the graphics settings all the way up and had little to no lag in about a 40v40 matchup last night. (Guessing the numbers as we had a 30 man raid with quite a few solo players and neither side had the tenacity buff.)"
40v40 is quite do-able. Several FPS games have supported 32v32 fights for some time. The sort of battles where this uber lag starts to dominate are reportedly several hundred per side.
"I play with the graphics settings all the way up and had little to no lag in about a 40v40 matchup last night. (Guessing the numbers as we had a 30 man raid with quite a few solo players and neither side had the tenacity buff.)"
40v40 is quite do-able. Several FPS games have supported 32v32 fights for some time. The sort of battles where this uber lag starts to dominate are reportedly several hundred per side.
Remember with our current technology we have been PvPing 40 vs 40 in AV for last few years.
# posted by : 4/12/08 04:33
: 4/12/08 04:33
Server side lag FTL.
Has distributing load across players rather than regions been tried? Ie. have a computer dedicated for every 100 players, or less? Or do they already try that, and the network communication is still a problem and more inherent in the exponential links than the strain on one computer?
Has distributing load across players rather than regions been tried? Ie. have a computer dedicated for every 100 players, or less? Or do they already try that, and the network communication is still a problem and more inherent in the exponential links than the strain on one computer?
# posted by : 4/12/08 07:08
: 4/12/08 07:08
Regarding client-side graphic lag, I agree that having an option to limit the number of different characters/weapons to ease up on the different things the PC has to render.
In Final Fantasy XI and Lineage II, they have very few faces/character customization options compared to other MMOs for a reason.
In Final Fantasy XI and Lineage II, they have very few faces/character customization options compared to other MMOs for a reason.
# posted by : 4/12/08 08:45
: 4/12/08 08:45
"Has distributing load across players rather than regions been tried?"
The problem with that is that each player is really only interested in a subset of the other players, and now they're scattered randomly across multiple services. So now to properly update your client, your server has to find out who's near you in the game world, find out which servers they're on, get all their updates and send them down to you. That's a lot of network communication and coordination on the back end network, and it's potentially increased application complexity - to work out who you're 'interested' in at any given time and keep you updated about it.
MMOs are inherently spatially partitioned - you only care about server updates from people near you. So it makes far more sense to have servers clustered by region, which is why every MMO released thus far has done it this way. I hear Bioware may be doing something different with their upcoming MMO, but it remains to be seen whether it solves the scalability problem properly or exchanges it for some different (perhaps lesser, perhaps not) set of problems.
Partitioning by region IS a pain, but it's a workable solution that lots of people have implemented successfully. It's difficult to do server handovers (when you move from region to region) - they're an infamous source of duping bugs. It's difficult to manage players in general near the boundaries - if I'm on one server in one region, and you're in another adjacent, I may still need to see you - this is one reason why games like WoW have very well defined border regions - you're always going through a tunnel or across some narrow gorge to get to an adjacent zones - there are strong technical reasons to design the world like this, it makes server handovers easier (there may be other design or artistic reasons too). It's also the reason games like EVE simply use a zone load.
So yeah, it's not a perfect system, but no one has come up with something workable that scales better or is easier to implement.
The problem with that is that each player is really only interested in a subset of the other players, and now they're scattered randomly across multiple services. So now to properly update your client, your server has to find out who's near you in the game world, find out which servers they're on, get all their updates and send them down to you. That's a lot of network communication and coordination on the back end network, and it's potentially increased application complexity - to work out who you're 'interested' in at any given time and keep you updated about it.
MMOs are inherently spatially partitioned - you only care about server updates from people near you. So it makes far more sense to have servers clustered by region, which is why every MMO released thus far has done it this way. I hear Bioware may be doing something different with their upcoming MMO, but it remains to be seen whether it solves the scalability problem properly or exchanges it for some different (perhaps lesser, perhaps not) set of problems.
Partitioning by region IS a pain, but it's a workable solution that lots of people have implemented successfully. It's difficult to do server handovers (when you move from region to region) - they're an infamous source of duping bugs. It's difficult to manage players in general near the boundaries - if I'm on one server in one region, and you're in another adjacent, I may still need to see you - this is one reason why games like WoW have very well defined border regions - you're always going through a tunnel or across some narrow gorge to get to an adjacent zones - there are strong technical reasons to design the world like this, it makes server handovers easier (there may be other design or artistic reasons too). It's also the reason games like EVE simply use a zone load.
So yeah, it's not a perfect system, but no one has come up with something workable that scales better or is easier to implement.
Most likely Warhammer will be upgraded in some fashion to deal with the hundreds of people showing up. It's probably one of their main focuses right now, since the game is about "large scale RvR warfare" in my opinion. With the recent server mergers, it *should* help with the balancing issues also, but with that comes the problem with what this blog article is about, lots of people at one place at a single time.
# posted by : 5/12/08 03:40
: 5/12/08 03:40
An afterthought; WAR would indeed benefit from a really low detail mode which you can switch to on the fly. Reminds me of Adobe InDesign's low/med/high detail hotkeys.
# posted by : 5/12/08 09:17
: 5/12/08 09:17
The biggest reason this is even an issue right now is the lack of Cities for the other two matchups. If more than two cities were in-game, you'ld see a more natural distribution of players across the endgame zones... when one side pushes, they leave themselves open on two other fronts.
The only way I can see them solving that issue in the short run is to tighten up the zone-locking requirements to force at least two mostly simultaneous pushes to contest a city. Of course if you make it TOO tough, no one will bother. Maybe buff up the GvD/HEvDE Fortress rewards or add PQ's and other objectives to them?
In the long run, they have GOT to get the other cities in-game.
Post a Comment
The only way I can see them solving that issue in the short run is to tighten up the zone-locking requirements to force at least two mostly simultaneous pushes to contest a city. Of course if you make it TOO tough, no one will bother. Maybe buff up the GvD/HEvDE Fortress rewards or add PQ's and other objectives to them?
In the long run, they have GOT to get the other cities in-game.
# posted by : 5/12/08 19:24
: 5/12/08 19:24<< Home
