www.Tobold.com
Tobold@GMail.com
A blog with my thoughts regarding games I am playing and other stuff in life. Please read my Terms of Service
RSS Feed
Thursday, September 02, 2010
Billing as the weakest link
So Square Enix found and fixed the bug in the Final Fantasy XIV beta and decided to start the open beta today. Which would be good news if you could actually get into the open beta, which you can't: Square Enix failed to give their account and billing server enough hardware / bandwith, and if you try to sign up for the open beta you only get a "server overloaded, try again later" error message.
Actually that happens quite a lot. Not just for betas, but also right after release, there have been numerous cases where the game servers were up and running, but only very few people could play, because the server for making accounts and entering credit card details had broken down.
All those players wanting to get into a new game as early as possible are somewhat similar to a Denial of Service attack. But that onslaught is predictable. If a game company shipped a million copies of their game, they must install the hardware to handle a million players wanting to open an account. Or they must manage account creation and billing differently, so you can already make a character and play, and have 30 days time for the administrative stuff. Square Enix is obviously failing here, and I'm pretty sure they won't be the last ones.
- posted by Tobold @ 8:53 AM Permanent Link
Comments:
<< Home
Newer› ‹Older
Let's see what information comes out in the next weeks. I might still cancel my pre-order.
I won't support yet another 'AAA' MMO launch that enriches managers who think that a game with bugs is just a game with bugs - and not a bad game.
I won't support yet another 'AAA' MMO launch that enriches managers who think that a game with bugs is just a game with bugs - and not a bad game.
I thought that one of the points of an open beta was to act as a practise launch and stress test the chain. I'd argue that you want your hardware to fail. Then you know how many people it took to bring it down. Combine that with how many people you expect for the launch and you'll have a rough idea of what capability you need on launch.
# posted by 
: 2/9/10 11:20
: 2/9/10 11:20
Tobold,
your statement:
"But that onslaught is predictable. If a game company shipped a million copies of their game, they must install the hardware to handle a million players wanting to open an account. "
Is slightly erroneous... No one plans a data center support infrastructure by retail box sales.
Planning the "top end" user base based on these numbers can get you into er... bankruptcy faster than anything else.
No you do your infrastructure planning on "reasonable prior metrics".
Unfortunately Online games have VERY wierd datacenter useage patterns. So you have to 20% guess above said prior metrics and hope. The trick is who pays the 20% that you have that bandwidth and server reserve?
BTW I really like your blog and your example of Billing in this case is interesting. It is the small things that make the deepest impressions sometimes.
your statement:
"But that onslaught is predictable. If a game company shipped a million copies of their game, they must install the hardware to handle a million players wanting to open an account. "
Is slightly erroneous... No one plans a data center support infrastructure by retail box sales.
Planning the "top end" user base based on these numbers can get you into er... bankruptcy faster than anything else.
No you do your infrastructure planning on "reasonable prior metrics".
Unfortunately Online games have VERY wierd datacenter useage patterns. So you have to 20% guess above said prior metrics and hope. The trick is who pays the 20% that you have that bandwidth and server reserve?
BTW I really like your blog and your example of Billing in this case is interesting. It is the small things that make the deepest impressions sometimes.
I still don't understand why billing isn't handled by 3rd parties companies, using SSO (or equivalent) to connect to them on the back end. Why do game companies keep trying to either 1. reinvent the wheel or 2. keep control of servers not in their core competency.
If they had external hosting for billing, the external company could swap in a few appservers for a few days, then take them out. Cheaper for everyone involved as well as more reliable.
If they had external hosting for billing, the external company could swap in a few appservers for a few days, then take them out. Cheaper for everyone involved as well as more reliable.
No you do your infrastructure planning on "reasonable prior metrics".
I wasn't asking MMORPG companies to invest in huge datacenter that can withstand a million new players every day, knowing very well they'll only get a million new players on day one, and the less and less. But that is what cloud computing has been invented for. If you can foresee a huge rush of incoming players, needing bandwith and billing server capacity, you can rent that additional capacity "in the cloud" for a couple of weeks.
What you shouldn't do is shipping a million boxes, knowing that it would take weeks for your billing server to handle them all, thus creating a lot of players who gave you money for a game and are now unable to play it.
I wasn't asking MMORPG companies to invest in huge datacenter that can withstand a million new players every day, knowing very well they'll only get a million new players on day one, and the less and less. But that is what cloud computing has been invented for. If you can foresee a huge rush of incoming players, needing bandwith and billing server capacity, you can rent that additional capacity "in the cloud" for a couple of weeks.
What you shouldn't do is shipping a million boxes, knowing that it would take weeks for your billing server to handle them all, thus creating a lot of players who gave you money for a game and are now unable to play it.
It doesn't make a whole lot of sense to by the hardware to handle a million people at once when you will only have to do that for a few days. Cause then you have a lot of hardware you spent a ton of money on that you should just turn off to save electricity. Better to deal with a few days of frustration at the start.
They could have also had additional billing servers hosted for this month to help with the extra need.
They could have simply implimented some kind of queueing systems so people could virtually wait in line. Blizz did that with their store so they could sell blizzcon tickets.
They could have even simply separated the various tasks to separate days. say a week ago you sign up and get an open beta key, then 2 days ago you register the key and create your account. Then today you actually log in. Doing everything at once at a pre-announced time which happens to be primetime in the US? That is a little bit more than simply not too smart.
Alternatively, they could have picked a day, simply a 24 hr period, told everyone, go to the website to sign up for a beta key. Everyone will be able to sign up for the whole time period. Once it is over, 10,000 keys will be issued at random in addition to those who get keys for being in the closed beta. That way, at least, you spread the load out over 24 hours rather than subtly encouraging everyone to slam your systems right at the begining because you are doing it first come, first serve.
This whole thing has been handle very poorly. Did they really not anticipate this happening? Very poor first impression.
They could have simply implimented some kind of queueing systems so people could virtually wait in line. Blizz did that with their store so they could sell blizzcon tickets.
They could have even simply separated the various tasks to separate days. say a week ago you sign up and get an open beta key, then 2 days ago you register the key and create your account. Then today you actually log in. Doing everything at once at a pre-announced time which happens to be primetime in the US? That is a little bit more than simply not too smart.
Alternatively, they could have picked a day, simply a 24 hr period, told everyone, go to the website to sign up for a beta key. Everyone will be able to sign up for the whole time period. Once it is over, 10,000 keys will be issued at random in addition to those who get keys for being in the closed beta. That way, at least, you spread the load out over 24 hours rather than subtly encouraging everyone to slam your systems right at the begining because you are doing it first come, first serve.
This whole thing has been handle very poorly. Did they really not anticipate this happening? Very poor first impression.
If they intended to have limited capacity because they didn't want to spend the money then they knew they wouldn't be able to handle the load and should have implemented at least a queuing system.
But even so, You can lease equipment when your datacenter is that big.
You can lease datacenter space from another company.
You can buy the extra machines and then use them for other projects. Datacenter flexibility is worth having for its own sake. More datacenters are using a cloud architecture or on a smaller scale some like vmware esx. Use the servers for what they need to be used for at the time and then move different virtual servers on to them when needs change.
Besides, what is a good first impression worth in terms of box sales for a game like this? Having enough capacity is its own type of marketing.
Finally. regardless what they choose to do. There is simply no excuse for page timeouts. You don't throttle your server utilization by telling everybody to jump on this evening at 7pm and then letting them slowly push though by refreshing repeatedly after getting a timeout. That is simply amatureish.
But even so, You can lease equipment when your datacenter is that big.
You can lease datacenter space from another company.
You can buy the extra machines and then use them for other projects. Datacenter flexibility is worth having for its own sake. More datacenters are using a cloud architecture or on a smaller scale some like vmware esx. Use the servers for what they need to be used for at the time and then move different virtual servers on to them when needs change.
Besides, what is a good first impression worth in terms of box sales for a game like this? Having enough capacity is its own type of marketing.
Finally. regardless what they choose to do. There is simply no excuse for page timeouts. You don't throttle your server utilization by telling everybody to jump on this evening at 7pm and then letting them slowly push though by refreshing repeatedly after getting a timeout. That is simply amatureish.
Well this is a common practice in todays mmorpgs. Alot of companies use third party billing that don't give a damn about bugs they just wanna see the cash flow. Take Mortal online for example. Who cares just charge $150 dollars for a "limited edition" 1 year before launch. Huge "bugs" in billing is just a common practice it seems now.
I really wanted to love FF14 but the more I deal with Square the more it feels like I'm dealing with a low budget developer who has never dealt with a release before.
The lack of information and help regarding the beta is staggering. I haven't pre-ordered FF14 yet and I don't plan on it. I did however consider buying it a few days after launch if the open beta doesn't suck.
If I can't log into beta at all I'm just going to write it off.
The lack of information and help regarding the beta is staggering. I haven't pre-ordered FF14 yet and I don't plan on it. I did however consider buying it a few days after launch if the open beta doesn't suck.
If I can't log into beta at all I'm just going to write it off.
Tried to play UBISOFT's Might and Magic Heroes Kingdoms last night, which think launched earlier in the week, and had the same problem.
I can only imagine that Free to play servers experience this 10times more, and they usually have less cash to deal with this in the beginning than the subscription games.
It will be interesting to see how this plays out in the future as more games adopt the free to play model.
I can only imagine that Free to play servers experience this 10times more, and they usually have less cash to deal with this in the beginning than the subscription games.
It will be interesting to see how this plays out in the future as more games adopt the free to play model.
you should see the series of messages I have been having over the last week with a friend about this game. I had been getting really excited. I had been defending their choice to go with the fatigue system on some of the mmo forums. I was really looking forward to this. But the last 24 hrs has been completely deflating.
First, this poor showing so far for a beta.
Second, their history with FFXI of having an absolutely unplayable movement interface.
Third, the fact that I couldn't even subscribe the last time I tried to learn to enjoy FFXI because they would only take credit cards that had been signed up for some weirdo security thing.
With all of this I am simply not willing to take a chance. I simply can't give it a go, site unseen, based on the assumption that I will surely get at least a few weeks of diversion from the game. I couldn't even get a few hrs out of FFXI. I don't expect companies to allow my to use their beta as a trial but in this case I really dont have a choice. If I can't get into this beta I will cancel my pre order and wait for a free trial to be offered.
First, this poor showing so far for a beta.
Second, their history with FFXI of having an absolutely unplayable movement interface.
Third, the fact that I couldn't even subscribe the last time I tried to learn to enjoy FFXI because they would only take credit cards that had been signed up for some weirdo security thing.
With all of this I am simply not willing to take a chance. I simply can't give it a go, site unseen, based on the assumption that I will surely get at least a few weeks of diversion from the game. I couldn't even get a few hrs out of FFXI. I don't expect companies to allow my to use their beta as a trial but in this case I really dont have a choice. If I can't get into this beta I will cancel my pre order and wait for a free trial to be offered.
This shouldnt be surprising to anyone who played FFXI even once.
The game had absolutely horrible interface, login, account management, etc, etc, etc.
All the problems I am hearing about all come back to the same thing: they have no idea what they are doing in this department.
It should surprise no one, and I for one never even considered FFXIV because I *knew* that SE's style is shoddy, amateur, over-complicated and plain stupid when it comes to MMOs.
That's not even getting into asinine movement and "features".
The game had absolutely horrible interface, login, account management, etc, etc, etc.
All the problems I am hearing about all come back to the same thing: they have no idea what they are doing in this department.
It should surprise no one, and I for one never even considered FFXIV because I *knew* that SE's style is shoddy, amateur, over-complicated and plain stupid when it comes to MMOs.
That's not even getting into asinine movement and "features".
This is just another common trend of todays mmo companies. They launch a bug ridden game and expect people to be happy and keep paying like its nothing.
Billing continues to be the weak link too. What ultimately ran me off from Warhammer for good was my brother being billed over 30 times for one account and over 15 for the other, completely blowing his bank account up and putting him into economic limbo for a week.
I don't care how good your game is. I'm not risking my financial well-being over it.
I don't care how good your game is. I'm not risking my financial well-being over it.
lol,
tobold you are so funny
Get it from the cloud indeed...
Ok so here's a senario to consider.
Your billing computer is swamped. You say to your infrastructure provider. "Hey get me more powar from the cloud".
Your infrastructure service provider gently reminds you that according to SET (Secure Electronic Transactions - http://en.wikipedia.org/wiki/Secure_Electronic_Transaction). They cannot "outsource" processing to the "cloud" without prior Accreditation of said "cloud stuff". [people who deal in real money transactions tend to be touchy about not being able to verify when and by who a cash transaction takes place... go figure]
Dev - But... But... Cloud computing promises processing on demand...
Service provider - True the infrastructure provider says. However, if we use a cloud provider like say Google we have no way of verifying that the server/bandwidth/data resources are within an enforceable legal jurisdiction like say China/India.
Dev - But we could just make Google...
Service provider - NO Google's cloud is architected to be ad hoc martialing with best match routing of resources. Our data storage could be in Europe, Our processing in China, our encryption to VISA back end processing in Saudi Arabia...
Dev - But Saudi Arabia does not allow strong encryption...
Service provider - Yes you see the problem now... Mr game developer.
Dev - I guess Clouds wont work in the real world...
Service - nope pretty much like all outsourced data services only true hosting will live on... Deja vu all over again see in the 1970s there was this company called EDS... that thought if they just rent out mainframe processing from datacenters world wide and THEN they would be able to take over the computer processing for ALL of the large companies... GM thought so much of this idea that they bought EDS in the 1980s right when this toy thing called a Micro Computer was just comming out...
tobold you are so funny
Get it from the cloud indeed...
Ok so here's a senario to consider.
Your billing computer is swamped. You say to your infrastructure provider. "Hey get me more powar from the cloud".
Your infrastructure service provider gently reminds you that according to SET (Secure Electronic Transactions - http://en.wikipedia.org/wiki/Secure_Electronic_Transaction). They cannot "outsource" processing to the "cloud" without prior Accreditation of said "cloud stuff". [people who deal in real money transactions tend to be touchy about not being able to verify when and by who a cash transaction takes place... go figure]
Dev - But... But... Cloud computing promises processing on demand...
Service provider - True the infrastructure provider says. However, if we use a cloud provider like say Google we have no way of verifying that the server/bandwidth/data resources are within an enforceable legal jurisdiction like say China/India.
Dev - But we could just make Google...
Service provider - NO Google's cloud is architected to be ad hoc martialing with best match routing of resources. Our data storage could be in Europe, Our processing in China, our encryption to VISA back end processing in Saudi Arabia...
Dev - But Saudi Arabia does not allow strong encryption...
Service provider - Yes you see the problem now... Mr game developer.
Dev - I guess Clouds wont work in the real world...
Service - nope pretty much like all outsourced data services only true hosting will live on... Deja vu all over again see in the 1970s there was this company called EDS... that thought if they just rent out mainframe processing from datacenters world wide and THEN they would be able to take over the computer processing for ALL of the large companies... GM thought so much of this idea that they bought EDS in the 1980s right when this toy thing called a Micro Computer was just comming out...
FYI -
Why can't the outsource Billing - SET Visa won't allow it... unless you do the transaction as a passthrough. This has BIG security risks but can be done... I don't recommend it if you are running above $100k in dollar value transactions. If you are SOMEONE will redirect your passthrough to Nigeria real quick.
Why can't they just rent servers for peaks? - What do you do with said server when you are done with "peak" - give over to pron hosting perhaps? what about the data and programs that facilitate c card transactions? Who takes care of those? Wipe it... in what way data stays around after wipes. You want those c card numbers recoverable? Whats the liability of not securing the "rented" server? (it's big by the way)
Oh by the way C Card validation/payment software is quite expensive PayPal was luuuccckkky that they came along BEFORE the authorization networks realized how much they could charge online retailers for "hooking up"... but that is another story.
Why can't the outsource Billing - SET Visa won't allow it... unless you do the transaction as a passthrough. This has BIG security risks but can be done... I don't recommend it if you are running above $100k in dollar value transactions. If you are SOMEONE will redirect your passthrough to Nigeria real quick.
Why can't they just rent servers for peaks? - What do you do with said server when you are done with "peak" - give over to pron hosting perhaps? what about the data and programs that facilitate c card transactions? Who takes care of those? Wipe it... in what way data stays around after wipes. You want those c card numbers recoverable? Whats the liability of not securing the "rented" server? (it's big by the way)
Oh by the way C Card validation/payment software is quite expensive PayPal was luuuccckkky that they came along BEFORE the authorization networks realized how much they could charge online retailers for "hooking up"... but that is another story.
I'll take the obvious and slightly unfair potshot here:
The sense of entitlement in this thread is strong. Because it's obvious that dealing with immense network loads is just a matter of throwing more hardware at the problem, right? I realize developers/publishers are as guilty as players for treating betas like glorified demos, but to expect them pre-release to have solved a problem endemic to the infrastructure of the internet is a bit rich. Cloud based services are the new hotness and a great way, in principle, to deal with scalability issues.
However, they're not a panacea. The "pipes" of the internet are only so wide and the routers can only switch so far. Unless you have the distributed resources of a Google, these sort of short term spikes will be problematic. There is also the institutional conservativeness of large companies reluctant to (or technically unable due to legacy systems) hand over proprietary code to a place like Amazon. Have an irrational dislike of downloading games because "WHAT IF STEAM GOES AWAY!?!" Imagine what that's like for a vertically integrated, old guard developer/publisher like Square-Enix.
The sense of entitlement in this thread is strong. Because it's obvious that dealing with immense network loads is just a matter of throwing more hardware at the problem, right? I realize developers/publishers are as guilty as players for treating betas like glorified demos, but to expect them pre-release to have solved a problem endemic to the infrastructure of the internet is a bit rich. Cloud based services are the new hotness and a great way, in principle, to deal with scalability issues.
However, they're not a panacea. The "pipes" of the internet are only so wide and the routers can only switch so far. Unless you have the distributed resources of a Google, these sort of short term spikes will be problematic. There is also the institutional conservativeness of large companies reluctant to (or technically unable due to legacy systems) hand over proprietary code to a place like Amazon. Have an irrational dislike of downloading games because "WHAT IF STEAM GOES AWAY!?!" Imagine what that's like for a vertically integrated, old guard developer/publisher like Square-Enix.
AAAAAAAAHHHHHH - I did not just read this...
"Billing continues to be the weak link too. What ultimately ran me off from Warhammer for good was my brother being billed over 30 times for one account and over 15 for the other, completely blowing his bank account up and putting him into economic limbo for a week.
"
NEVER EVER GIVE OUT A DEBIT CARD TO AN ONLINE VENDOR...
You don't leave your wallet in a mall do you?
A debit card is a direct line to your bank account without (in the US) any protections except the kindness of your bank.
with normal credit cards you are limited to $50 loss a day period
Debit - Unlimited Losses
Credit - $50 dollars a day
Which should you use?
http://en.wikipedia.org/wiki/Fair_Credit_Billing_Act
Post a Comment
"Billing continues to be the weak link too. What ultimately ran me off from Warhammer for good was my brother being billed over 30 times for one account and over 15 for the other, completely blowing his bank account up and putting him into economic limbo for a week.
"
NEVER EVER GIVE OUT A DEBIT CARD TO AN ONLINE VENDOR...
You don't leave your wallet in a mall do you?
A debit card is a direct line to your bank account without (in the US) any protections except the kindness of your bank.
with normal credit cards you are limited to $50 loss a day period
Debit - Unlimited Losses
Credit - $50 dollars a day
Which should you use?
http://en.wikipedia.org/wiki/Fair_Credit_Billing_Act
<< Home
