Monday, April 15, 2013

In discussions on the internet, when discussing groups of people, quite often these people are divided into two groups: The haves and the have-nots, the good and the bad, the intelligent and the stupid, the hardcore and the casual, and so on. Statements about these two groups are usually made as if there was a clear distinction between them. In mathematical terms, if you plotted something like video game skill on the x-axis and the number of people having this skill on the y-axis, people talk of it as if the distribution was bi-modal; that is as if the curve would have two distinctive humps, one of good players and one of bad players.

Scientifically speaking that is utter nonsense. The Central Limit Theorem says that if you make for example this plot of video game skill of a large enough population, what you will get is a bell curve with a single hump in the middle. That is why this curve is called a "normal distribution". The nature of this curve is that 68% of people are withing one standard deviation of the average. For example 68% of people have an IQ between 85 and 115, and are thus of average intelligence. Of course people are notoriously bad at estimating their own IQ or other qualities, so that if you rely on self-assessment you end up with the observation that most people are above average, which is a mathematical impossibility.

Why is that important in a discussion about games? For example I was reviewing a pen & paper roleplaying system yesterday and remarked that it was designed for experienced game masters and groups. And I got a comment saying "The rules system ain't helping a crap GM.". You see the pattern of thinking I described above: If a GM isn't experienced, he must be crap. The reality is that most game masters are average, and what I wanted to say in my review was that this system wasn't suitable for the average group. Yes, a "crappy" game master can ruin any system. But "crappy" GMs are exactly as rare as brilliant ones, and most GMs are simply average. And in my opinion certain rules systems are more suitable for average GMs and players than others are.

The same consideration is true for any other discussion about e.g. video game skill or dedication. Most people have an average skill and average dedication to a game. For a game to work well, it needs to work for the average, because that is most of the audience. "The good" and "the bad" are two more extreme, and much rarer cases, and are thus less important to consider in game design. Of course the extremes can be important for business models, for example the Free2Play whales who subsidize the game for everybody else. But that only works for games with specific business models, it would be a lot more difficult to give thousands of dollars to Blizzard for World of Warcraft, even if you purchased all possible pets and mounts. And while I am on a spending spree trying to buy every possible 4th edition D&D book and adventure, I doubt that will make a noticeable impression on the finances of Wizards of the Coast.

Designing for the average is actually rather difficult, as they aren't easy to get hold of. Various games for example had extensive beta tests, and then found to their surprise that average players in the release version of the game behaved very differently than the beta players, who by definition were a more dedicated part of the total audience. People voicing their opinion on game forums are likewise usually not average players. So if you design for the vocal minority of either extreme, you can run into problems with the silent majority.

In summary, I find it helps to think of people as being mostly average, as opposed to dividing them into two groups. Most of the people talking about "bad players", or "morons & slackers", or any of these terms are just chest-thumping to demonstrate their inflated opinion of themselves. The reality, as usual, is much more mundane. Most people are average.

Completely wrong and proven wrong. To avoid game discussions, let me just link you this post: http://greedygoblin.blogspot.hu/2010/05/its-not-normal-distribution.html

While "skill" is indeed normal distribution, it would only mean that a randomly selected group of people, given the same training in a game and given the same game task, would indeed provide a bell curve as result.

But in games there are indeed two very distinct groups: those who CARE about the game and those who just play "for fun". The latter will perform horribly, simply because he doesn't read up, doesn't ask questions and doesn't listen to advices.

The best example to games would be language speaking. If you display the skill of English speaking of people, you would find a huge group who don't speak at all or just basic words and find another group which is more or less fluent, with very little in between. If you zoom on the "average" English speakers, you'll find them "newbies", people who are actively learning English.

The latter will perform horribly, simply because he doesn't read up, doesn't ask questions and doesn't listen to advices.

Typical example of the chest-thumping mentioned by me: Everybody who distributes people into good and bad players ALWAYS sorts himself into the good category.

While the few horrible players tend to stick out the most in a pickup dungeon or raid, having played through thousands of them I can say with certainty that the most likely result is being in a group of average players. Which, depending on game design prevailing at the time, might or might not be sufficient to beat that dungeon or raid.

The people who can't play WoW don't play WoW, just like the people who can't speak English will just speak something else and only try their few broken words when every other mode of communication failed.

There is absolutely no scientific evidence of the existence of two distinctly separate groups of "people who care" and "people who don't care" in World of Warcraft or any other game.

while I was reading the article I thought exactly what gevlon said, but he was first so I will not repeat it :)

some people just will never try to be better and when they see an obstacle they will quit instead of improve. And I am not only talking about in-game skill. Some people will not even listen advices about keybinds.

I still know people who click their abilities and they swear that they can move/turn/target and cast abilities at the same time as fast as someone who use keybinds. But when we are in difficult situations you can clearly see their disadvantage.

Gevlon, I don't see any reason why the level of "care" wouldn't be normally distributed. Most people have an average level of "care" and then there is a small minority of those who consider the game more important than real life, and a small minority of those who don't care about it at all.

Ok, so according to the central limit theorem, we all have an average of 1.0 testicles.....

In probability theory, the central limit theorem (CLT) states that, given certain conditions, the mean of" will be approximately normally distributed.[1] The central limit theorem has a number of variants. In its common form, the random variables must be identically distributed. In variants, convergence of the mean to the normal distribution also occurs for non-identical distributions, given that they comply with certain conditions.

There are some hypotheses which must be verified for the CLT to work, and it's not clear at all if they are. Even worse, it's not even clear what this "skill" thing is and how you would measure it. Is it raid progression? DPS? K/D ratio? WoT efficiency?

Even assuming we know how to measure skill, are we really sure that we're dealing with " a sufficiently large number of independent random variables, each with a well-defined mean and well-defined variance"? People who play a game are a self-selected sample, additionally, the game itself can perform a pre-selection favoring a specific type of players.

Honestly, I would not be surprised if the result turns out to be anything, where "anything" can be different from one game to the other.....

There is no disagreement over "average skill". However "average dedication" is a nonsense. Dedication is binary, you are or you are not. It comes down to the very binary question: do you connect your "fun" concept to participation or victory. I'm not saying (now) that either is wrong, but they both exist with no in-between.

@Gevlon - bullshit. Dedication is the amount of resources you are ready to throw at a problem to achieve desired outcome. I am very dedicated of living, and all my fun comes from it. But if I catch something fun like Altzheimers or terminal cancer I will definitely prefer to not fight it, waste resources of me and the society, ruin my quality of life etc ... check any hospital ward with late stage patients to see where "dedication" and "fighting spirit" can take you.

There are some hypotheses which must be verified for the CLT to work, and it's not clear at all if they are.

Yes, but even if the distribution derives from a Gaussian distribution, it is still most likely to be mono-modal. You are falling into exactly the same trap again: "If the curve isn't a perfect normal distribution, it must be a strict digital yes/no distribution". That is simply not true, there are thousands of possible variants in between, for most of which there is a single hump in the distribution.

However "average dedication" is a nonsense. Dedication is binary, you are or you are not.

Says who? I can perfectly imagine average dedication, for example most people have an average dedication for their job: They do more than the strict minimum to not get fired, but much less than the maximum they could do.

You can also have situations where players are okay in easy dungeons but useless in harder ones.

Both Gaussian and Yes-No are terrible models IMO.

Yes-No fails for a bundle of obvious reasons, and Gaussian fails because raid performance is not the average of a set of independent parameters.

I think you are applying the terms and structures of one discipline to another, unrelated discipline. I would have thought a century of failure to predict human behavior accurately using these tools would have made this apparent.

Just as a "for example":

"people are notoriously bad at estimating their own IQ or other qualities, so that if you rely on self-assessment you end up with the observation that most people are above average, which is a mathematical impossibility"

Corollary: the average person, having thought about his or her standing in terms of intelligence, will act as though he or she is of higher-than-average intelligence even though this is not true. Therefore when marketing a product which requires some level of intelligence to use you will need to take account of the fact that the majority of your potential customers will think they have a higher facility to use it than they actually do have. In other words, your product needs to seem to be harder to use than it actually is.

If you market your product as "appropriate for anyone of average intelligence", by your own logic people's "inflated opinion of themselves" will lead to them treating this product as if it is beneath them. If they do decide to use it anyway they will do so while feeling they are slumming.

As for your contention that "Everybody who distributes people into good and bad players ALWAYS sorts himself into the good category" that's simply not borne out by common experience. It's entirely commonplace in-game to hear people discussing their abilities in terms that make it unequivocally clear that they recognize a hierarchy of skill in which they place others above them and below them. Many players are entirely able to recognize that they aren't good at something and that others around them are better.

I'm bloody awful at WvW in GW2 by any absolute standard and I'll happily acknowledge it. I can see other people performing much better than I do. I can also see that I'm not the worst and I can see that I have improved over time. I don't know if this kind of self-analysis is normative but it is certainly commonly observed.

But in games there are indeed two very distinct groups: those who CARE about the game and those who just play "for fun". The latter will perform horribly, simply because he doesn't read up, doesn't ask questions and doesn't listen to advices.

This seems pretty self-evidently wrong. I hate to rely on anecdata, but using myself and WoW...I don't care much. Outside of raids, I do not play the game. I do not do dailies, do not get rep, do not get valor points, do not get reroll tokens. I don't farm obviously, so during raids everything is handed to me by the guild. Repair bill, pots, food, flasks, everything.

I also perform consistently well in raid. I am certainly not an upper tier raider at all, not even close, but I am way above average at my role. I only really continue to play for the social interactions with my raid during raid times.

I've known people that were passionate about the game and played poorly, and I knew people who were extremely casual about the game and very, very good at it. All this is definitely anecdotal, but it isn't like you are showing any proof either. It seems clear to me that dedication to the game and "skill" are separate. They are definitely related, maybe even strongly, but aren't anywhere close to being perfectly correlated.

If you market your product as "appropriate for anyone of average intelligence", by your own logic people's "inflated opinion of themselves" will lead to them treating this product as if it is beneath them. If they do decide to use it anyway they will do so while feeling they are slumming.

Goodness gracious this is silly. You presuppose that people don't like doing things they see as requiring less intelligence than they posses. I've never come across that opinion anywhere outside of gaming. I love playing solitaire (Klondike). Not even the las vegas variant where 3 cards are dealt at a time, regular old one card dealt solitaire. I'm fairly sure that a sufficiently trained animal could play that game, it requires only the smallest minimum of strategy or intelligence possible. Why would I see it as slumming to play it? That's silly.

I'm pretty sure that people buy products requiring less IQ all the time. Just because you can understand Ulysses doesn't mean you don't enjoy watching American Idol in the evening. In MMOs some people raid, and some people fish, but I would never assume that the people who fish are those who aren't able to do anything that requires more intellect.

You've made a common mistake regarding the Central Limit Theorem. That theorem is about the sampling distribution for sums of large collections of independent random variables, not about the observed distribution of large samples of cases. To see the problem, consider flipping a coin and scoring one side as 1 and the other as 0. Then make a histogram of your results. Can flipping the coin any number of times ever generate a bell curve? It can't: it can only generate two values.

The IQ example also doesn't help: the IQ scale is calibrated to be normal, so its distribution is true by definition and not for any more interesting reason.

All of this means that we simply can't draw any a priori conclusions about the distribution of video game skill. Some real-world distribution ( e.g. having diabetes, or voting or not in an election) are dichotomous. Others (e.g. US public opinion about gay marriage) are continuous but strongly bimodal. Some (e.g. height) are continuous, single-peaked, and symmetric, like the normal distribution. Some (like wealth or athletic ability) are continuous, single-peaked, and strongly skewed. A distribution with a lot of skew is one where the median is quite different than the mean, i.e. most people are to one side of average.

We just can't know which of these patterns describes video game skill without getting way more specific about the concept, finding a way to measure it, and actually applying that measure to a large random sample. In the absence of such data, I can see no reason why video game skill couldn't be strongly skewed --- a lot of acquired traits are...

You misinterpreted my statement.

You said the rules system was fairly arbitrary, and that the players need to have a lot of trust in the GM to be fair.

My point was the rules system wasn't going to contain someone you couldn't trust because the rules system does not contain a GM who doesn't want to play by the rules. I wasn't talking about experience at all.

You misinterpreted my statement.

You said the rules system was fairly arbitrary, and that the players need to have a lot of trust in the GM to be fair.

My point was the rules system wasn't going to contain someone you couldn't trust because the rules system does not contain a GM who doesn't want to play by the rules. I wasn't talking about experience at all.

My point was the rules system wasn't going to contain someone you couldn't trust because the rules system does not contain a GM who doesn't want to play by the rules. I wasn't talking about experience at all.

You don't think that GM fairness is something which is at least partially based on experience? In my experience a GM can appear to be terribly unfair, just because he didn't know better and pulled some decision out of his ass without actually *wanting* to be unfair to some player.

Well, you guys are all obviously way smarter than I.

That said, Bhagpuss, I can't think of any examples where a product is marketed towards people of "average intelligence". In fact, I can't think of any reason why anyone would want to market a product in that way. If I wanted more people to watch "American Idol" (apparently the new gold standard in "average") I certainly wouldn't promote it with a "you want to feel more average?" line!

Also, in a way it seems that you helped reinforce Tobold's point. You clearly belong to a relativistic group that recognises that you belong in a complex matrix of skill/dedication/interests etc, whereas the group Tobold pointed to was the one containing the us/them individuals. I agree with Tobold: those people inevitably place themselves above the others: you're either with us or you're below us.

I think there's a huge difference between dedication and ability.
lets take gaming. I can be pretty dedicated. look up strats, stat weights, rotation, test practice.
and yet I will never be anything other than average. I'm aware of that. I don't care. I'm not trying to compete with other people, I'm not trying to beat someone else. I'm only trying to play MY best.

while different people may show different levels of dedication? most people are somewhere around my level give or take. neither terrible, nor amazing.

just average.

and btw? I click. I understand how keybindings work and I realize that they CAn make OTHEr people more efficient. however, they don't work for ME, personally. and contrary to popular belief on the gaming forums, keybindings will NOT turn an average player into a great one, anymore then switching from crayola pencils to faber castell would make someone a better artist.
its all just tools. tools are nothing without the person behind them, even if some tools can be easier to use than others.

When you say someone is being unfair, to me that means they are intentionally being unfair, perhaps by blocking the players from doing something the GM doesn't want them to do, or putting a thumb on the scales for players he likes more than others, or something in that vein.

As far as inexperience goes in applying the rules, no, that wouldn't really bother me. Honestly, I'd rate a GM who always applied the rules correctly to be a fair bit worse GM than someone who occasionally made somebody flip a coin, if everything else was equal.

Oscar--- as far as marketing to average people, they don't say "Are you kind of mediocre? Then try Pepsi!" but look for advertising that implies that everyone is buying this product, that this product is what normal people buy, this product will make you fit in, stuff like that. That is the the pitch to the average.

4c,

Sure thing, but that's marketing stuff to the average, not pitching it as "appropriate for anyone of average intelligence", which is what Bhagpuss mentioned.

Your Pepsi will probably sell, but I doubt we'll see Bhagpuss' pitch anytime soon. But I'd be happy to be proven wrong!

Athiev's comment is correct. To expand on it, your interpretation of the CLT is that if you flip a coin enough times, the probability distribution of each coinflip will begin to approach the normal distribution. As he noted, this is impossible, each coinflip remains discrete, 50% heads, 50% tails. What the central limit theorem says is that if you consider samples of 100 coinflips, one will have 60H,40T, the next 48H,52T, etc. If you plot this distribution for 100 samples of 100 coinflips each, this distribution will approach the normal distribution. If you run the math for the normal distribution, then you will see that for the sample size around 30 or larger, the distribution will be normal enough for everyday work.

So, in the case of gaming skill, let's take a measurement such as "average time before dying at donkey kong, the 1980's arcade game, measured as a mean over 10 trials." run this test for 100,000 people. there's no guarantee that the result will be normal, in fact I personally would estimate that it would not be normal. You'll have a clump of people in the middle, a clump much lower of people who are very old, or have never played video games, and a clump of people who love these games or other twitchy games off to the right. What the central limit theorem says is that if we analyze samples of 50 people at a time, then the means of each 50 person group, not of each person but of all 50 merged together, will be fairly normally distributed. The reason that the CLT is so important, is that this situation, where we imagine 100,000 people doing something, and how to draw conclusions about those 100,000 people by looking at samples of only 50 surveys, is a very common situation for people whose career involves analyzing surveys of 50 people from a population of 100,000. these people, knowing that the mean of these survey groups is normally distributed, can thus use the normal distribution to make estimations of the probability that their conclusions are representative of the larger group.

Now, if one is interested in deducing the actual shape of the distribution of the entire population from the survey sample of 50, the central limit theorem is not helpful. You would need to involve much more complicated statistical machinery. You'd also need an actual survey/sample, which you don't have here.

Applying the mathematical facts above to your post more generally:

"
Scientifically speaking that is utter nonsense. The Central Limit Theorem says that if you make for example this plot of video game skill of a large enough population, what you will get is a bell curve with a single hump in the middle. "

Well, scientifically speaking, what you've said is utter nonsense. As explained in my prior post, that's not what the CLT says.

"Statements about these two groups are usually made as if there was a clear distinction between them. In mathematical terms, if you plotted something like video game skill on the x-axis and the number of people having this skill on the y-axis, people talk of it as if the distribution was bi-modal; that is as if the curve would have two distinctive humps, one of good players and one of bad players."

I believe, though without any data, that "video game aptitude" is relatively normal. However, video game 'skill', which denotes more of a learned ability, seems much more likely to be strongly skewed. Take League of Legends. For most measurements of skill, the population of humans as a whole will be massively skewed: since most humans have never played LoL, they will be off to the left in a giant pile. Restricting the population to only people who've played LoL before, one would still expect a skew distribution: the 'time played' statistic is probably skewed, and if 'time played' correlates to skill to any high degree, the 'skill' variable will also be highly skewed. I of course don't claim that this correlation is 1, that would be inane.

Diablo 3 is a poor example to use for a developer failing to gauge average player behavior. It never had a full end-to-end beta test, like most MMOs. The only beta available to anyone outside of Blizzard contained no more than the first half of Act 1, so there was no meaningful public test of the AH before launch. How the average player would behave in endgame content was almost pure guesswork on Blizzard's part.

Random fact of the day. The overwhelming majority of people are above average with regard to the number of legs that they have.

Oscar said: "That said, Bhagpuss, I can't think of any examples where a product is marketed towards people of "average intelligence"."

Any product whose makers claim to be "easy to use" (or suchlike) is surely being implicitly marketed in this fashion.

I tend to express your title as "nearly half the people are below average." If memory serves, men asked to rate their sexual skills results in even more mathematically impossible results than intelligence.

I think there are two separate ways that I see "average" misused in forums:

1) selection bias: when most potential customers don't read forums and a small minority of those post, there is little justification for saying that posters are a reasonable proxy for the population. It is not that much of an exaggeration to say that Blizzard would be more profitable if they did the *opposite* of what beta forum posters suggested.

2) "average" is relative to a population. Most members of the World Cup/Superbowl football team are average members of the team. That does not mean they are average relative to the entire footballer population. If CCP deleted the accounts of the 80% "worst" EVE players, then afterwards most players would still be average even though that average skill level would be much higher than before.

Gerry Quinn,

I buy lots of those easy to use products so surely they can't be... hey!

This comment has been removed by the author.