www.Tobold.com
Tobold@GMail.com
A blog with my thoughts regarding games I am playing and other stuff in life. Please read my Terms of Service
RSS Feed
Tuesday, December 29, 2009
Survey
MMORPGs becoming more and more mainstream, there is now also more and more academic research performed on them. Thus once in a while I get students asking me to post a link to a survey they made, so they get a more players to fill out the survey, and can use those data to write a paper or something. I used to post all of those, but nowadays I first go through the survey itself to see how biased it is. Some surveys, unfortunately, are so leading, and show so much bias, that you could basically write that paper by just reading the questions.
30 years ago, when I was a teenager, I was jobbing at a research institute which specialized in the research of how to do surveys. They did opinion polls using survey forms in different version, and showed how posing questions differently could lead to different survey results. There is a whole science to asking questions that are neutral, aren't "loaded", and don't try to lead the participant down a predefined path. Knowing that, I only link to surveys which are well done, and broadly neutral, without giving away the conclusion in the questions.
A student from the University of Hamburg sent me a link to such a well-done survey on MMORPGs, which has a range of general questions about what you play, and what aspects are important to you when playing MMORPGs. It isn't very long, so if you have 10 minutes to help out a student with his work, I encourage you to do this. I promise it is not one of those horrible polls which reek of a prejudice towards some sensationalist subject like "MMOs make you addicted".
- posted by Tobold @ 3:35 PM Permanent Link
Comments:
<< Home
Newer› ‹Older
Done and done. Survey's are alwasy interesting when they ask the same question(s) multiple times with only a few word changes to make it appear different.
During poltical season in the US I'm always hammered with telephone polls, which, by the way, are the most leading surveys known to mankind. "Does the fact that Candidate X hates you make you want to support his health care bill more or less?"
I always ask them who's paying for the survey, if they don't answer I don't either.
During poltical season in the US I'm always hammered with telephone polls, which, by the way, are the most leading surveys known to mankind. "Does the fact that Candidate X hates you make you want to support his health care bill more or less?"
I always ask them who's paying for the survey, if they don't answer I don't either.
Done.
It seemed to me that some questions were asked repeatedly, only with different wording though. And the text on the end screen proved to me that yes, this survey is made in Germany.
It seemed to me that some questions were asked repeatedly, only with different wording though. And the text on the end screen proved to me that yes, this survey is made in Germany.
# posted by 
: 29/12/09 18:31
: 29/12/09 18:31
That repetition of questions with different wording is actually a sign of a good survey. By double-checking thus they make sure that the wording doesn't affect the answer, and make the answers statistically more relevant.
I love doing surveys but unfortunately I ran into a problem on the first question, which I feel is exactly the kind of biased, leading question you said this survey didn't have.
I play MMOs for anything from 20 to 40 hours a week and have done for a decade. It's been years, however, since there was a game that I "mostly" played. As a rule I play at least 3 MMOs about equally and maybe another 3 - 6 for smaller chunks of time. Currently I'm playing WoW, Fallen Earth and Allods Beta for something like 10-15 hours each per week, and NeoSteam, Dawntide Beta and Warhammer for maybe 2-3 hours a week each.
I can't foresee a time when I will ever again "mostly" play a single MMO, although there will no doubt be times when I play one more than the others for a while here and there.
In order to complete the survey I plumped for Fallen Earth, since I probably spend slightly more time there than the others, although it is hardly conclusive. Then I find out that the survey is not for online games in particular, but for the specific games mentioned only.
Since I am currently subscribed to WoW and have played it a fair amount over Christmas, I tried again, selecting that one. This time the survey rejected me on the grounds that I had already completed the survey!
If this is an example of a "well-done" survey, I'd really hate to see a bad one. You might want to forward my comments to the student who compiled it.
I play MMOs for anything from 20 to 40 hours a week and have done for a decade. It's been years, however, since there was a game that I "mostly" played. As a rule I play at least 3 MMOs about equally and maybe another 3 - 6 for smaller chunks of time. Currently I'm playing WoW, Fallen Earth and Allods Beta for something like 10-15 hours each per week, and NeoSteam, Dawntide Beta and Warhammer for maybe 2-3 hours a week each.
I can't foresee a time when I will ever again "mostly" play a single MMO, although there will no doubt be times when I play one more than the others for a while here and there.
In order to complete the survey I plumped for Fallen Earth, since I probably spend slightly more time there than the others, although it is hardly conclusive. Then I find out that the survey is not for online games in particular, but for the specific games mentioned only.
Since I am currently subscribed to WoW and have played it a fair amount over Christmas, I tried again, selecting that one. This time the survey rejected me on the grounds that I had already completed the survey!
If this is an example of a "well-done" survey, I'd really hate to see a bad one. You might want to forward my comments to the student who compiled it.
It was an interesting survey. I don't know if I'm the norm or not but I never try out a new MMO until it is at least a year past the initial release and some major patches have been rolled out to fix the bugs. I figure that if the game is still going strong a year later then it must be half-decent.
That repetition of questions with different wording is actually a sign of a good survey. By double-checking thus they make sure that the wording doesn't affect the answer, and make the answers statistically more relevant.
Or that the participant isn't trying to paint a too rosy picture of himself/herself. But for that to be effective, the questions need to be distributed throughout the questionnaire so that the participant forgets what he/she "should" answer. This one just had several two-page sets of almost-identical questions.
In any case, I'd hardly call this survey unbiased. They just pushed the hipster/tourist angle instead of the addict angle. I guess that counts as an improvement.
Or that the participant isn't trying to paint a too rosy picture of himself/herself. But for that to be effective, the questions need to be distributed throughout the questionnaire so that the participant forgets what he/she "should" answer. This one just had several two-page sets of almost-identical questions.
In any case, I'd hardly call this survey unbiased. They just pushed the hipster/tourist angle instead of the addict angle. I guess that counts as an improvement.
The mention of 'leading surveys' remind me of this excellent clip from one of the best sitcoms ever produced:
http://www.youtube.com/watch?v=3gMcZic1d4U
http://www.youtube.com/watch?v=3gMcZic1d4U
I hate any survey that asks the same question five times to make sure "you got it right".
yuck, closed browser.
yuck, closed browser.
I missed the question "Are you willing to pre-order and try a new MMO to see for yourself if it sucks totally, is fun shorttime or has longtime playability?". That would have been the essence of the 'new game' section for me.
Also, the country clusters miss out on Belgium, I guess 'the netherlands = BeNeLux'. =]
Also, the country clusters miss out on Belgium, I guess 'the netherlands = BeNeLux'. =]
Survey completed.
You raise an important point in our information age where surveys are often propoganda and marketing tools used to engineer opinions and product popularity.
You raise an important point in our information age where surveys are often propoganda and marketing tools used to engineer opinions and product popularity.
I have seen the repeated questions thing before - I guess its on eof the ways they validate the data.
What was all the "I like to be unique" stuff about. Is there a known link between mmo playing and wanting to stand out from the crowd?
What was all the "I like to be unique" stuff about. Is there a known link between mmo playing and wanting to stand out from the crowd?
Kudos to you Tobold for knowing the difference between a biased and an unbiased survey. And double kudos for caring.
Post a Comment
<< Home
