Yesterday, video game retailer Game Crazy released the results of a recent market research survey which claimed 90% of kids ages 8-17 expect to ask for a video game this holiday season, and that music games like Guitar Hero and Rock Band topped their wish lists, followed by Mario Kart. The story was picked up by the likes of Gamasutra, GameDaily and CNET’s Crave.
Though we don’t doubt games will be incredibly popular this holiday season, we have reason to believe Game Crazy’s study was flawed.
As repeat readers are no doubt aware, we’re none too trusting when it comes to video game research. Why?
Well, among the fact that most studies are paid for by an organization that stands to benefit directly from biased results; the fact that most would-be game researchers know so little about games that they write inherent flaws into their own study; and my personal favorite, the general failure of game researchers to adhere to basis statistical principles of sampling (in a nutshell, a study that surveys white, middle-class, social-science-studying college kids cannot say anything about anyone who is not) most studies “prove” only that someone took a lot of trouble to run numbers through a computer.
So when I stumbled across the Game Crazy study yesterday morning, I was rather excited. Here was a piece of research funded by a company which might not have a direct interest in the results (Rock Band or Guitar Hero, it’s all the same to a retailer) and which, for once, prominently displayed signs of proper sampling and statistical significance in the press release.
The Game Crazy survey was conducted online August 17-31, 2008, by Weekly Reader Research. One thousand U.S. male and female participants, ages 8-17, were recruited from Weekly Reader Research’s INSIDERS survey research community. Random sampling procedures were employed to ensure the respondents accurately reflect the nation’s 42.7 million 8-17 year olds in terms of age, race/ethnicity, gender and census region. The margin of error is +/- 3.1 percent at the 95 percent confidence interval.
In other words, if Weekly Reader Research managed to conduct an completely unbiased study, the results would most likely be accurate for the entire country. You’d be able to say things like “9 out of 10 kids want video games this Christmas” without batting an eye. So, determined to find out if anything would inadvertently bias the study results towards games and/or music, I had a look at Weekly Reader’s website, where I was greeted by:
Games.
And music.
And more games, and more music. (And a Macbook, and stuffed animals, yes…)
What I didn’t find, though, were surveys. I discovered that while the above page is one of several locations from which survey respondents are recruited, the actual surveys take place at a different site, WRInsiders.com. However — for purposes of insuring an unbiased survey sample – what I found at WRInsiders wasn’t much better.
Described by Weekly Reader Research itself as a pair of portals “based on video game metaphors in which kids earn points for answering questions,” WRInsiders gives kids a chance to convert answers into prizes (around 1/3 of which are video games); play a variety of flash games; or go back to Weekly Reader and play games there.
Unless the kids that visit WRInsiders do so because of deep-seated humanitarian ideals (which is possible; 10 of the available prizes are charity donation) it’s probably safe to say that they’re in it for the games — which could make their survey responses biased towards games, and that 90% figure that much less likely to apply to every kid in the USA.
We wanted to give Weekly Reader the benefit of the doubt, though… so we called them up. This morning, I spoke to John Fredricks, director of market research at Weekly Reader and their parent company Reader’s Digest.
Fredricks told me a variety of wonderful things about WRInsiders — how they’re 100% COPPA compliant, how they receive all their new members via word-of-mouth, how they automatically filter candidates to make sure that children don’t become “pro respondents,” how this particular study had a huge sample of 1000 individuals and none were “prequalified to be gamers” — but after looking at the questionnaire kids actually filled out online, none of that could convince me that the study was any less inherently biased that I’d originally feared.
First, though Game Crazy is calling this study the “Holiday Gift Tracker Survey,” the questionnaire is, as you might expect, entirely about video games. The press release line “Parents and grandparents can expect video games to be on the top of children’s holiday wish lists this year,” is entirely misleading, as the study did not survey children’s desire to ask elder generations for, say, stuffed animals, pocketknives or the latest Roomba.
Second, question 11 — the answers from which Game Crazy derived that 90% figure — was worded like this:
11. This holiday season, how likely are you to ask for at least one video game?
Very Likely
Somewhat Likely
Somewhat Unlikely
Very Unlikely
The problem here is not with the question, but how the answers were counted. Fredricks told me that both “Very Likely” and “Somewhat Likely” were counted towards that 90%. You don’t need to be a statistics guru to see how kids who might have only been considering the idea of asking for a video game would be roped in.
Third, though the full list of games kids could designate as their holiday pick did indeed include the violent Gears of War 2 (as well as Need for Speed Undercover, Lego Batman, Sonic Unleashed and “Imagine Series Game or Petz Game”) as one of 10 available options, several earlier questions may well have primed kids to choose the games they did:
18. What video game accessory can you not live without?
Battery Pack
Car Charger
Racing Wheel
Online Points Card
Guitar
19. What video game accessory do you wish you had in real life?
Wii Dance Mat
Guitar Hero Rock Guitar
Wii Stearing Wheel
Rock Band Drum Set
28. What video game would you like to give Santa?
Mario Kart-if he can drive a sleigh around the world in one night, he can handle these tracks
Wii Fit- extra sugar cookies means he could stand to lose a few pounds
High School Musical Spotlight World Tour- check out the vocals on the world’s best gift giver
Guitar Hero- the idea of Santa rocking out is too amusing to pass up
29. What video game do you think Santa is most likely to give Mrs. Claus?
High School Musical Spotlight World Tour- for the budding North Pole singer
Wii Fit- she also enjoyed her sugar cookies a bit too much this year
Petz and Dogz- for the animal-lover in head of the household
Rock Band – to see Mrs. Claus rocking out on the drums
It’s worth noting that neither High School Musical Spotlight World Tour nor Wii Fit were on the list of games kids could choose from.
Fredricks told me that the survey questions were were not composed by Weekly Reader, but rather passed along by Cone PR, who served as an intermediary for Game Crazy. Weekly Reader did, however, have a role in recommending changes to the questions. Asked about potential fallacies in the study and if there was anything that could be improved, he told me “No, I wouldn’t change anything here.”
We doubt this study was intentionally biased in any way — like I said, Game Crazy could care less whether kids pick up Guitar Hero or Rock Band — but for the purposes of advancing video game research we hope others will learn from this study’s potential failings.
Leave a Reply