The EEDAR Interview, Part Two

Mon, Apr 14, 2008

Join GameCyte for part two of our exclusive interview with the co-founders of Electronic Entertainment Design and Research (EEDAR).

Last week, we spoke to EEDAR president Geoffrey Zatkin about how statistical analysis could shape the future of gaming. Today in part two of this GameCyte exclusive, we introduce Geoff's partner and co-founder, EEDAR chairman Gregory Short, and discuss statistics, genres and classifications, the Nintendo Wii, and the findings both presented in their eye-opening MI6 panel.

At the MI6 conference, Greg Short was all business. His Australian accent surfacing only occasionally as he helmed a panel presenting hard statistical research to an audience of marketers and game industry professionals, Short possessed a cold, no-nonsense quality one might well attribute to a leader of men; a quality that continued to show as we directed pointed inquiries into the statistical basis behind the panel's findings.

But with the right question, GameCyte discovered Greg's humble fanboy roots as an enchanter named Baelish -- and a mirth that belied his professional demeanor.

GameCyte: Now that we've got you both together... We noticed both of you worked at Sony Online Entertainment, but not at the same time. How did you meet?

Gregory Short: Now this is actually a really good question. I was actually the founder of a very large Everquest fansite called Caster’s Realm back in 1999-

Geoffrey Zatkin: Which originally started out called…

GS: EverQuest Enchanters, because I was an enchanter, and was having trouble finding information on enchanters at the time, and Geoff was the lead spell designer and magic system designer for EverQuest, so we got basically talking… I would ask him questions about spells and the way the magic system worked, and he would write me back, “I’m very very busy right now, and I’ll try to get to you as soon as possible…”

‘all laugh’

GZ: Well that’s not entirely true! I gave you full responses and he posted them verbatim on his website, which gave us an outlet where people could actually get factual stuff, because we didn’t have any idea of doing that ourselves, and he soon grew that to all four mage classes, to every spellcasting class, and was soon running a whole site on that.

GS: We were one of the first sites to ever have character profiles online, we were one of the first sites to ever have a lookup-able spell system online; at that time it was us, Stratics, and Allakhazam, and we basically defined modern fansite networks for massively multiplayer games.

So Geoff and I have known each other now for over ten years, and we’ve talked a lot about how to improve the industry during that time, and I feel great that we’ve been able to work together and apply each of our different perspectives on the gaming industry. Geoff really comes from the development perspective and knows how to make development work better, and I come from the business and operations side, and I think togther we provide a comprehensive product and service which really does meet the needs of the industry at a time when getting more cost effective, being smarter about the decisions we make, and dealing with far more discerning and demanding consumers than ever before, means that we really need to go back and start making more informed choices.

GC: Today, you released data and recommendations of a sort for the marketers here at MI6, on achievements and their diversity, downloadable content and digital storefronts. What would data like this be worth if companies wanted to buy it on their own?

GZ: What we released today is aggregate data; it’s really good information to start with, and we’re quite happy you can use that and do well, but for a couple grand we can give a game publisher an exact breakdown of what they’re doing, and how to monetize it the best.

GC: So you’re in the business of selling these personalized reports-

GZ: We try to sell solutions. If people come to us with a question – and everyone asks the same question – we have a ton of research on this, and we can give them something, fairly cheaply, that will help them monetize their game far beyond that. And the earlier you start, the easier it is to engineer into the game. Anyone who’s done any programming will tell you, making changes at the beginning is a very easy thing to do. Making changes after a year and a half of work, when all the systems are built on top of one another, is much harder, and in some cases logistically impossible.

GC: But speaking more generally -- you've proven today that you're more than capable of projecting larger estimates. I wonder, how can [Wedbush Morgan Securities Analyst] Michael Pachter continue to do business when research firms like yours exist?

GS: What we provide are facts. Analysts right now use facts to be able to put their predictions together. But analysts look at things in a very broad perspective – they look at entire portfolios of companies, how the industry is moving as a whole, things of that nature.

We see ourselves actually in a good position to help them with their job. The truth is, we exist to help people make better games – to help them make more informed decisions in the game-making process. That’s where we’re targeting. I believe we can help analysts justify some of their claims and also make more accurate forecasts as data suppliers to those in the industry. Analysts by their very nature are giving a subjective opinion, and there’s a very easy line you can draw there. We’re looking at being the best provider of objective quantifiable data for the gaming industry, and then using that data to make projections, so people who are analysts can make their opinion, and do so on solid footing.

GC: How can you objectively quantify something like a video game?

GZ: We only classify what we call “objectively quantifiable data” – things that have no subjective nature to them, and if we see something that looks subjective we will write a rigid definition to make sure that we can identify it time after time, because we have multiple values looking at games. Every answer they give has to match, on each one. We can’t have some guys thinking it’s 'this,' and some guys thinking it’s 'that.' At that point it’s a subjective call and our data would be inaccurate.

GC: Internal validity.

GZ: Yeah, and so we check it; and then we have a different team check it; we have statistical programs that look for out-of-bounds answers. To give you an example, one thing we had been looking at but just removed was the style of names that accomplishments use.

One of the little checkboxes said, “Are these humorous?” And we had someone go back and recheck and they said “Hmm, what I think is humorous isn’t what this guy thinks is humorous.” So we realized that we’d put in a subjective item, and yanked that whole little section out, because you can’t judge if a name is humorous or not. That’s subjective.

GS: We changed it to whether the name was obvious or non-obvious. One of the Call of Duty achievements is Hot Potato; you get it for throwing grenades back five times. It’s an non-obvious name for an achievement, whereas “Throw Grenades Back x5” is an obvious one. “Collect 100 Flags.” One of the analogues we’re looking at is if an achievement has a non-obvious name, does it help drive sales differently than achievements that have obvious names?

GC: I have to ask: have you found any correlation there?

GS: We haven’t run any regressions on it at this point in time. These are obviously interesting questions and interesting facts, and they are things that may or may not really matter at the end of the day, but we’re doing our best to go out and try to help marketers, developers, producers, publishers, investors have somewhere to go when they have a question like that, and to provide answers which they can consistently feel are based on high-quality objective data, and not based on, purely at least, personal experience.

GC: You mention publishers; What do you feel you bring to the industry that isn’t possible for publishers, particularly large ones, to generate on their own?

GZ: Independence. We’re coming in as an outside group. We have no emotional ties to the product they’re developing, we don’t have any ego resting on it, it’s not our pet project, it’s not our boss’s pet project -- we have no real agenda. And because of that, we can provide that objectively quantifiable data, and let them make a decision based on that.

GS: Sometimes you come across people are adamant that their game is a particular style of game. They might say “Listen, this is a hunting game. It is a sports game. This is for the sports demographic.” We classify hunting games as-

GZ: They’re not a sports game. For us, a sports game is something that has usually a field, usually a team, a score, it’s timed in some manner – hunting has none of those things. You can’t compare a hunting game against a baseball or a soccer or a football game; those are games with something in common.

GS: Independence is huge, but we don’t only provide that: we also provide a specialist level of detail which is unfeasible for someone who is not just focused on that in their day-to-day job. For large publishers to basically sit there all day and play every game that has ever been released and catalog it and classify it, that’s not their core business. Their core business is making games. For them to build all of that out, and how long it would take them to develop that when they have us available today and they can leverage that knowledge immediately, allows them to focus their resources on what they do best, to make good games

GC: Back to genres for a moment, I'm curious: how do you detail with the commonly identified genres of action, adventure and platformer?

GZ: We completely rewrote them. This was one of the very first things we classified – every single major publisher and every single major news website had a different genre classification system. None of them matched. So we took a long time, took down every game we could think of, and put them into the equivalent of big buckets. We came up with 13 genres, plus sub-genres, that we felt compared actual games with features and styles of play that were like each other and that we’d always feel comfortable comparing to each other.

As an example, our extreme sports games now – take the latest Tony Hawk – despite being called a sport, it has very little in common with Madden. It has a lot in common with the latest Grand Theft Auto. Big open world, environment to play in, different things you do in different places, quests you pick up along the way to prove yourself to unlock other things… there are some very obvious differences, but there are a lot of similarities when you look at how it works, as opposed to the trappings of style that you put on top of that. And that’s something that, having been a hardcore game designer for a long time, I look at that and ask "what does it actually do," as opposed to "what are the special effects on top of it?"

GC: Say there’s a new game released today. How long would it take you to evaluate it?

GZ: It really varies, because there are some games that are relatively quick to do, and there are some games that are multi-day to multi-week projects. Wii Sports, for example, didn’t really take us that long, because it was fairly easy to see what they are doing with each segment of the game.

You take something like BioShock, which unlocked different mechanics at different times in the games – photographing things to determine their weak point came in about halfway through the game, you couldn’t start modding your body till about a third of the way through the game, you unlock different features different parts of the way through, and it was a fairly rich game considering the different types of features that it had.

GC: And the boss fight at the end was completely different from anything else in the entire game…

GZ: Uh-huh -- that kind of game takes us a lot longer. Sadly, there isn’t an exact time per – they take a lot longer than you would think because we want to make sure we don’t get things wrong.

GC: Are we talking days… months…

GS: It’s never months… but we haven’t fully tried to classify an MMO yet.

GC: One of the common misconceptions when dealing with statistics is that correlation (a set of coincidences implies casuality (the idea that one thing actually causes another). We were very pleased when you mentioned that although your data shows that games tend to sell better when they have more diverse accomplishments, it’s more likely that’s because well-thought out titles have more diverse accomplishments to begin with.

However, you also admonished the marketing audience to take advantage of accomplishments, and viral accomplishments in particular, without providing casual data. On what do you base that suggestion?

GS: Obviously we showed correlative data that showed those games with viral accomplishments sold about 28% better than those that don’t. The admonishment was only in the sense that it doesn’t seem marketing is getting an opportunity to be planning accomplishments. It was hopefully an inspiration for them to go into meetings and become more involved, armed with real reasons that consumers are being impacted by these things.

To your point, I said that they should put more reward points against these accomplishments because right now less than 1% of reward points are associated with viral marketing, and it’s not much of an incentive for you to go out and do it. If you say “I’ll give you an empty cup of coffee if you get your friend to buy a coffee,” I’m not so excited to do that. If you say “I’ll give you a free coffee” well now, I’m motivated. It’s all about finding the right tipping point where someone is motivated to do the action you’re looking for from a psychological perspective, and marketers understand these concepts implicitly. What we’re simply pointing out was that they can apply a lot of the knowledge and traditional marketing principles they already have, to things like achievements and downloadable content strategy, and have greater success.

GZ: We took every single game that had accomplishments to find that 28% on the viral, and for almost everything you do in video games you’re never going to be able to prove causation because there are so many independent variables out there that you cannot perform a controlled study. You’d literally have to release the same game with minute variations on all these different levels… anybody who tells you they have casual data is probably pulling your leg.

But we can point out very strong correlation and say the trends most likely suggest – 28% is a large percent, across a large volume of games –

GS: A correlation which shows something of significance is still something you should pay attention to. Causation is obviously the holy grail of everything: if A+B=C and I can put those together consistently then great, and maybe over time we will get to the point where we have enough data in certain specific instances to say, with validity, causation is this. But the truth is as a company that specializes in data, unless we’re certain, and it’s a law, we’ll stick with correlation over causation.

GC: In October, you told Gamasutra that you were working on causation. Has any progress been made?

GZ: We’ve been researching things for a couple years at this point, and it will probably take a couple more before we get to a significant enough number that we could ever say causation. I’m always going to hedge my bet on that, because our whole company principle is we’re going to give you the best information possible, and assumptions that say “watch out for these couple things,” and as we can eliminate “these couple things,” we’ll be able to say just “watch out for this.”

GS: And the truth is that what may be causative today might not be causative tomorrow. The gaming industry is constantly evolving, the consumer is constantly maturing, and the experiences we’re having are shifting. We will always do our best to provide the highest quality data, and point out the significant correlations that exist in the market at any given point in time, but it’s difficult to claim that, forevermore, the causative approach is A+B=C.

GC: Speaking of the Nintendo Wii in particular, we noticed that the Wii was kind of left out of your discussion of achievements and DLC because, well, it doesn’t have any. Would you recommend that Nintendo try to catch up, and if so, how should they go about that?

GZ: Yes.

GS: Obviously yes, it can significantly improve your sales. Three things to keep in mind. Look at achievements, look at Club Pogo, look at Whirled, look at Kongregate. Look at these casual sites that kids are flocking to. They all have achievement systems, badge and such that you earn for whatever you do.

GZ: It can’t help but help. Short of them screwing it up massively, which is doubtful Nintendo would do, it cannot help but add stickiness and fun to the system.

GS: And given the consumer they’re attracting, meaning non-traditional, more mainstream, older and younger demographics – I really think that type of interactivity would add a different dimension to their system. The Wii’s a virtual arcade in a box these days because of all the games you can download, and I think it only stands to reason that with Wiiware coming out, they’ll have downloadable content in the near future. The only drawback for them currently is hard disk space.

GC: Lastly, with this tremendous wealth of data you’ve accumulated, what’s next for EEDAR?

GZ: We have a long product roadmap, most of which we can’t talk about yet, because we haven’t announced it, but everything we ever do will revolve around the fact that we have a really large database, and are trying to get that to people in ways that will help them. That is the core of our company – we are supplying data to help people make smarter decisions earlier.

GS: We have enough projects to keep our engineering team busy for a very long time, and we’re starting to form some wonderful relationships with our partners to do some very exciting things, and those will be announced throughout this year.

GC: I want to thank you both very much for your time. This was incredibly valuable.

We'll have more on EEDAR's projects in the near future - watch this space.