EEDAR’s Geoffrey Zatkin on the Value of Game Statistics

Fri, Apr 11, 2008

Learn how gaming statisticians are shaping the future in part one of our exclusive interview with the co-founders of EEDAR.

Geoffrey Zatkin has been a gamer his whole life. Weaned on pen-and paper role-playing games from the age of five, he wrote his first games on an Apple IIe, taught himself to program in college, and after happening upon a Usenet post asking for 3D MUD designers, spent six years building the worlds of EverQuest.

Today, Geoff is still a gamer – but he’s also the president of Electronic Entertainment Design and Research (EEDAR), a firm that is poised to revolutionize the gaming industry. For as little as a few thousand dollars, Geoff and co-founder Greg Short will evaluate your game – even from just a rough pitch document – and predict, with 80% accuracy, how many copies it will sell.

In part one, we ask Geoff how he went from game designer to ground-breaking statistician, and get a peek into the inner workings of the EEDAR system following the firm's eye-opening MI6 panel.

GameCyte: You mentioned you'd played games since you were five years old; when did you decide you wanted to get into game design?

Geoffrey Zatkin: In college it never really occurred to me that people made a living – or got paid, I should say – for designing video games. I suppose it should have. I was doing it for fun and a hobby; I was the equivalent of a modder back then. I modded an old Bungie game called Marathon, which was the precursor to Halo in a lot of ways. I used to mod levels for that and play them with my friends, and designed MUDs which were kind of like text versions of massively-multiplayer games now, and so I was a self-taught game designer.

GC: So after your career took off with these massively-multiplayer games, why the interest in statistical analysis?

GZ: Game development today – and don’t let anybody tell you differently – it’s a gut-driven thing. Almost nobody has any really good, hard data on what makes games fun. It’s a hard thing to quantify… fun is just hard to quantify, or write down on paper.

GC: Hard to define, even.

GZ: That’s a very accurate statement. So every time I started a new game, I’d ask around and say, “Do we know who we are building this for? Do we know what they think is fun?” I know what I think is fun, I know what I like in it, and I hope that lots of other people like what I like, but you can’t always be working on a game that’s the exact thing you want to build.

GC: There’s that… inherent bias in the development team.

GZ: If you’re making a My Little Pony game, you should be making the damn best My Little Pony game you can, for the people who are going to enjoy that. And as a professional designer or professional design team, or a publishing team, your taste really shouldn’t matter. You’re making it for the group who is supposed to enjoy that property or that kind of game. And so you don’t always have the luxury of saying, “Oh, I’m making the exact type of game I would play.”

And that’s where starting to look for real data about what people playing those kinds of games liked -- and you know, surveys are really good at gathering opinions, but it’s hard to get a really large group to do a survey. Another one of the big problems with surveys is if you ask somebody, “Do you like X,” they’ll say “Sure!” If you say, “Well, would you be willing to pay $60 for that,” they may say “Oh, maybe.” But if you say, “I have it in my car, give me sixty bucks and I’ll get you a copy,” that’s a whole different response.

GC: ‘laughs’

GZ: So, a little bit of what we’re looking for at EEDAR, with all this data collection, matching it up to things like NPD and Metacritic, is not just seeing what people thought, but what they went out and purchased, and being able to break that down on a feature by feature level.

GC: We didn’t see much of Metacritic in the panel today. How do you interface with Metacritic?

GZ: We use them as one data point, out of the 15,000 data points we look at per game. Metacritic’s score is one point in there, same as ESRB rating is a point, type of multiplayer is a point, style of mini-map is a point, use of fog of war on the mini-map is a point; there are thousands and thousands of points that we look at regarding what a game could have in it, and we’ve classified them…

There’s a lot more and a lot less than you’d really think, which is a strange way of saying it, but if you think, rhetorically, about how many things could end up in a game, you might say “Wow, there could be an infinite variety.” Well, there’s not an infinite variety; patterns repeat, and there’s probably 10 or 15 ways of doing a mini-map and even those really boil down to two or three common ways.

GC: Could you illustrate for our readers?

GZ: Take something like maps. There are lots of maps: There’s a world map; there’s the game map which shows you your whole level; there’s your mini-map, which usually shows you what’s around you… some games have different variations, so on a shooter game your mini-map might show your immediate surroundings; on a RTS it may actually show the entire map of the level because you’re not character-centric, you’re usually multi-unit-centric. These are interesting data points. What would it be like to have a shooter where your mini-map was the map of the level? That’d be a lot different. I’m not saying it’s good or bad, but it’s a different point.

You put things like fog of war in there – there are different ways of doing fog of war. Is it radius-based? Can you only see what you see? Can you see what your allies see? Can you see what your friendly units see? When you see a structure and then you move away, does it stay on the map as kind of a silhouette until you go back and look at it, or does it immediately disappear? How much longer after something leaves the fog of war can you still see it? These are all specific things that games have done in the past.

GC: When did you start thinking of these game elements as data points?

GZ: Greg [Short, Chairman of EEDAR] thought up the idea a couple years ago, because he and I had both watched some of our friends’ games turn out to be good games, but because they didn't have the right information at the start of development, they had to spend a long time reworking different elements of their game four to five months from launch, when they finally got users who could look at it and perform real focus testing.

That’s the hard way of doing it – because if you have a two year dev cycle, and you have a game that’s really only playable six months before launch, you spend 18 months working blind, not knowing what the consumer response is actually going to be like. You’re putting in all of the stuff you think is going to work well, and a more experienced team will make more of it work well sooner, but you’re always going to have rework, because no one is ever completely right – no design team, no management team, no marketing team.

GC: So what makes you right—that is to say, what makes your collection of data right?

GZ: So at this point we’ve looked at over 1500 games, and we track up to about 15,000 pieces of data per game; we have over 1-point-something million data points at this point. Obviously, not every game has everything it possibly could. So we have this massive data set – how much money did each one make, what was the Metacritic each one got, what time of year did they launch, what kind of advertising did they do – we look at all these points together and we run what’s called a regression on it. We start comparing them against each other.

When you come and say “I want to know how a game with this set of features would do,” we can say, even from a rough pitch document, “Oh, that combination of features together, if we put it with a historical model knowing how much all of these other ones made, and at the Metacritic level you think it will come out at, are going to roughly equate to this many hundreds of thousands of units being sold,” and we’re surprisingly accurate.

We have over an 80% accuracy rating – from a pitch level, multiple years out – and the 20% where our predictions are off tend to be games that sell less than 30,000 units (because we know it will sell less than 30,000, but it’s hard to know how much less) and games that sell 800,000 and over, because we know it will be over that, but there have been so few games statistically that sell more that it’s hard for us to know how much over.

GC: How fine an estimate can you give?

GZ: We’ll give you an exact prediction down to the single unit – the closest we’ve ever been is 300 units.

GC: When making purchasing decisions, consumers often consider other high-profile titles being released around the same time. How do you integrate this factor when creating predictions?

GZ: As strange as this might sound, it’s all cyclical. If you’re releasing a game in September, you can be pretty sure that there’s always going to be X amount of other high-budget games coming out in the same month. It’s surprisingly close almost every time, and so we can build a model that says “Oh, if you’re releasing in this month, on this platform, in this genre, here are the other close competitors that you’re statistically going to have, and here are the ones that are going to be so good that regardless of platform or genre people will be thinking of buying them anyway.”

We’ve programmed in ten years of historical patterns, and things shift… but they don’t shift that much.

GC: You say ‘programmed in;’ what are you running on?

GZ: Proprietary. We build everything we do. It requires a lot of computing power, but we don’t need a dedicated room full of servers. That’s partly because we have a very well engineered series of regressions. It’s actually a combination of regression and a neural network. It learns from its past to help make future predictions.

GC: What kind of investment was needed to get this all started?

GZ: We’re angel-funded all the way through, and have been going for close to three years now; it took a year just to get the classification system started. It’s a very nimble system. We know there will always be new things, and so we have the ability to rearrange it on the fly, plug in new nodes… it’s not like it’s a static industry. It’s pretty badass, if I do say so myself.

GC: I’m with you there.

GZ: I’m used to building tools that have to support communities of hundreds of thousands of players over time, and have to always be working and running correctly, and that’s what I’m good at – and so that’s what we’re doing.

Follow this link to part two of our exclusive interview, where we add EEDAR chairman and co-founder Gregory Short to discuss EEDAR's MI6 panel findings and the viability of statistical analysis for the gaming market.