We go behind the scenes at Valve, Bungie, and Epic Games to see how user feedback helps shape the game development process.
Inside a dark building high above Seattle, a group of gamers wait patiently to be strapped to a voltage battery. Thin wires peek out from under their sleeves as tiny round contacts are stuck carefully to each of their hands. Somewhere down the hall, a PC registers the first electrical pulse, rippling across the skin of one of the candidates. Whoever he is, his nerves have just given him away.
The playtesting rooms inside Valve's headquarters resemble a large, untidy science lab. PC monitors line the wall, spewing out steady streams of graphs, patterns, and charts. Some keep track of candidates' heart rates; others record their eye movements. A pile of resistors and voltage batteries lie discarded in one corner of the room; candidate information reports, video recordings, audio transcripts, written questionnaires, and the results of thousands of hours of direct observation lie in the other. Valve takes its playtesting seriously.
While playing video games for a living may sound like the dream ticket for any avid gamer, for those involved in the process, it represents much more than just another routine part of game development. Testing games before they are released gives game developers a rare insight into the end user experience, helping them validate the quality of the game and isolate potential problems. We now see games as much more than simple products; we see subjective experiences that affect individuals in different ways. For this reason, video game testing not only has to be more rigorous and pervasive than product testing in other industries, but also more precise. Can feelings be measured? Why does one player enjoy a particular game, while another does not? How can user feedback be used to make games better?
Publishers and developers are constantly seeking answers to these questions, trialing a variety of playtesting methods to get the best results. In this feature, GameSpot will go behind the scenes of three major studios--Valve, Bungie, and Epic Games--to find out how tools like science and psychology are helping game developers better understand the nature of player behavior.
Three years ago, Valve hired experimental psychologist Mike Ambinder to head up the playtesting department of its development arm. With a B.A. in computer science and psychology from Yale University and a Ph.D. in psychology from the University of Illinois, Ambinder was the perfect man to lead Valve in a new direction in the field of user research. Using his background in visual recognition, Ambinder began treating Valve’s playtesting sessions as a series of scientific experiments designed to test various game design hypotheses. Since most psychological research revolves around trying to isolate mechanisms of behavior to find out why people are motivated to do the things they do, Ambinder thought game playtesting should strive to do much the same thing: use extractable data to make an altogether better user experience.
"For us, playtesting is the most important part of the game development process," Ambinder says. "It's not something we save for the end of the development, or use as a quality assessment (QA) or balancing tool. Instead, it is the dominant factor that shapes our decisions about what to release and when to release it."
This is a relatively new attitude for the games industry. While playtesting has always been a vital part of the game development process, the role it plays in shaping the final user experience has never been as important as it is now. Traditionally, playtesting methodologies focused on video games as products rather than as variable experiences that can affect players in different ways. Even as video games became more complex and publishers began to employ dedicated quality assessment teams rather than simply having the development teams playtest their own games, user feedback practices remained fixed on highlighting problems surrounding the more objective aspects of game design--coding errors and bugs--rather than exploring the experience of what it's like to actually play the game. Methodologies like functionality testing (looking for general problems in the game's overall design), compliance testing (checking that the game complies to publishers' technical and legal requirements), compatibility testing (testing the game on various configurations of hardware and software), localization testing, public beta testing (which lets users pick up any errors the developers may have missed), and regression testing (testing to make sure previously reported bugs have been eliminated) have become industry standards, with publishers and developers mixing and matching various methods to suit individual project needs.
But things are changing. The growth of gaming audiences and the subsequent push towards more diverse gaming experiences has led some publishers to rethink traditional playtesting methods in favor of something a little more relevant. According to Ambinder, the games industry is starting to move towards more innovative ways of gathering data, willing to spend more time, energy, and resources on its accumulation.
"I think more and more companies are starting to see the value in hiring folks with backgrounds in psychology or related fields that provide skilled training in extracting meaningful data from playtesters," Ambinder says. "For us, playtesting is crucial, as it is the most effective and honest means of validating our products. We would be foolish to release a game that went through minimal playtesting, as we could have little confidence in its quality had it not gone through rigorous testing. To that end, we start playtesting as soon as we have something playable, and we basically never stop as we are constantly updating our products after shipping."
When developers first became interested in user research some 10 or so years ago, the standard practice was not to waste too much time on it; therefore, only the first hour of the game would undergo outside playtesting. Things are a little more serious these days. Most publishers would consider it madness to release a game that hasn’t been subjected to hundreds of hours of rigorous playtesting, combing over every single part of the game right up to its release. A decade's worth of knowledge has come down to one thing: perspective.
After spending years as a playtester himself, Epic Games senior game test manager Prince Arrington saw the value of perspective in the work of those around him.
"I always got a kick out of giving feedback to developers and then seeing my suggestions come to life in-game. But sadly, it has the tendency to be one of those things that can easily be undervalued. It's very easy for us, as developers, to get too close to our projects and fall into the trap of not realizing that our baby isn't perfect. This often leads to poor design remaining poor. The value in having outside feedback is that it's always nice, if you're open to constructive criticism, to get those checks throughout the development cycle so that you can get the confirmation that you're making something kick-ass…or otherwise, the painful realization that you're not."
Just like Ambinder, Arrington's professional career is rooted in psychology. Earning his psychology degree from North Carolina State University, Arrington left academia to become a contract tester when his father told him that he played games so much that he should probably start making them. During his three years as Epic's QA manager, Arrington has come to see playtesting as a valuable tool in keeping developers in check.
"Playtests are as much a part of the development cycle as design meetings or code reviews," Arrington says. "While different in execution, it's something that should be done daily, with different combinations of participants with varying skill sets willing to give constructive criticism, with the sole purpose of making the game as good as humanly possible.
"Developers work on a project for so long, there's always the potential to lose perspective on what's working and what's not working. Without this form of genuine feedback, developers have a tendency to drink the Kool-Aid and become accustomed to the inefficiencies and flaws of a project, which leads to a failure to explore more suitable paths."
This perspective is often offered by people who have never played video games before. Both Valve and Epic aim for a wide demographic when deciding whom to bring in, from seasoned gamers, to people who have never played a shooter before, right down to non-gamers. With the gaming audience growing every day, developers have come to understand the value of reaching out to all skill levels, fighting to keep existing audiences while trying to snare new ones.
"We have a sign-up page for playtesters on Bungie, and we bring in a whole range of people: people who are experts, people who play casually, everyone," says John Hopson, the user research design lead at Bungie. (He too has a psychology degree). "For Halo testing we even bring in people who have never played a shooter before. It's painful to watch because they have a lot of trouble, especially with things that we don’t even think about anymore, like coordinating the use of two thumbsticks or knowing where the buttons are. But Halo is supposed to be a game that is fun for everyone, so it's necessary to see how people who have never played a shooter before react to it."
Arrington says if developers lean too far in one direction they risk skewed playtesting results, ones that don't take into account information from players who have the potential to account for a large portion of the game's user base.
"If you fail to account for players new to your game or genre, it's typically those players that will stop playing, or avoid your game, if the barrier to entry is too high. So this is very important for making the game easily accessible to the casual gamer as well as challenging enough for the veteran, hardcore gamer."
Laura Parker is the Associate Editor of GameSpot Australia. She loves adventure games, sparkly stuff, Trivial Pursuit, cake, Master Chief, earthworms, and rhetorical questions. She once stole a sandwich from Peter Molyneux.
great article! it is pretty clear that Valve (quite typically and admirably) is going into more experimental testing strategies, but as a result they are likely missing the mark. Games like L4D would have benefitted from a much more human approach to playtesting--that being said, I still love their games!
Amazing article! I enjoy reading about this process very much, as it is something that is lost often in every day discussion of games, but is equally as important to most game studios and should be to most gamers as well.
@m4a5 i realize it is a job and not playing games, but in a certain manner i would still be doing what i like. Besides, you have to like what you do for a living and it would be great for me to help make good games, my dream is being a game developer.
@Morphine_OD I have done my homework enough to know that. But that really doesnt say much for the company eh? Why make a sequel to a GOTY and critically acclaimed game on a crappy old buggy engine.Just says how much they gave a crap about it.
yet crappy ass games with bad camera angles or poor framerates for example still get released.... to call it a science is a slap in the face of science...
They should do an article on the science of advertising since game companies spend much more on that than on playtesting. Heck, some companies spend more on advertising than they do on the making the game.
One thing's for sure. Given the blatent and obvious bugs and 'release week' and sometimes even 'release day' patches for the majority of retail game releases these days, this so-called science needs to get it's act together.
They should make a current gen game that played like StarWars Jedi Power Battles for PS1/etc. 1- player Co-op, with updated graphics, and of course maybe less linear.
@paradroid90 not all modern games need massive developing teams or funding, like the first Portal was a fairly small, low budget game and it was developed by by like a dozen people.
@WolfGrey Amnesia - completely in-house engine, a lot of freedom what to add and what to modify FONV gamebryo - licensed 6-8 yrs old technology with limited customization options and without first-hand support. Do your homework, bro.
This is one of the driest, most boring articles I've ever read. I tried. Really, I tried...but....zzzzzzzzzzzzzzzzzzzzzzzzzzzzzz
The thing is I bet on paper games testing sounds amazing but when it comes down to the crunch I don't think it would be. I think it would ruin an enjoyment of just playing a game normally when you get your hands on it. Image you walk in the office in the morning and told, "OK, I want you to spend the next couple of hours doing the last section of the last level in this game and walk backwards while jumping!". It just will show you the last level of that game plus you can never play it normally! I swear once understanding about how a certain game works some of the magic disappears! I studied animation and now I can not just watch an animation film without deeply studying all its parts. I can not just sit down and enjoy a piece anymore!
@Evtutchenk0 The fact is, the games industry is not exactly unexistant here, but it only hires programmers and system analists. Like most cheap labor countries (I have to be realistic), transnational companies only use brazilian workers to do the hard work, in the case of games, programming and coding. Other than that, they (and only the big ones) only keep small PR teams here, and sometimes a few people to deal with imports, logistics, etc. I am, on the other hand, a marketing professional, now taking law classes. I could only work on the gaming industry if the administrative decisions where made here, not on the heardquarters (usually in the US or Europe). If I'm lucky enought, maybe I could be hired by Blizzard to deal with taxes when I'm a lawyer... but then I'd not really be working with games close enought so it matters, right? :)
@Nodashi why don't yo start this Brazilian gaming industry then? i can say that you would make a lot of money out here, and maybe even revitalize the world of games.
@shadow551991 Yeah man, too bad, game industry took a "wrong turn" - following the wish of casual players and made a games which aren't requiring much intelligence, spent time, or player skill, they don't offer much choices...at the end they are dull and boring after an hour of play. And I agree, they are eye Candies...and that's all they offer.
Lol at you all arguing if money has a say.Many Indie games that hit it big have a very small budget and almost no bugs. One cult hit can argue easily. Amnesia: The Dark Descent. One of the best survival horror games i can ever mention.Small budget, third company.Not even huge profit.But very highly acclaimed and all positive reviews.Not to mention almost entirely bug free AT RELEASE. New Vegas.Huge budget, tons of money involved.One of the most glitchy and buggy pieces of crap ever made.Even had to make a official apology THE DAY AFTER it released.Ya.... So yes hard work and actually caring about their games makes a big difference.Money really doesnt.
Actually, it's not in Seattle. It's in Bellevue and Redmond, WA. Though for those that don't live here in WA, I guess saying Seattle is the most recognizable way for people who don't live here in WA to understand where this takes place. Man, it feels kind of weird living only about 15 minutes away from these top game companies, and I hardly think much about it.
@UltraredM-I have to say you are right.If i remeber we have had what are called game drougths where many games are made but are poor,but when halo was made the attention and detail it got really shows what bungie is.Same with BioWare.BioWare always has a way of opening your mind.Valve makes you use your brain.Now i will say that yes Halo has flaws but what game doesnt?The only games i cant stand are COD games.They just dont have the story that halo,Mass Effect or Gears has.and thats why i really dont play them and for people who keep telling me "Halo sucks" well your wrong.If it wasnt for Halo,where would we be?
Valve is most definitely the epitome of the playtesting-design method, but Bungie? I refuse to believe they had peopel play through Library in the first Halo and choose not to make any tweaks. In fact, most of their game is just room-with-encounters followed by another-room-with-encounters, unlike more scripted titles. The only thing that needs playtesting is that you don't fall through the world, much less help with the actual design.
@paradroid90 I for sure see your point and agree. And jus so everybody knows, I'll play a game with tons of bugs, as long as it doesn't break a game, I jus perfer not too. It's jus like New Vegas, it sat on my table for months, than a few patches later, I give it another chance and put 150 hrs in it and really enjoyed it. I guess as gamers we all have the same thing in mind, we pay alot for these games and jus want a good product
@illagueril1 GTA IV was monstrously expensive, EA doesn't spend that much per game even though they are big. The point I was trying to make is that modern games are so complex that just having committed people developing them isn't enough, you need lots of money. Since there is a limited amount of money in the computer game market, there can only be so many AAA titles with lots of content and few bugs. Also I don't believe that Rockstar doesn't playtest their games and rely on seeing all the problems themselves. If you have crafted a piece of code you often become blind to its limitations. I do however agree that Rockstar couldn't be successful without people with talent and vision. But they need the cash to make a finished game out of that talent and vision. Just mad skillz isn't enough.
Bungie has enough pride to walk away from a series that, should they have continued to work on, would have continued to generate massive amounts of revenue for them.
These comments on Halo games being crap make me laugh. The gaming industry is flooded with so many poor or downright awful games that it has corrupted a vast amount of people's taste in gaming. Every Halo game, with perhaps the exception of ODST, was crafted with love and dedication that you just don't see often. For every AAA title video game you can name, there are 30 more that aren't worth the plastic they're shipped in. Also, before you rant on me being a Halo fanboy, this is coming from a person who won't be buying future Halo games. The Halo series has run its course, and I won't support it following in Call of Duty's steps.
@paradroid90 Money doesn't make games, people do. EA is one of the biggest game comapanies in the world and produce more games then any other company per year, there MONEY is unlimited, yet more than have thier games are crap. You can't jus throw money at a statue and expect it to come to life. Games are like movies, jus because a movie has a 150 million dollar budget doesn't mean it's goin be good, it depends on the director. The director style, commitment, dovotion and PRIDE. More money means longer dev cycle, that's it, it doesn't mean better game, ala Duke Forever!!!!!!
@illagueril1 You say that Rockstar makes bug free GTAs every 2 1/2 years. The lack of bugs in GTA IV depends on the massive amount of resources they put into developing it. And by "resources" I mean money. So "love and devotion" and "pride in your work" aren't the key factors. Earlier GTA games had major bugs. Remember the Purple Nines glitch in GTA III? Or the garage glitches in pretty much every game in series?
The main problems is who's developing these games, they rely on testers to tell them what is good to play, instead of playtesting the game themselves. And a game can be shipped without bugs, let's see here "every Metal Gear ever made!!! " Hideo makes his games bugs free with the best current graphics when his games come out. Rockstar makes bug free GTAs every 2 1/2 years, how do they did it, it's called love and devotion. It's having a lil something called pride in your work and never wanting to say, yeah skip it, we"ll jus leave it like that, it's a minor bug. What if car company's did that, or the people who fabricate airplanes said yeah, jus a minor bug. That would be insane. Do these fools not make enough money?? The dev cycles get smaller cause jus like any job you give a time frame, the sooner it's done the more money the company saves in the end. It's all about, well this is good enough cuz were reaching are " not gonna make 159% profit mark" Ive been gaming since before there was a Super Mario!! And the gaming world has changed time and time again, but that was innovation an such, now it's changing fir the worst, greed an such. Games can be bug free and innovative!!!! Trust me, I've seen em!!
@slavgunner Though there are several games that are exceptions, on the whole I agree with what you are saying.SP (at least in the FPS genre) seems to have been killed by its more popular brothers: MP and Laziness.
if bungie playtested with people who play games but never played halo (like me) then they would realize that having the left trigger as the frag button is not a good idea. i played halo 3 and 1st thing i did was blow myself up trying to aim down the sight. the weapons didnt feel powerful enough (and look like water pistols) and the vehicles are hard to control, i'm used to acelerating with trigger and getting my head round using the sticks with this strange contraption with a big wheel was harder than driving a tank, which also use stick controls.
Hope Codemasters (Dirt 2 and Dirt 3 freezing, no one noticed that for some reason. F1 2010 numerous obvious and ridiculous bugs), Konami (every current gen Pro Evo has some cheap goal scoring method) and Rockstar (GTA 4 doesn't work in 1080p on some machines for no apparent reason) took this as seriously I'd wait another month to have a game that worked. If the game sells well when it is broken a patch will not be coming. Half Life 2 is breathtaking, the industry moved on from there with jokes like the 25 minute Homefront, and continuously respawning enemies in CoD (until you move past some arbitrary marker) I keep getting piles of s##t that cost £40, having 'redeem' codes that I need to input on live to get half the game; and I greatly look forward to paying (fighters costumes Capcom? wtf!) for tiny pieces of extra content. If these idiots released games that had no bugs and lasted more than 15 minutes then you wouldn't find them second hand for £15 two months later. Playtesting my a###! In a couple of years we'll all have ps4 and xbox 720 consoles that developers know nothing about. Games will cost £70 because of development costs, be riddled with bugs and be 10 minutes long. Except Half Life 3....probably.
"Bungie playtested its Halo games with people who had never even seen a shooter." And it shows. Halo 1 was the best of the series, but, there were PC games long before Halo with better graphics and better gameplay. Halo did for XBOX what Goldeneye did for N64. It made an otherwise unnatractive system - attractive! Thing is, console gamers have no idea what they'd been missing and thought crappy games like Halo were the greatest things since sliced bread. And because all the game companies are working together with Gamestop to control release schedules, they are guarranteed to sell because each subsequent game offers less filler to keep us playing. I have a BETTER IDEA. How about they get REAL GAMERS to play test these games so the end product DOESN'T SUCK and isn't a REHASH of the game before it - only with a moderately different story? If I'd been on the Playtest team for Duke Nukem Forever, either that game woulda never been released, or it would have been released PERFECT.
@Kaine852 They can take their time with the game,but I want at least some news about it,I want to know if it even EXISTS,if they are making it!
@Kaine582: Some games take 12 years to make, and end up being one of the worst games of time...
a great game must have a public beta (im talking to you infinty ward) and lots of playtesting the feedback counts towards making the game near perfect i mean look at the call of duty franchise its annual relaese every year in november has in my opinion ruined the seires. the only reason i play black ops anymore is because of zombies if the game had a beta then feedback can be given to make sure it doesn't end up like modded warfare 2 or glitchapalooza, sequels need time to be made great.
There's an awesome documentary out on DVD on the subject of videogame playtesting, it's called Grandma's Boy.
I don't see why people are rushing Episode 3 (And some wanting a team fortress 3), I mean good games take time to make, not a freaking sequel every 1 year or 6 months
No wonder effing Halo Reach sucked, if you give it to noobs, it'll end up being noob friendly
@DemannameD well i think by the time they release episode 3 i just have to go over all of the half lifes to remember wtf was the thing all about... @hitmanxmk too right mate, yeah i confess i played a GTA (vice city) game and that was the first and the last time i ever touch a gta or any gta kind game because at that time i was 12. today i can hardly find a game suitin me. only things like dead space 2 or assassins creed standards are appealing to me. i have a younger brother that "shadow of the damned" was great to him and his friend while i just played the first 5 minutes and i couldnt bare to play more. @No_Intelligence and how am i supposed to play something that isnt entertaining? just pay for it and throw it in the corner of the room? this isnt called support it makes us look dumb and they think they can sell us more of their craps.
I've always thought that being a playtester would be the best job ever!! Until I watched a show about it and saw that 30 people sit in a room and play a broken game for 16 hours a day!! Yes, most of the time, 16 hours a day playing the same death match after deathmatch, or level after level. F--k that. I give em mad props thou, cuz even though I've gone on many 10-15+ game runs, I couldn't play the same unfinished game for 6months straight, 16 hours a day
@Rocker6 You have no idea how hard the SEs work to get rid of the bugs. They have no choice but to ship the game with "shippable bugs." They don't need a bigger playtest team, they know that the bugs exist, but they simply couldn't fix them all in time. What they need is a longer deadline and a bigger budget and simply more time to develop the near perfect game. I feel that you are a concerned gamer, which is good :), but don't know the actual process of game development. Fixing a bug, might end up causing 2 bugs. Because the game is built upon an engine that they have probably used for 3 years, there are limitations as to what they can fix. Framerate issues especially, the memory allocation is limited, when more content is added, the more memory is required. If the producer suddenly wants a couple more characters in a cutscene, that alone, can potentially break the ENTIRE cutscene. So the cutscene now requires more memory, which means less memory for the next thing, and so on... Which is why there are endless bugs. Ultimately, if you want better games in the future, please buy the games and support the industry.
