PlaytestingPlaytesting

Playtesting is not…Playtesting is not…

• Focus groups – gathering opinions on Focus groups – gathering opinions on proposed game/feature ideasproposed game/feature ideas

• QA testing – the hunt for bugsQA testing – the hunt for bugs• Usability testing – is the user interface Usability testing – is the user interface

easy to understand and intuitive; how easy to understand and intuitive; how difficult is it to learn how to playdifficult is it to learn how to play

• Playtesting – does my game achieve its Playtesting – does my game achieve its goal (very often, is it fun?)goal (very often, is it fun?)

Early and oftenEarly and often

• The feedback from playtests is The feedback from playtests is useless when it comes too late in the useless when it comes too late in the project to make changesproject to make changes

• Infrequent playtests generate less Infrequent playtests generate less revealing and less important revealing and less important discoveriesdiscoveries

• You always need to playtest more You always need to playtest more often than you thinkoften than you think

Why playtest?Why playtest?

Quote from Peter MolyneuxQuote from Peter Molyneux

(Fable, Black & White, Populous)(Fable, Black & White, Populous)

Outside observersOutside observers

• Whenever possible, hire outsiders to Whenever possible, hire outsiders to conduct your playtestsconduct your playtests

1> Relieves playtesters from having 1> Relieves playtesters from having to be tactful (you want brutal honesty)to be tactful (you want brutal honesty)

2> Keeps your from justifying and 2> Keeps your from justifying and defending your work (they don’t care)defending your work (they don’t care)

• If not, at least invite playtesters to If not, at least invite playtesters to help you most by being criticalhelp you most by being critical

Outside observersOutside observers

• If not outsiders, then conduct tests If not outsiders, then conduct tests using those farthest from the work or using those farthest from the work or you’re asking for debates (you can you’re asking for debates (you can even divide the game between even divide the game between project teams)project teams)

Why? We love our babiesWhy? We love our babies

• It’s hard to watch someone cut up It’s hard to watch someone cut up your baby, we defend it ruthlesslyyour baby, we defend it ruthlessly

• There are even social/psychological There are even social/psychological terms for this and other related terms for this and other related phenomenon, such as “sunk cost phenomenon, such as “sunk cost fallacy” and “loss aversion”fallacy” and “loss aversion”

• The more you invest, the more you The more you invest, the more you irrationally protect the investmentirrationally protect the investment

Dollar AuctionDollar Auction

• I am auctioning off a $100 billI am auctioning off a $100 bill• It goes to the highest bidder, who It goes to the highest bidder, who

simply pays their bid and receives the simply pays their bid and receives the $100$100

• But the second highest bidder must also But the second highest bidder must also pay their highest bid, yet gets nothingpay their highest bid, yet gets nothing

ANY BIDS?ANY BIDS?

Environment of honestyEnvironment of honesty

• Encourage your playtesters to be Encourage your playtesters to be brutally honest and unforgiving in their brutally honest and unforgiving in their comments (helps if they know they are comments (helps if they know they are not going to hurt your personal feelings)not going to hurt your personal feelings)

• Sets an environment where they won’t Sets an environment where they won’t hold back nit-picky commentshold back nit-picky comments

• People don’t like to be mean, you must People don’t like to be mean, you must give them permission / force them togive them permission / force them to

How to begin…How to begin…

• IDEAL: say nothing and just let them IDEAL: say nothing and just let them playplay

• Concession: explain the controlsConcession: explain the controls• Concession: explain the goalConcession: explain the goal• Anything else will completely subvert Anything else will completely subvert

your attempt to see what the player can your attempt to see what the player can discern on their owndiscern on their own

• Remember that you can’t be there when Remember that you can’t be there when they actually play your gamethey actually play your game

Don’t taint the resultsDon’t taint the results

• It’s hard NOT to subliminally tip your It’s hard NOT to subliminally tip your hand and ask leading questions:hand and ask leading questions: ““The second level was pretty easy, right?” The second level was pretty easy, right?”

Makes them embarrassed to say it was way Makes them embarrassed to say it was way too hard, they’ll say “it wasn’t that easy”too hard, they’ll say “it wasn’t that easy”

““How did you like the large open world?”How did you like the large open world?”

““Large” and “open” are positive adjectives, Large” and “open” are positive adjectives, they think they are supposed to naturally they think they are supposed to naturally agreeagree

• Never make a statement with a questionNever make a statement with a question

Keep ‘em separatedKeep ‘em separated

• If conducting long playtests where there If conducting long playtests where there are breaks or you have an open space, are breaks or you have an open space, playtesters will want to chat with each playtesters will want to chat with each other about the game. DON’T LET THEM!other about the game. DON’T LET THEM!

• You don’t want to muddy someone’s You don’t want to muddy someone’s perspective, or allow assertive testers to perspective, or allow assertive testers to dominate other people’s opinionsdominate other people’s opinions

• Unless your game RELIES on groups of Unless your game RELIES on groups of people playing together in the same people playing together in the same space, don’t let playtesters minglespace, don’t let playtesters mingle

Let them sufferLet them suffer

• It’s irresistible to help a playtester figure It’s irresistible to help a playtester figure it out, rather than sitting in agony as it out, rather than sitting in agony as they become frustrated and hate your they become frustrated and hate your gamegame

• You want to see the things they will tryYou want to see the things they will try• Ask them what they are thinking as they Ask them what they are thinking as they

fail, what they THOUGHT should happen fail, what they THOUGHT should happen (then change your game to make them (then change your game to make them right next time, fail to success…)right next time, fail to success…)

What is unclear?What is unclear?

• Find out what images, icons, text, Find out what images, icons, text, missions, features, etc. are not clearmissions, features, etc. are not clear

• If they don’t use certain buttons, ask If they don’t use certain buttons, ask what they think would happen if they what they think would happen if they pressed them (based on the icons)pressed them (based on the icons)

• Be invisible as much as possible, but Be invisible as much as possible, but actively solicit feedback when they actively solicit feedback when they DON’T try things you need to be DON’T try things you need to be testedtested

They don’t careThey don’t care

• Never explain or justify your decisions Never explain or justify your decisions to the playtesters. It’s instinctual when to the playtesters. It’s instinctual when they criticize but you don’t need to they criticize but you don’t need to convince them your choices were wise, convince them your choices were wise, you just need their honest feedbackyou just need their honest feedback

• The more you defend the less they The more you defend the less they attack, and you WANT the onslaught attack, and you WANT the onslaught (joining in actually helps them speak (joining in actually helps them speak out)out)

Target demographicsTarget demographics

• Test the target thoroughlyTest the target thoroughly

but also test outside to getbut also test outside to get

things you hadn’t realizedthings you hadn’t realized

• Example: Any user interface that can be Example: Any user interface that can be understood by 40-50 year olds will be understood by 40-50 year olds will be understood by 20-30 year olds. But the older understood by 20-30 year olds. But the older testers provide great value in showing what is testers provide great value in showing what is possibly “on the fence” for the younger testerspossibly “on the fence” for the younger testers

Get statisticsGet statistics

• Not just what doesn’t work, but how Not just what doesn’t work, but how many people did it not work formany people did it not work for

• You can ignore problems that only You can ignore problems that only strike a minimal percentage of strike a minimal percentage of testers and you MUST address testers and you MUST address problems that most of them hadproblems that most of them had

• Breadth is more valuable than depth Breadth is more valuable than depth (more sessions with more people (more sessions with more people rather than longer sessions with less)rather than longer sessions with less)

Statistics in gameStatistics in game

• Where possible, log important eventsWhere possible, log important events• It’s useful to know where your It’s useful to know where your

players are struggling, how long players are struggling, how long certain tasks take, etc.certain tasks take, etc.

Statistics in gameStatistics in game

• Refer to Microsoft’s processes for a Refer to Microsoft’s processes for a look at how detailed statistics, and look at how detailed statistics, and filtering thereof, benefits trouble filtering thereof, benefits trouble spotsspots

• WIRED articleWIRED article• MS playtest from tester perspectiveMS playtest from tester perspective• GDC PresentationGDC Presentation

Kleenex testersKleenex testers

• You want lots of fresh testers “Kleenex You want lots of fresh testers “Kleenex testers” who have never played your testers” who have never played your gamegame

• Getting testers through the first five Getting testers through the first five minutes of your game is the toughest minutes of your game is the toughest partpart

• Experts in your genre are less valuable Experts in your genre are less valuable since they know the territory and guess since they know the territory and guess well when in doubt (can’t rely on them)well when in doubt (can’t rely on them)

Recurring testersRecurring testers

• Kleenex testers show you where your Kleenex testers show you where your game should improve for first-timersgame should improve for first-timers

• Recurring testers are required to Recurring testers are required to discover replayability and when your discover replayability and when your game tiresgame tires

• Crucial for longer game experiences, Crucial for longer game experiences, particularly for MMO’sparticularly for MMO’s

Prepare questions aheadPrepare questions ahead

• Do NOT improvise a post-play Do NOT improvise a post-play interviewinterview

• Follow surprises and interestingly Follow surprises and interestingly unique aspects of a playtester’s unique aspects of a playtester’s experienceexperience

• But have your crucial questions But have your crucial questions prepared in order to make sure they prepared in order to make sure they are includedare included

Most important question to ask:Most important question to ask:

““What are the three worstWhat are the three worst

things about this experience?”things about this experience?”