More Playtesting

Isbister, K., & Schaffer, N. (2008). Game Usability. New York: Morgan Kaufman.

Chapter 15
TRUE Instrumentation: Tracking Real-Time User Experience in Games
Eric Schuh, Daniel V. Gunn, Bruce Phillips, Randy J. Pagulayan, Jun H. Kim, Dennis Wixon
Microsoft Game Studios

This chapter presents a method in research to improve user experience called TRUE instrumentation. Focus groups, usability testing and playtesting provide limited feedback and are labor intensive.  Automated collection of feedback for the entirety of the game is a better solution.
The chapter starts with a discussion on Voodoo Vince, a game that helped Microsoft realize the importance of collecting nuanced data.  The issues uncovered in this game spurred Microsoft to build an application for users to log their behaviors (like when they level up, etc).  This gave a rough idea of where problems existed.  This system turned into TRUE – Tracking Real-Time User Experience.  This instrumentation includes: attitudinal feedback, contextual data, and captured video.
To capture attitude, the researchers added a one question survey to periodically come up during gameplay. The surveys are event-based, on-demand, and time-based.  To provide context, the researchers capture the build number, test name, participant ID, timestamp, difficulty setting, chapter name, and position coordinates with the surveys.
Halo 2 was evaluated with TRUE.  They looked first at the number of deaths in each mission, then looked at where exactly in the missions the players were when they died, and finally how they died.  Then they presented a video of the deaths to the design team who could recognize the problem.
Most data collection happens at the end of production to do a final check on a design that is considered shippable. TRUE also has a place in beta and demo testing.
The chapter ends with lessons on conducting successful implementation:
-Plan ample time for iteration
-Start with research questions in mind
-Keep the number of variables tracking down (no more than 15 events)
-Mock up a report before setting hooks
-Represent the data visually
-Evaluate your instrumentation
-And still get other forms of feedback

—

Chapter 16
Interview with Georgios Yannakakis, Assistant Professor at the Center for Computer Games Research, IT-University of Copenhagen
Interviewer: Katherine Isbister

Yannakakis studies the connection between user satisfaction, human response and online learning.  He developed the Player Satisfaction Modeling (PSM) task force.  This is a quantitative player satisfaction modeling that looks at using the game’s AI to increase the overall play experience and player satisfaction.  The interview stresses that middleware that captures player satisfaction will raise the market value of the game, and also automate user testing for player satisfaction.

—

Chapter 17A
Usability for Game Feel
Steve Swink

Game feel is tactile, kinesthetic control of a video game.  Good game feel is intuitive, deep and aesthetically pleasing.  Making a game mechanic feel good is not easy for the designer.  Here Swink lays out a method for testing/creating good game feel.

The pieces of game feel are:

Input: the controller needs to feel good for the player and have natural mappings. There should be little or no explanation needed for how the controller works.  Designers need to keep in mind the inherent sensitivity of the input device.  A computer mouse is sensitive; a button is not.

Response: The response to the input can feel sensitive or not depending on the design.  A mouse (sensitive) can only control the x-axis of on-screen movement (not so sensitive).  Nuanced reactions are found in games like Mario where a not so sensitive input (buttons) creates sensitive response (Mario jumping at different distances and times).

Context: This is the environment for interaction.  Designers should build a test context to tune the game feel, to give the motion meaning.  These contexts should include a variety of objects and constraints.

Polish: These are effects that enhance the game world and convey the physical properties of objects and object interactivity.  Pieces of physical polish include motion, tactile, visual and sound.  Polish is time consuming but vital for good game feel.

Metaphor: People have built-in constraints on how things should feel.  Designers use these preloaded expectations to execute on how something should feel in a game (a car, an animal, etc.).

Rules: Rules give more meaning to the feeling of control/mastery.  Goals and rules in a game need to be sustainable and meaningful.

“The first, last, and most common thing a player will experience when playing your game is its feel” (280).
—

Chapter 17B
Further Thoughts from Steve Swink on Game Usability

– No one reads in-game text if they can skip it.
–Experiential testing: the objective view on the current game.  Here we compare live experience with the designer’s vision.
–Defect testing: Bug hunting.  This is rigorous, systematic and intense testing that excludes experiential concerns.
–Usability testing: this is debugging the experience.
–Why do we test?: A game is a collaboration between the player and the game.  It does not exist without the player (like a movie would).  We test because we can’t control the player’s experience, only guide it, so we need to guide it in the closest way to the designer’s goal.
–Defining experience: Designers watch players play to look for increased fun and decreased boredom.  “Fun” is not the only way to enjoy a game or have an enriching experience.  A designer can break a game into moments of interactivity.
–Usability behavior: is binary. It is “got it” or “don’t got it.”  You can test for usability.  Give the player a task that requires her to use a certain button – this is a usability test.
–Experiential behavior: This tests the essence of design for enjoyment and interest.  It is not binary.
–Challenge vs. Obfuscation: You need to know the challenge of your game before you can playtest.  Challenge, usability, and game design are all inextricably linked.
-Detailed planning: Usability – does the player understand the UI and buttons.  Experience – what should the player be feeling or thinking in the important moments of the game. Challenge – be specific!
-Swink continues with the TETRIS TEST giving a specific plan on page 299.
—

Article of choice:

Capturing the Spirit of Sesame Street
Interview with Nathan Martz and Tim Schafer of Double Fine

http://www.gamasutra.com/view/feature/6295/capturing_the_spirit_of_sesame_.php

This article is actually a moving piece about members of Double Fine (a super creative game developer) who started years ago with an idea for a game based solely around the notion of “upbeatness,” that the game experience would be uplifting.  They tossed around a bunch of ideas and finally came up wit monster that you have to interact with to help them solve problems.  The more they worked on designing the monsters, the more the game started to resemble the experience of Sesame Street.  Finally, this company that have always done original material, decided to make a game with licensed content.  Sesame Street: Once Upon a Monster is based on the idea that a game should provide a creative, uplifting experience, and that the game feel should have something to do with mupeteering.
To provide a game feel that was as physically accurate to control puppets as possible, Double Fine chose the Kinect as their platform.  They aim for the monsters on screen to look at feel like puppets, not like CG animations.
This interview focused almost exclusively on the game feel and game experience, and didn’t even touch on the real plot of the game.  This is probably because the game is still in development, and the designers started with feeling instead of story.

Read Full Post »

Research Methods: Think-Aloud Protocols

Toler, T. (2009). Validity and Think-Aloud Protocol. Solid State UX Blog.

Validity and Think-Aloud Protocols
A think-aloud protocol is when the moderator of a usability test asks the tester to say what he’s thinking out loud.  This is a direct measure of what’s happening in the subject’s short-term memory.
Level 1: verbalizations: the emphasis is on pure thought, minimal explanations.
Level 2: verbalizations: verbalizing non verbal information (like shapes)
Level 3: verbalizations: “thinking plus explanations” or retrospective reports
Inactive moderation: creating a unified experience for all test subjects. ex. researchers behind glass.
Active moderation: using focused, open-ended questions to focus the participant’s attention.
These kind of probing questions are usually bad practice for think-alouds.  The moderator shouldn’t pose additional cognitive load.
Some say observing behavior is not enough to tell what’s happening in terms of higher-order thinking and the researcher must ask some questions.
—
Wired Magazine Issue 15.09
Halo 3: How Microsoft Labs Invented a New Science of Play
Clive Thompson

The testing lab for Halo 3 at Bungie Studios looks like a psychology testing lab.  It has a play station set up to look like a living room, the game is digitally recorded on screen, and there are cameras on the players.  Randy Pagulayan’s job is to find flaws in the game that the designers wouldn’t see.  He studies rival game titles as well to see how Halo matches up.  “They’re trying to divine the golden mean of fun” (2).  The game should accommodate both hardcore fans and new players.
The lab uses heat maps on the games to find unfair terrain advantages and see where players die.  The game has not been designed in harmony because the development team is so big, so this kind of testing can find discrepancies between levels designed by different people.  The designers first create weapons, levels and situations.  Then the test leads monitor hundreds of people playing them.  Every two weeks there are 20 playtesters brought in.  Sometimes they engage in think-aloud protocols, and sometimes they answer questions from pop up boxes on screen every couple minutes.  The team records snapshots of where players are located after specific time intervals to judge the flow of the game.  Problems with the game are then analyzed and designers iterate by making small changes that subtly direct players’ movement along the correct path.
—
Isbister & Schaffer (2008). Game Usability, chapters 5.

Let the Game Tester Do the Talking: Think Aloud and Interviewing to Learn About the Game Experience

Henriette (Jettie) C.M. Hoonhout

In usability testing the researcher wants to look inside the tester’s head.  This isn’t really possible but we come close with think alouds and interviews where testers verbalize their experiences.  Typical issues include:
-interesting and adequate challenge
-continuous challenge
-different elements contribute to the experience
-player learning the game
-social interactions developing
-easy controls

Think Aloud:
Here players verbalize their thought process as they play.  This is concurrent or retrospective.  This results in valid data if the researcher is prepared.  However, it’s time consuming, players don’t always have words and they think faster than they speak.  Researchers should give reminders to keep talking, record and transcribe the report, and take notes.  They should analyze the material into categories, and compare with other researchers.  An alternative is to not conduct a formal think aloud, but still record what testers say.

Interviewing:
An interview collects qualitative data regarding opinions and attitudes.  Mistakes can be detected and corrected, and testers can elaborate in interviews.  A semi structured interview has defined topics but not defined wording or order.  Interviewers may show testers their playtest footage to get their retrospective response.  Questions should not be coercive, leading or condescending.  Interview analysis includes transcribing, dividing into chunks, categorizing and analyzing.
Interviews require social skills and training, and sometimes produce conflicting data.

The best approach for usability testing is to use mixed methods.

—

Article of choice:
Successful Playtesting In Swords & Soldiers

Jeroen van der Heijden

The game is an RTS where the player has to build soldiers and cast spells.  The playtesting session was for the game’s release on a new platform, moving to Playstation from the Wii.  The point-and-click mechanic may be a problem on the Sony controller.
The research questions addressed the learning curve and the controls/UI.  The target audience were people familiar with PS controls, aged between 14-30, RTS players.  8 participants were selected.
The sessions included an inconspicuous camera focusing on the player, a lightweight camera on the controller, an eye tracker, recording of spontaneous player comments and an observation room.  The testers played for 30 minutes and then had a semi-structured interview.
The company didn’t have enough time or money to use biometrics, so they instead observed players emotional state (facial expressions, body language).  They did employ eye tracking, which was good for observing menu/interface use, but not gameplay.
The issues they found included a confusing menu for upgrading soldiers and spells and lengthy tutorial text.
They learned:
Eight players is more than enough for testing usability, menu issues.
The controller cam was only useful in the post-test interview, but confusing to watch during testing.
Observing player experience was good but limited.  Watching posture to see engagement was more useful.

Read Full Post »

Playtesting

Isbister, K., & Schaffer, N. (2008). Game Usability. New York: Morgan Kaufman. Chapters 6 – 9

Chapter 6

Heuristic Evaluation of Games

Noah Shaffer

Shaffer talks about heuristics as an alternative to user testing; it’s the “discount method” (79).  This is good for game studios with little time and money.
Heuristic evaluation is basically guideline-based methods, or shortcuts, for usability evaluation.  This method is not so useful for academics, but very useful for getting feedback on specific interfaces.
Nielsen’s 10 Heuristics are mentioned (80) that include good categories but don’t help game developers find the problems with usability.  Nielsen’s heuristics are also not specifically about games.  A number of other heuristic list developers are mentioned who have heuristics that suffer mostly from lack of specificity.  Shaffer himself created a list of heuristics for evaluation of game usability that included points like, “use natural control mappings,” and “don’t make it easy for players to get stuck or lost” (82).  But even his list cannot be applied to all games.
Heuristics help when they are implemented early and often, to avoid having to redesign or recode later (after finding out that the game isn’t usable).  However, it’s good to keep in mind that, even with heuristics, it’s impossible to anticipate all user problems (there is still room for user testing!).
How to do heuristic evaluation?
1. Three to five evaluators (at least one usability expert) look for violations of the heuristics in the game INDIVIDUALLY.
2. They combine their findings into a master list.
3.  They organize the problems in order of severity, and give recommendations.

Remember: don’t confuse heuristics with usability testing or with standards.  They are simply guidelines for evaluation, not rules for design, and they don’t represent the reactions of all players.
—
Chapter 7
Usability and Playability Expert Evaluation
Sauli Laitinen

Expert evaluators look for problems, create reports, and suggest solutions.  Their work is meant to aid game developers.  In expert evaluation, this is what’s being evaluated:
-Game Usability – is the game easy to learn? does it support user interactions?
-Gamplay – the user interface is ignored and the game itself is focused on. The goal is to remove unintended challenges, and increase fun.
-Platform and game type – the platform should support the gameplay.
All three of the above aspects are related and need evaluating.

How evaluation is done:

-Two or three evaluators review the game, for different experts find different problems.  More than three experts is too many.
-Double experts are experts in both usability and gaming.
-Evaluators should be external, not the same people as the designers.

When to evaluate:

-Early, with design documents and paper prototypes. This allows evaluation of basic usability and gameplay.
-Working prototypes, you can evaluate parts of the game independently.
-Nearly complete game, study all the aspect of the game – it’s bad to wait until this point to evaluate because it takes time to fix problems.
-The expert evaluation should be done BEFORE playtesting sessions.
-The process of evaluation should be iterative.

The process:
-Plan the work: developers introduce the game to evaluators and inform them about bugs, missing pieces, or places to pay attention to.
-Evaluators review the game, maybe with cheats/short cuts to reduce time.
-Evaluators create a list of problems and possible solutions.  This usually takes one-three days.
-Evaluators create a report of the findings, in order of severity, rating, description, solution, etc.  And often with a summary of key findings.
-Evaluators report the finds to developers.  The whole process takes one – one and a half weeks.

Heuristics:

Here are two long lists of heuristics.

Usability Heuristics (103 – 105):
-consistency – no unnecessary exceptions
-provide feedback – after each action taken within the game
-easy to use/understand terminology
-minimize player’s memory load -(this is like gee’s on-demand, just-in-time information)
-avoid errors- prevent the player from making mistakes
-provide help – learning how to play
-simple and clear menus
-avoid mixing user interface and game interface
-screen layout is efficient and visually pleasing
-audiovisuals support the game
-game controls are flexible and convinient – and players can configure their own controls

Gameplay Heuristics (106 – 108):
(This list uses abbreviated language)
-clear goals provided
-player sees the progress in the game
-rewards are meaningful
-player is in control
-balance of challenge, strategy and pace
-encouraging first impression
-story supports gameplay
-no repetitive or boring tasks
-game supports different playing styles
-the game progresses (not stagnate)
-the game is consistent
-use of orthogonal unit differentiation – different objects have different purposes
-player doesn’t lose hard-won possessions
-players can express themselves
—
Chapter 8
Interview with Eric Schaffer, Ph.D., CEO of Human Factors International

Interviewer: Noah Schaffer

This interview focuses on usability standards.

Usability standards increase the speed and cost of development about 10%.  They are:
design principles: like “use short words, write in the active voice”
design standards: so the game meets players’ expectations
methodological standards: systematic processes to help usability professional, not the design of the game

Design standards should be reusable templates or examples.  This includes details like colors, fonts, and positioning.

The creation of these standards is collaborative, made by a committee.  This committee designs example screens, makes a game with those screens, evaluates the game, and iterates to create the standards.  Finally, the usability committee enforces these standards on the designers.
—

Chapter 9
The Science Behind the Art of Game Design
Chris Swain

This chapter lists 8 metric-based game design techniques.  Metrics help the game designer’s revision process, for they are something that can be measured, adding the scientific method to game design.

Techniques:
1. Feature Design: Metacritic
Metacritic is a website the compiles reviews on games from many sources into one score. High scoring games on metacritic usually have 20+ hours of content, choice for the player, replayability,     quality audiovisual, easy controls, a storyline, interactive world (AI), and a responsive camera.  Low scoring games are not unique, too linear, and too difficult.  These are important points, but the author cautions “don’t be a slave to metracritic data” and “know the rules so you can break them” (124).

2. Feature Design: Morphological Analysis = Analytical Creation
(from Michael Michalko’s Thinkertoys)
An idea box is when you define a set of parameters for the topic, list as many elements under each parameter as possible, and then select and combine listed parameters.  Here, examples are given for character design (125), parameters being head, eyes, ears, etc., and elements being round, egg-shaped, hero chin, etc.   This process helps generate a ton of ideas quickly, provides an analytical approach to creativity, and is repeatable.

3. Mechanics Design: Quantifying Emotions
Nicole Lazzaro defines four types of emotion in player experience:
fiero (personal triumph)- ex. puzzles
curiosity – ex. role-playing
amusement – ex. cooperation
relaxation/excitement – ex. repetition/rhythm

Games that evoke three or more of these emotions do better in the marketplace.  These games offer more ways for the player to feel.

4. Level Design: Heat Maps
A heat map shows where events happen in the game level, like where a player dies.  Heat maps are more convincing than written reports, because
they are so precise and are unbiased analyses.

5. Level Design: “Time Spent” Reports
Here the in-game behavior of playtesters is tracked by instrumentation software.  The software looks at relationships of time spent doing different activities in the level.  Playtester perception may be skewed by cool features or by a frustrating event.  Designer perception can be skewed by their own infatuation with their design.  Numbers are useful to help negotiate when team members have different opinions.
“One measurement is worth fifty expert opinions” Howard Sutherland (133).

6. Level Design: Track Engagement with Bio-sensors
Sensors can measure brain waves, temperature, breath, heart rate, physical motion, and eye movement.  They can track adrenaline, thought, and positive emotions.  This gains more precise data than simply asking players.  Studies have shown that a balance of high and low engagement (in 5 minute intervals) creates player satisfaction (134-135).

7. Control Design: Simplifying Controls
A good control scheme is integral to a game’s design.  This includes both hardware and on-screen controls.  The rule of thumb is “as few controls as possible.”  Activision developed a system called Control Dimensionality that scores games based on their control schemes.  A game with really simple controls has a lower score. Activision rates its own games against games in the same genre to see if the control schemes are similar.

8. Experience Design: Playcentric Design
The player experience should be the central focus when designing a game.  So playtesters should be integrated in the design process form the very beginning.  Applying metrics to the playtest sessions can help make informed creative decisions.

—
Article of Choice:

Resetting Accessibility in Games
by Dennis Scimeca

http://www.gamasutra.com/view/feature/6239/resetting_accessibility_in_games.php

This article talks about improving usability in games for disabled populations, and how that actually improves games for everyone.  Difficult user interfaces are most often deal-breakers, especially for newer gamers (and there are more new gamers than experts).

Some control schemes are difficult for motion impaired users who can’t hold down two buttons at a time (they are also difficult for non-gamers in general).  However, a control scheme that requires single keystrokes, not only helps disabled gamers, but lets everyone remember the control scheme easier.

The article continues to talk about the importance of pacing, scaffolding users into harder gameplay, and allowing them to pause or slow down the game.  Valve’s Left for Dead was acclaimed by the disabled gaming community for many features, including its pause button.  Using drills to practice within the game is also beneficial for all players – practicing attacks, for example, could be an early level in the game.

An accessable game company in MA explained that they talk to parents, therapists, and teachers before they even begin designing the game.  They need to hit their mark before they start designing because backtracking is very expensive, and it’s difficult to simplify a complex game.

The take-home point is that designing games for disables populations, older gamers, younger gamers, and newer gamers, can actually benefit many hardcore gamers as well.

Read Full Post »