Toler, T. (2009). Validity and Think-Aloud Protocol. Solid State UX Blog.
Validity and Think-Aloud Protocols
A think-aloud protocol is when the moderator of a usability test asks the tester to say what he’s thinking out loud. This is a direct measure of what’s happening in the subject’s short-term memory.
Level 1: verbalizations: the emphasis is on pure thought, minimal explanations.
Level 2: verbalizations: verbalizing non verbal information (like shapes)
Level 3: verbalizations: “thinking plus explanations” or retrospective reports
Inactive moderation: creating a unified experience for all test subjects. ex. researchers behind glass.
Active moderation: using focused, open-ended questions to focus the participant’s attention.
These kind of probing questions are usually bad practice for think-alouds. The moderator shouldn’t pose additional cognitive load.
Some say observing behavior is not enough to tell what’s happening in terms of higher-order thinking and the researcher must ask some questions.
—
Wired Magazine Issue 15.09
Halo 3: How Microsoft Labs Invented a New Science of Play
Clive Thompson
The testing lab for Halo 3 at Bungie Studios looks like a psychology testing lab. It has a play station set up to look like a living room, the game is digitally recorded on screen, and there are cameras on the players. Randy Pagulayan’s job is to find flaws in the game that the designers wouldn’t see. He studies rival game titles as well to see how Halo matches up. “They’re trying to divine the golden mean of fun” (2). The game should accommodate both hardcore fans and new players.
The lab uses heat maps on the games to find unfair terrain advantages and see where players die. The game has not been designed in harmony because the development team is so big, so this kind of testing can find discrepancies between levels designed by different people. The designers first create weapons, levels and situations. Then the test leads monitor hundreds of people playing them. Every two weeks there are 20 playtesters brought in. Sometimes they engage in think-aloud protocols, and sometimes they answer questions from pop up boxes on screen every couple minutes. The team records snapshots of where players are located after specific time intervals to judge the flow of the game. Problems with the game are then analyzed and designers iterate by making small changes that subtly direct players’ movement along the correct path.
—
Isbister & Schaffer (2008). Game Usability, chapters 5.
Let the Game Tester Do the Talking: Think Aloud and Interviewing to Learn About the Game Experience
Henriette (Jettie) C.M. Hoonhout
In usability testing the researcher wants to look inside the tester’s head. This isn’t really possible but we come close with think alouds and interviews where testers verbalize their experiences. Typical issues include:
-interesting and adequate challenge
-continuous challenge
-different elements contribute to the experience
-player learning the game
-social interactions developing
-easy controls
Think Aloud:
Here players verbalize their thought process as they play. This is concurrent or retrospective. This results in valid data if the researcher is prepared. However, it’s time consuming, players don’t always have words and they think faster than they speak. Researchers should give reminders to keep talking, record and transcribe the report, and take notes. They should analyze the material into categories, and compare with other researchers. An alternative is to not conduct a formal think aloud, but still record what testers say.
Interviewing:
An interview collects qualitative data regarding opinions and attitudes. Mistakes can be detected and corrected, and testers can elaborate in interviews. A semi structured interview has defined topics but not defined wording or order. Interviewers may show testers their playtest footage to get their retrospective response. Questions should not be coercive, leading or condescending. Interview analysis includes transcribing, dividing into chunks, categorizing and analyzing.
Interviews require social skills and training, and sometimes produce conflicting data.
The best approach for usability testing is to use mixed methods.
—
Article of choice:
Successful Playtesting In Swords & Soldiers
Jeroen van der Heijden
The game is an RTS where the player has to build soldiers and cast spells. The playtesting session was for the game’s release on a new platform, moving to Playstation from the Wii. The point-and-click mechanic may be a problem on the Sony controller.
The research questions addressed the learning curve and the controls/UI. The target audience were people familiar with PS controls, aged between 14-30, RTS players. 8 participants were selected.
The sessions included an inconspicuous camera focusing on the player, a lightweight camera on the controller, an eye tracker, recording of spontaneous player comments and an observation room. The testers played for 30 minutes and then had a semi-structured interview.
The company didn’t have enough time or money to use biometrics, so they instead observed players emotional state (facial expressions, body language). They did employ eye tracking, which was good for observing menu/interface use, but not gameplay.
The issues they found included a confusing menu for upgrading soldiers and spells and lengthy tutorial text.
They learned:
Eight players is more than enough for testing usability, menu issues.
The controller cam was only useful in the post-test interview, but confusing to watch during testing.
Observing player experience was good but limited. Watching posture to see engagement was more useful.
Katya's Educational Technology Experience 