Thermo: The Board Game

As many of you may know, I’m currently hard at work on Thermo (demo video), our next generation RIA design tool, along with the rest of the design team. I wanted to share one of the research techniques we’ve used that’s proven to be quite effective.

Many design and research experts have promoted the use of paper prototypes for early-stage testing of design concepts. Historically, I’ve been lukewarm on the idea; not because it’s a bad idea per se but because the kinds of projects I tend to work on (complex desktop software for application designers and developers) have many moving parts that are difficult to simulate on paper. I felt that it would be near-impossible to use paper to recreate the experience of using the product with enough accuracy that we would get good data through testing.

With Thermo, however, I had no choice. There were so many new and untested concepts that we had to start iterative design and testing as soon as possible if we hoped to make the product usable. Although we had working code in various stages of completion, the “real thing” lacked the refinement I needed. Besides, testing on the real build is dissatisfying during early-stage design. Code changes take time that slows the process and leads to fewer opportunities for design exploration and iteration. Building interactive prototypes in Flash or another rapid-development technology can help ameliorate this problem if you have the resources to do so, but even they take longer to change than I would like (Thermo, of course, will make it even easier to build interactive prototypes for testing, but we have a bit of a chicken-and-egg problem there… And even Thermo prototypes will still be digital, which is never as malleable as paper). So NJ and Ethan and I bit the bullet and sat down to build a paper prototype of Thermo.

I affectionately call it “Thermo: The Board Game”.

Surprisingly, the prototype actually worked out extremely well. Although there are definitely aspects of the interface we could not express via paper (detailed content manipulations, low-level feedback such as rollovers, and subtle motion effects are all examples), we were able to express enough to get really good data on where our design was helping users understand the new concepts and complete the key tasks and where it was leading them astray. We conducted eight rounds (!) of iterative redesign and retesting, not stopping until we were convinced our target users were grasping the key concepts and applying them to accomplish their goals after only a brief period of experimentation with the product.

During the course of designing and running the study, I learned a few things about paper prototyping that I’ll definitely be keeping in mind for the next time.

1. Figure out up front what parts of the design you’re likely to get relevant feedback on via paper and what parts you aren’t. This way you can guide people past the parts where paper just doesn’t provide enough information for them to figure it out on their own and get to the parts of the task it does test well. Ideally, you’ll have a chance later on to study a higher-fidelity prototype to get full coverage of your workflows.
2. Sketch out the workflow via hand-drawn storyboards before you or your designer starts cranking out high-fidelity comps. This will help you understand what pieces of the UI you need and how the user might interact with them. In the worst case, you can always hand-draw something on the spot, but it sucks to be missing whole sections of the product.
3. Make sure you have high-resolution printouts or your participants will have too much trouble viewing your UI elements. Laser printers are a must. We forwent color in favor of high-quality laser printouts and just used colored pencils to draw in the parts that had to be in color to support user comprehension of the design. If your design uses color more extensively than ours, get a color laser printer.
4. If you have lots of fiddly little bits of UI that need to be individually moved around, consider printing out your interface on large-sized paper to make them easier to grab and manipulate. I used tabloid (11×17) printouts for everything. This obviously doesn’t simulate how well your design will perform at lower screen resolutions, but paper isn’t the right way to simulate this anyway (see point 1 above).
5. Running paper prototypes is slow. Don’t be afraid to stop the participant from going down a path that is really going to get them into a bad state and waste lots of time in the session. However, use this as an opportunity to see why they wanted to go down that path – understanding their mental model will help you fix your design.
6. Strike a balance between formal study techniques and informal conversation. Avoid guiding the participant, but feel free to chat with them as they go to understand what they’re thinking. Correct any mistaken assumptions that are an artifact of the low-fidelity paper, but make sure you record points where you have to help them or explain something – these are design problems you need to fix.
7. Have lots of scrap paper, printouts of UI elements, and scissors with you when you run the prototype. You’ll be doing a lot of on-the-spot arts and crafts as you adapt the UI to what the participant is doing.
8. If the participant is really confused, feel free to make on-the-spot design changes to see if it helps them understand what to do next. In your later iterations, you should find yourself doing this less and less.
9. Have someone else take notes. You can’t take notes and interact with the participant and run the prototype all at the same time, and it’s important for the researcher to be the one running the prototype and interacting with the participant. Although not all note-takers do as good of a job as you might, I find almost everyone can do a good enough job after 5 minutes of training and viewing an example of a well-recorded session.
10. As with any iterative testing session, schedule debriefs and redesign sessions soon after each round. Digest the problems quickly and help your design team to come up with solutions and rev the prototype. Problems always tend to get solved quicker when there’s a hard deadline – I only allotted one week between rounds.

Even though designing the study and constructing the paper prototype was a huge undertaking, I’d definitely do it again. If your product makes use of new concepts or patterns that your users may not be familiar with, it’s an invaluable way to get feedback quickly, incorporate that feedback into a new design, then retest to make sure your solution works. Paper allows for a turnaround time that no other technology can match. I’ll paraphrase one of our senior engineers, Mark Shepherd, who said “it may have taken you eight rounds to get this far, but back in the old days it would have taken us until Thermo version 8!”