The design process in VR, AR, and MR

Note, my thoughts below do not represent the views of my company.

Lately, I have been working with numerous design teams exploring virtual reality (VR), augmented reality (AR), and mixed reality (MR). Regardless of design experience, most teams had an interesting story:

They would start with their previous design process, whether it was from making software or games; however, after they created the design documents, mocked up the wireframes, and began implementing, nothing worked because VR/AR/MR were just too new for anyone to understand what designs would or would not work. After this point, most teams concluded, “Let’s just start prototyping in code as soon as possible and iterate from there.”

Many teams would even point to the “Build-Measure-Learn” cycle from The Lean Startup Methodology to try to justify why it was great that their design process was getting shorter and shorter, yet the time it took them to actually create a compelling product took longer and longer.

In some ways, teams were missing the point. By skipping the early phases of design with the reason, “Well, we do not know if it will work until we test it on the hardware anyways,” design teams are now skipping an entire “Build-Measure-Learn” cycle that can happen in just the paper design phase of product development.

Product Designer Julie Zhuo once drew these images to describe the difference between a Junior Designer and a Senior Designer:

By jumping straight into prototyping in code, teams are behaving like a Junior Designer would, often asking engineering teams to chase leads as they emerge until they hopefully arrive at a solution. To get around this issue, my design teams have been following a few simple tips to ensure that we are always learning how to design for VR/AR/MR and over time, have cut down the iteration time needed to create a compelling product:

1. Make predictions. When you first start designing in VR/AR/MR, your designs will rarely work when you start implementing them in code. That is OK. However, encourage your design teams to still create and iterate on paper design documents and make predictions. Write down why the design might work in VR/AR/MR. Why might it fail? If it fails, what are the alternative designs that we could try? It is important to write down and create these hypotheses, so you can test and discuss the outcomes with the team when you do prototype in code. Over time, your design team will learn how to make better predictions, and paper designs will become more and more valuable because you will require less iteration time in code (which is arguably always more expensive than design iteration on paper!)
2. Learn new tricks. More than ever before, it is critical for designers to learn more about other fields like visual attention, social psychology, and even stage magic to understand the nuances of designing for VR/AR/MR. One of the most common challenges with VR is the notion that a designer no longer has control of the “camera,” and the user can look wherever they like. However, by learning about visual attention and stage magic, you can learn about how the human brain works, and exploit it. For example, a motion onset (something that moves) in peripheral vision will tend to capture attention. You can use tricks like this to guide the user’s attention where you want.
3. Evolve how you wireframe. Traditionally, a design team might do wireframes in 2D using an application like Sketch or Axure; however, these tools were designed for screen-based experiences and are typically optimized for web or mobile experiences. For some teams, simply creating wireframes for VR/AR/MR with these tools constrains their design thinking, resulting in experiences that feel pretty similar to a web or mobile application. Push your design team to be more creative with how they create their wireframes. One team, for example, started creating cardboard props for everything in an experience, and their team members roleplayed the interactions. It is pretty silly to watch a team member hold a cardboard prop (say, of a UI Menu object), and have to move around the room as another team member “grabs and moves” the UI object while walking around the room themselves — but, it re-creates what the experience really might feel like in VR or MR more than a 2D wireframe. It is a matter of time before tools like Sketch and Axure create features specifically for wireframing for VR/AR/MR; but, for now, push your teams to try different solutions that better emulate what the experience actually would feel like for a user.
4. Less is more. Many design teams are so excited about the new spaces they are exploring that they keep thinking, “Wouldn’t it be cool if..?” Each time they think of a new novel mechanic or feature, they want to squeeze it into the current product design. At every point in a project, always take a second look at your feature list and think, “How much value is this really adding for the end user?” “Is this something we only use once in the entire experience?” In most cases, you are better off going with fewer features and just choosing the ones that maximize user value.

Hopefully, these steps empower your design team to cut down the iteration time needed to create compelling products in VR/AR/MR. This is an exciting time for designers, and everyone is fighting to be the team that creates the next medium-defining experience; however, just because it is unexplored space does not mean we have to throw away everything we have learned about how we do design.

ABOUT THE AUTHOR:

Jeffrey Lin, Ph.D.

Dr. Jeffrey Lin was a Lead Product Owner and Lead Designer of the award-winning PC game League of Legends at Riot Games, one of Fortune’s Best Companies to Work For. He was also a Research Scientist at Valve Software, makers of the award-winning PC game Portal 2, and creators of the Steam platform. He obtained his PhD in Cognitive Neuroscience from the University of Washington where he was funded by the Howard Hughes Medical Institute. His design work has been featured in Wired Magazine, MIT Tech Review, The Verge, Scientific American, Times Health & Science, and Re/code. His research has been featured in numerous peer reviewed journals, including Nature.