The UX Workflow for Hololens & Mixed Reality

A couple of days ago, I was talking with a designer friend about an HoloLens project I was finishing with a company. That was an exciting time, but the more we were talking, the more he was telling me that even if he had the opportunity to work with HoloLens, as a Product Designer, he wouldn’t know where to start.

Well, it was exactly the same for me at first. The chance is, my first project was more a proof-of-concept than anything else, so it was okay to make some mistakes and trying to figure what could be a good workflow on such a project.

Maybe other designers were like us, trying to figure where they should/could start. After working on a second HoloLens project, I thought it would, maybe, be useful to share what is my UX workflow for HoloLens/Mixed Reality at this moment, and improve it as well.

As a side note, this article is not about best practices for HoloLens UX/product design, neither given thoughts about the question « Is Mixed Reality the next big thing? ».

Vision: define your goals

If you’re coming from a mobile/web product design experience, you may already have heard something similar. Defining the goal(s) of your project will help you having a better vision of what you should do and how you could make compromises depending on the situation. HoloLens is not a magic device, and even if I don’t wish you so, you could face some struggles.

But more on, having clear goal(s) will give you a clear thought about the right platform/tool you should use for your project.

Mixed Reality: is HoloLens the right tool for your project?

I can’t speak for everyone, but I heard a lot people talking about HoloLens as a platform. I disagree. Well, as a personal point of view, I tend to see HoloLens, and even more, Mixed Reality, more as a tool rather than a platform.

From this, I tend to ask « Is this the right tool for your project? ».

Few months ago, I was working with a start-up. They had the opportunity to get to work on HoloLens and had a very specific experience in mind they wanted to bring to people. But the more we were talking, the more it seemed Mixed Reality wasn’t the tool we should use for this kind of experience, mostly because it didn’t convey that sense of immersion Virtual Reality can bring (but one day, I’m sure Mixed Reality will).

As for it, some will talk better about this than me, so just a quick reminder of what are Augmented Reality, Virtual Reality and Mixed Reality:

AR: Enhances reality by adding layering information over a direct view of actual reality or real world datas (such as navigation cues…)without physical interactions, aka Google Glass, mobile and things like Pokemon Go.

VR: Replace reality and immerse user into a virtual world. Can be stationary or room-scaled, such as HTC Vive or Oculus Rift.

MR: Call it AR++. Combines digitally-render, interactive objects with the real environment, which can be treated as real-work counterparts. It’s HoloLens and the upcoming Magic Leap.

Table of Elements: in-depth palette for composing HoloLens experiences

If you’re reading this article, you may already know what HoloLens is capable of. As for myself when I first started working on it, I do know things, but I couldn’t figure out all the options which were available to me. I was looking for a sort of table of Elements with design patterns. After many research, I tried to summarize HoloLens, but I find out that the people from 8Ninths were also working on something similar, and they brought online a « design research document, intending to serve as a dynamic foundation that will forge the first generation of HoloLens experiences ».

Following what I’ve learn from their researches, I really invite you to consult their work and blog.

I added the Mixed Reality Design Methods to their Table of Elements. Use this as a reminder so you always know the palette of options available to you and your project. On top of that, it will be very useful to share your ideas with your team, and especially with your developers.

Table of Elements for Mixed Reality

On top of that, do not hesitate to make your own researches and take a look at what others are doing with HoloLens. Microsoft HoloLens Youtube Channel, Hologram Reviewer and Next Reality are good places to begin with.

Storyboard: conceptualize and share your ideas for a 3D space

As a designer, I’m sure you’re used to sketching and wire-framing. If you already have directed short movies or machinimas, you may even be used to storyboard.

This will be your best tool for a Mixed or Virtual Reality project.

No one will ask you to be the best drawer ever when it comes to storyboard, but you should learn how to share 3D ideas on a 2D surface and a sheet of paper, differentiate what is from the physical world, what are holograms and how the user will interact with to have a good experience.

Storyboard are usually black and white stuff, with arrows to get a sense of action and movement. But as we’re working on digital stuff, you may use colors on top of that. From my experience, here’s how I draw a storyboard for an HoloLens experience, using 1st and 3rd person view:

• Black & grey shades: used for the physical world
• Blue & green: used to illustrate holographic stuff
• Orange & yellow: used to highlight elements and movement
• Purple: Spatial sounds and audio clues

See? I am no Leonardo Da Vinci.

Prototype: test your ideas as soon as possible

Even if Virtual Reality isn’t that new, it’s already hard to find prototyping tool for this, so guess what, it’s even harder to find one for Mixed Reality. I wish we could have some kind of InVision for AR/MR, and sure it will come soon enough as the market will get bigger and bigger. For now, here’s the 3 methods I’ve used so far:

Use your imagination: Yes, it sounds goofy as hell, but it can work. If don’t have an easy access to the HoloLens, know nothing about 3D Engine, or aren’t very efficient with glue and paper, it still can help you. For my 1st project, I remember wearing glass and trying to figure how I would move around an hologram and how I would interact with it. It worked, for a time.

Paper work for real world: If you can keep in mind holograms mimic real world counterpart, then why not use paper models and even toys? That’s how I proceeded with my 1st project as well, I used glue and paper to create stuffs, and I sticked them to the ceiling with strings. With that, I could turn around and get the illusion (with a bold use of imagination). Once again, it’s not perfect, but it helps.

3D tools for 3D world: If you already are familiar with what a vertex, an edge and polygons are, then that’s a big plus. If not, we will talk more about that later in this article. I always thought designers should code, but more of that, designers should have knowledge in every field they are related to, so they can easily speak and understand people from different fields. If you’re going to prototype for 3D experiences, let’s use 3D Software like Cinema4D, 3Ds Max to create 3D models, even if you’re only using primitives, and implement them in Unity for HoloLens. Microsoft made a great case of providing tutorials about this: Microsoft 101

GrayBoxVR is a tool for quickly constructing and visualizing VR scenes using basic shapes and advanced 3D models, it’s not actually related to Mixed Reality, but if you want to get familiar with 3D construction and a first simple path to prototyping for 3D, you should give it a try.

How to share & give inputs to engineers and product managers

You may assume to create the UX & UI part for a given project, and you, generally, won’t be alone. You will work with product managers (well, maybe not for now since it’s the really beginning for Mixed Reality) and most of all, engineers.

This part was, I have to admit, really hard at first. While I was still figuring the workflow, I didn’t really know what sort of informations I should share and how. What type of design documents would be the best to show & illustrate the flow and the actions. From now on, here’s what I share and how I do it.

Flow Diagram

Remember that Table of Elements with talked earlier from 8Ninths? Now will be a perfect moment to use it!

By definition, the Flow depicts the movement. When we talk about Flow in UX, we are generally talking about how users will navigate through a system to perform task(s) and complete a process. From that, you may already be familiar with the idea of flow diagram, depicting a chart of screens with paths to go from screen A to B to C.

With Mixed Reality (and VR as well), you don’t really have a set of screens as you’re into the experience itself. So how can you process? By combining the storyboard you made earlier and the table of content.

Through the same way flow diagram for mobile illustrates the path through a system, Flow Diagram for Mixed/Virtual Reality will depict each action the user will be tasked to perform a process and how he can interact with the holograms. The storyboard already illustrated the process, and now, it’s time to complete it with the design method you’re using, the user input(s), the physical space, the holographic form & cie…

Here’s an example:

I assume you will make your own look more… Professional?

Concept art

Okay, to be honest, this part is not necessary, since the Storyboard and the Flow Diagram from above should already do the work. But it can only help if that’s something you’re able to do. I’m not sure it will be the case, but if you’ve some concept artists in your team or if you know how to make concept art on your own, that would be a good plus.

Concept art will help illustrate key actions of your experience with clear visuals and indications, helping you to communicate the mood around this project and give inspiration to your team. I actually did not already create this in my UX process for Mixed Reality, but when I was working on a game for Virtual Reality, it worked really well as one of my friend was able to get me some incredible assets.

It really is taking inspiration from the video game industry workflow, and since that, Microsoft’s Galaxy Explorer for HoloLens is the best example I could find when it’s time to show inspiration using concept art for an HoloLens experience:

UI & 3D models

Consider this part as a bonus, since for this kind of project, 3D Artist will be involved from the beginning and they should be the one working for the most time on this part. Sure, you will produce UI stuff, but as we’re talking about a 3D experience, most of it will be polish by a 3D Artist.

But as I said before, I truly think a designer should have knowledge in every part of the process so he/she can easily communicate with everyone and maybe do some stuff by himself/herself.

3D models are made from various tools, tools you can buy or get for free. You can pick one like Cinema4D (the one I use), 3Ds Max (the one I used), Maya, which aren’t free. Blender is a free 3D software you can begin with. If you’re using Windows 10, Microsoft just gave a really useful tool called 3D Builder which can help you create simple but effective 3D scenes & models and get you the basics about 3D modeling.

3D models are made of polygons. Polygons are faces made of 3 edges (minimum) or more. One edge is made of 2 vertices, aka dots associated with three spatial coordinates.

That’s the basics. If you want to get more about the process of creating 3D assets, I recommend to you this very accessible article wrote by Nick Pettit: Asset Workflow for Game Art - 3D Modeling

3D engines used for Virtual and Mixed Reality may have a limited budget in terms of how many polygons it can draw and show, that’s why working with a 3D Artist will always be the best.

Conclusion

From there, Product and UX Designers, you should get a good starting point for your next HoloLens project! Well, at least, I hope this article helped you finding a good workflow to begin with, as I wished to when I first started working with Mixed Reality.

This one may not be perfect, but so far, that is the one I figured out and tried on several projects with great results. As I would summarize it:

• Goals & Vision
• Table of Elements (Research) for Mixed Reality
• Storyboard
• Prototyping
• Flow Diagram

Bonuses (for you):

• Concept Art
• UI & 3D models