The difference between AR, VR, MR, XR and how to tell them apart

For the past couple of years after getting into the AR/VR industry, I’ve been asked this question many times. While I’ve done my best to simplify things down to answer the question, it takes a bit more explanation to really make things clear. And that’s what I’ll try to do in this post.

What is Virtual Reality?

To start, let’s look at what is virtual reality (VR) and the goal and the role it serves. VR has been an idea that was mostly science fiction, from movies like The Matrix to books like Snow Crash to anime like Sword Art Online. However VR has been in existence longer than most people realize, even before some of the movies and books that made the concept popular. However for this post I wouldn’t dive too deeply into its history, and will instead focus on the modern VR (post Oculus).

The ultimate goal of VR is true immersion — the feeling that you are in another place without been physically there. And it comes at no surprise that it’s the video game industry that demanded VR technology first and give birth to modern VR technology. Players wish to enter the digital world where they already spend a lot of their times and want a way to be closer to the story and the action.

Gamers playing video games

Currently, most video games are played using an input device like joysticks or keyboard and mouse. Actions in the game is communicated to the player through monitors, sound, controller vibrations etc. What this means is that any action the player want to do must first translate into an intermediate step first, which breaks the immersion. In VR, the player controls the game directly using their head, hands and receives the visual and auditory information back through a VR headset. The VR headset provide a larger field of view than most monitors out there. This means the player have direct control and feedback of digital world interaction. The actions are more intuitive and direct.

The value of virtual reality is a more intuitive way to interact with the digital world and a deeper immersion in the digital experience. The content for VR was already there, and it’s only recently that modern VR came about. VR is a hardware innovation. VR is comparable to the invention of mouse and keyboard. In conclusion, virtual reality is an input and output device that provide intuitive control and deeper level immersion of digital experiences.

This also helps to look at the application of virtual reality. While VR could be used in many situations, it’s not always the best solution. VR is best used for storytelling and conveying experiences. So things like taking a tour through a remote location (or non-existent location), training experiences (which otherwise would be expensive or difficult to do) such as operation of heavy equipment, or fantasy experiences (think digital tourism of fictional world). When evaluating if VR is the right solution, ask yourself whether additional immersion and better user experience would create additional value for the end user. At Babylon VR, this is what we do to help our customers explore the use cases of web based virtual reality technology.

To take virtual reality further, the industry need to develop better ways to interact with the digital world and make the user experience more immersive. The headset will need higher resolution, lower motion-to-photon latency (or more commonly known as less lag) and bigger field of view. And on the design side designers have to figure out how to create better user experiences and what are the best applications. As VR will completely transform how we interact with digital world just like how the mouse changed how we interact with computers.

What is Augmented Reality?

Next, we’ll take a look at augmented reality or shortened as AR. AR has already been used for years even before Pokemon Go made it popular. In fact, Facebook’s ability to tag your friends on photos, Instagram face filters, are all examples of AR applications. Unlike VR where you are bringing the user into a digital world, augmented reality is used to bring digital information into the real world.

Snap chat effects

Things became a bit complicated when Apple released their ARkit. ARkit does more than just overlay a digital image over the camera, it also automatically detects the ground and calculate the lighting around the virtual object. This brings the standard for AR a lot higher as it incorporates complex computer vision algorithms. Furthermore, we now have similar thing going with Google Lens and Samsung Bixby, which uses computer vision and machine learning to provide digital information straight from your camera.

The technology to overlay information on top is already plentiful and been around for years, but the key differentiation of AR is how much information it can extract from the real world. As a result, AR is a software innovation. It relies heavily on machine learning and computer vision rather than hardware. The hardware required have been already there (just need a camera and a display) for decades. Augmented reality is the ability for software to extract data from visual information of the real world.

The ultimate goal of AR is to provide contextual information to the user as they need. Application of AR include highlighting traffic signs and providing GPS information while driving, visualizing products in your room before buying, providing in context help while troubleshooting a broken dishwasher etc. When determining if an AR experience is the right solution, ask whether or not you are looking to gather data from the environment and provide additional information on top of that.

To take AR further, there need to be more development on the software side. More advanced machine learning to recognize object and provide semantic meaning to the camera vision. And also better recognition of objects using cameras already available. As long as there is a camera, a display and enough processing power, AR can be used regardless of the hardware.

What is Mixed Reality and Extended Reality?

Mixed reality (MR) and extended reality (XR) are more abstract concept than AR and VR. Since they aren’t technology itself, but combination of these technologies. The general understanding is that MR is in between of AR and VR and XR is an over-compassing term that includes AR, VR and MR. However this just makes everything confusing and add irrelevant complexity to everything. One of the most used explanation on the web is through the use of reality-virtuality spectrum

The Reality-virtuality spectrum is a common graph used to explain things, but it is misleading and confusing

There are a couple things that make it confusing. First of all, ARkit would be counted as mixed reality in this classification while it’s called “AR”kit. And similarly with The Void, which market itself as virtual reality experience. In addition, this classification makes VR and AR mutually exclusive, which is not the case. So for a better explanation, it would be better to use a Venn diagram:

While we could still use the MR and XR label, it does not add much value and only add more confusion. (For these of you into set theory, you can use AR∩VR and AR∪VR). The important thing to note is that AR and VR can both coexist. In summary, if the experience requires special hardware to view (such as a headset), it’s VR. If it uses computer vision to read in real world information, it’s AR. If it involves both scanning the environment for data and displays through a headset for deeper immersion, it’s both AR and VR.

Ultimately, both of these technologies will become inseparable. As both AR and VR are aimed at enhancing the human experience. Imagine in the future, you’ll be using your headset to help you navigate your way back home and check e-mails in an electric self-driving car, then turn it in to VR mode and enjoy an immersive Netflix experience in the comfort of your own living room.

I hope this provides people a better understanding of AR and VR. Let me know if there are any parts that you want a deeper explanation on and I’ll add it in.