A comprehensive resource list for building engaging Augmented and Virtual Reality experiences using Web technologies This post originally appeared on the Halo Labs blog A few months ago I joined Halo Labs to help build our AR/VR design and prototyping platform. After the first interview, I received a “take home” exercise, to build a simple VR scene. While I was excited to build a VR app for the first time, I was also a bit afraid. I’m coming from a web development background and I never worked on a VR project before. To my surprise, it took me roughly three hours to finish the exercise, and it even looked pretty good (well, at least to me…). Over the past six months, whenever I told someone that I’m developing a VR product using Web technologies, I got puzzled looks. The will take place in two weeks, and I thought it’s a great time to show web developers that . first annual WebXR week VR & AR + Web technologies = ❤ The main goal of this post is to allow Web developers to enter the AR/VR world quickly and easily. The approach I have taken is not to write a guide about a specific technology or library, but rather to build a “curriculum” that will take you from zero to expert, so you would be able to build complex AR/VR experiences. Even if you don’t plan to develop AR/VR experiences, reading this guide will give you a glimpse into the current state of the WebXR world. Let’s begin. Starting the journey — Getting to know the VR world Before we begin, let’s line up the terms: Virtual Reality is the use of computer technology to create a environment, so when you are in VR, you are viewing a completely different reality than the one in front of you. Augmented Reality (AR) on the other hand, is an version of reality created by adding digital information on your real-life reality (like in Pokemon GO). The term XR is often used to described any of the two. simulated enhanced While you could easily skip this step and jump directly to the WebVR frameworks, investing some time learning the basics of the XR world will greatly improve your learning speed and understanding down the road. The following resources will help you gain some background on VR and AR development, as well as the required (very basic) mathematical background: course by Udacity — This free course is a great place to start. The course introduces the main VR platforms available today and explains how they work while teaching some basic (but important!) VR terms. Introduction to Virtual Reality — Knowing the meaning of those basic XR terms will help you better understand articles and XR frameworks documentation. Another good resource is website. I really like their infographics section, as it helped me to get my head around some VR terms and topics. VR/AR Glossary The VR Glossary Basic 3D math — The subject I was most afraid of when I entered the VR world was math. I’m not a big math fan, and I thought that dealing with 3D requires a thorough math knowledge. Luckily, it turned out that I was wrong. The frameworks I’ll present below are relatively “high level” and doesn’t require any mathematical background. From my experience, the only important thing to know before moving on is . the difference between the left and right-handed coordinate systems Rendering 3D content on the web Now that we have some basic understanding of the XR world, we can start looking at XR web frameworks. The main framework for XR development is (supported by Mozilla). The next section will go deeper into A-Frame, but before that, it’s important to understand how A-Frame is built in order to use it effectively. Let’s dive in! A-Frame In 2007, Mozilla first introduced Canvas 3D, which allowed rendering interactive 3D graphics on the web. The next step was to expose an API, and by 2009 started the WebGL Working Group. The first version of the specification was released in 2011.But what exactly is WebGL? to quote from Mozilla: The Khronos Group WebGL enables web content to use an API based on OpenGL ES 2.0 to perform 2D and 3D rendering in an HTML in browsers that support it without the use of plug-ins. WebGL programs consist of control code written in JavaScript and shader code (GLSL) that is executed on a computer's Graphics Processing Unit (GPU) _canvas_ In short, WebGL is an API that enables rendering 3D content in the browser, without the need to use plug-ins. Today, all main browsers support the WebGL API, so we can safely use it to render 3D content on the web. The main problem? writing WebGL is hard and tedious. It’s enough to see the required to display simple 2D shapes to get discouraged. The solution? using Three.js. amount of code The aim of the project is to create an easy to use, lightweight, 3D library. The library provides <canvas>, <svg>, CSS3D and WebGL renderers.(source: ) Three.js GitHub page Three.js is a “high level” library that simplifies the creation of WebGL environments. It handles the lower level programming for you and lets you focus on building the scene. To see how much it simplifies the development, take a look at the code example below, which on the screen: renders an animated 3D cube In the above code example, we initialize the scene, the camera (which is our “eyes” in the scene) and the renderer. Then, we create a box geometry, which defines the cube , a material which defines how it’ll and finally we create a cube by combining the two into a mesh. After that, we add the cube to the scene and attach a simple animation to constantly rotate it. Finally, we render the scene. shape look This is a big improvement from the hundreds of lines of WebGL code, but it’s still not very simple. In order to display a cube, you have to understand what’s a material, a mesh, a renderer and how they all connect together. In addition, presenting 3D content is not the end of the story. In order to create “serious” VR content, we’ll also have to allow user input, physics, integrating with various VR headsets, and more.While all of these can definitely be built in three.js, it’ll be difficult to do so without deeper understanding in the 3D and VR domains.But don’t worry! A-Frame to the rescue! A-Frame — VR for the people The A-Frame framework was created in 2015 by the Mozilla VR team in order to allow web developers and designers to author 3D and VR experiences with HTML without having to know WebGL. A-Frame is based on HTML and the DOM, which makes it very accessible and easy to use. While using only the HTML layer allows getting an impressive result, HTML is only the outermost abstraction layer of A-Frame. **Underneath, A-Frame is an entity-component framework for three.js that is exposed declaratively.**A-Frame’s true power embodied in the last sentence, so let’s break it down to make sure we understand it: A-Frame is an entity-component framework for three.js To quote : Wikipedia ( ) is an architectural pattern […] An ECS follows the principle that allows greater flexibility in defining entities where every object in a scene is an entity (e.g. enemies, bullets, vehicles, etc.). Every Entity consists of one or more components which add additional behavior or functionality. Therefore, the behavior of an entity can be changed at runtime by adding or removing components. Entity–component–system ECS Composition over inheritance Let’s clarify with an example:Say I want to build a vehicle with the ESC pattern. Vehicle entity, composed of multiple components First, I would need a vehicle entity, which is practically an object with an Id. Next, I would use components to define the vehicle’s look and behavior. I would have multiple components such as color, wheels, seats, and engine. Finally, composing all those components into the entity we created earlier will give us a functional vehicle entity. While the above example is very simplistic, it should give you a rough idea of what an entity-component architecture is. A-Frame allows writing Three.js code in an ECS way, which makes VR development much easier. One main reason is that ECS makes it very easy to reuse components — so If I built a component, most chances you can use it too. The A-Frame community is taking advantage of that, and there is a which is available for you to use. Now that we understand the first part of the sentence, let’s examine the second part : big library of components that is exposed declaratively This part refers primarily to the HTML abstraction layer. This layer allows us to build a scene declaratively, which means we create a scene by defining it should do, and not it should do it. It can be done thanks to the underlying layers which allow us to create components. After we create a component, we can just say we want to do — the component already knows (that’s what the component code is all about). what how what how Now that we understand what A-Frame is and how it works, let’s see A-Frame’s Hello-World example: Copied from . You can move in the scene using the keyboard. A-Frame’s official examples In this example, every tag under is a Primitives are just syntactic sugar for entities with default components. For example, the primitive is an entity with multiple components (such as depth, height, and width) added by default. Each HTML property is a component added to these entities— we add (or override the default) position, rotation, color, and shadow components to our box entity. a-scene primitive. a-box A-Frame provides a set of primitives to help you create basic scenes quickly and easily, and you can also create your own primitives. I won’t get deeper into A-Frame since it’s not the purpose of this article, but here are some good resources to jumpstart your A-Frame journey: — A-Frame official documentation is quite comprehensive and I highly recommend reading it. It probably contains the answers for all your “beginner questions”, so make sure to check it out before searching other places. A-Frame documentation — An interactive A-Frame course built by A-Frame creators. Using Glitch, the course provides step-by-step exercises to help you get started. A-Frame school — Despite the fact that using A-Frame with React might result with poor performance, I find it to be a great combination (actually, that’s our setup here in Halo Labs). If you like React, this tutorial is using and it’s a great place to start. (p.s — If you prefer Angular, check out ) Creating Your First WebVR App using React and A-Frame aframe-react angular-aframe-pipe Augment your skills So far we talked about VR, but what about AR?Since we still don’t have any broad consumer AR headsets today, the existing WebAR solutions mainly focus on . mobile AR Today, there are three main libraries you can use to build AR scenes, all three work with A-Frame, but each has different capabilities. Let’s go over them one by one: AR.js provides both an A-Frame and a three.js extension that allows building AR scenes. AR.js was built with WebGL and WebRTC, so it’s the only one of the three that works with almost every smartphone, regardless of its OS version. AR.js marker-based If you want to play with AR.js, check out Akash Kuttappa’s . article aframe-ar The common way to build mobile AR application is to use (for Android) or (for IOS) both are native SDK’s. In order to provide a way to use those SDK’s capabilities (like surface detection) on the web, Google released two experimental apps: and . which are actually browsers that expose a JavaScript API to the aforementioned capabilities. On top of that, they released a library called , which provides three.js helper functions for building AR experiences. Since A-Frame is built on three.js, was created in order to provide an easy to use A-Frame wrapper. How easy? all you need to do is to change your A-Frame scene tag from to and you have a working AR scene! ARCore ARKit WebARonARCore WebARonARKit three.ar.js aframe-ar <a-scene> <a-scene ar> If you want to play with aframe-ar, check out Uri Shaked’s . excellent article aframe-xr is based on and they both created by Mozilla. Its main difference from aframe-ar is that it complies with the proposed using the . The main implication is that aframe-xr enables building “progressive experiences” — experiences that change according to the device in use. Simply put, it allows you to move between AR and VR seamlessly.Here in Halo Labs we are big believers in the WebXR API, so aframe-xr is our chosen framework. aframe-xr three.xr.js WebXR Device API webxr-polyfill If you want to learn more about the WebXR API, check out . Also, Mozilla has regarding Progressive WebXR. Dan’s blog post a great blog post After playing with WebAR for a while, it’s obvious that it’s not yet mature. However, even today, using the libraries I mentioned above, you can build some impressive AR experiences. Down The Rabbit Hole So far we’ve covered all the basics. That’s enough to create basic AR/VR experiences and gain some confident in your abilities, but if you want to create some more complex stuff, you’ll need to extend your knowledge.Here are some resources to help you get deeper understanding: — A Udacity course teaching basic principles of 3D computer graphics (meshes, transforms, materials, and more). Interactive 3D Graphics — A series of posts written by Rachel Smith, teaching Three.js basics with a lot of code examples. Beginning with 3D WebGL — An introduction to Three.js. @necsoft talks about all the important stuff in one blog post. Three.js 101: Hello World! — The lower you go on abstraction level, the greater the mathematical knowledge required from you. From my experience, if you want to strengthen your math knowledge, Khan Academy is your best friend. Linear algebra — Khan academy — An example of how to implement a VR Minecraft demo using A-Frame. This step by step guide will help you better understand how to build a robust VR app with A-Frame. Building a Minecraft demo with A-Frame Content As we all know, on the internet- . This is also true for the process of creating XR experiences. In order to build convincing XR experiences, 3D assets are required. While the amount of free and easy 3D creation tools increases rapidly, many of us prefer to use existing content instead of creating it ourselves. Currently, there are 2 main sources for free 3D assets: content is king — A library containing thousands of 3D models for use in VR and AR applications. Poly models are published under license ( ), which means you can use them freely, even for commercial use, as long as you provide attribution to the author. Google Poly Creative Commons CC-BY 3.0 — A marketplace of 3D models that contains more than 2M models. Sketchfab contains thousands of free models, also licensed under Creative Commons license ( ). Sketchfab models are usually high quality, and as a result — “weigh” more. Sketchfab CC-BY 4.0 Both sites support multiple 3D formats, include . You can choose the required format when downloading the model. glTF It’s very easy to add models to an A-Frame scene, by using the primitive (there are as well): a-gltf-model loaders for other formats <a-gltf-model src={http://model.url} /> The easiest way to avoid CORS problems is to host your assets on a publicly accessible CDN. Mozilla provides one for free: https://cdn.aframe.io/ Final words As web developers, the AR/VR world often seems inaccessible. The truth is that the required tools for us to enter and create in this world already exist. Libraries like three.js and A-Frame allow us to use our existing web development skills to build rich VR scenes. In addition, complementary libraries add capabilities that enable the creation of AR scenes, and even progressive experiences that adapt to the capabilities of the device on which they run. Hopefully, my post will help other web developers enter the AR/VR world and together we can finally build the ! :) Metaverse
