Every June, tech bros, pension portfolio managers and Twitter threadbois the world over turn their eyes to Santa Clara as hosts its annual Worldwide Developer Conference (WWDC, or ‘dub-dub’). Apple Among other things, dub-dub is best known as being the birthing ground of almost all of Apple’s suite of ubiquitous consumer technology. The original iPhone, iPad, Apple Watch, and Apple TV and subsequent upgrades have all been launched at the event. This year, this stable of consumer devices is joined by an entirely new product for the first time since 2015. That new product is the VisionPro. There is nothing I can tell you about the VisionPro or Apple’s vision for the future of extended reality that a quick browse of , or the internet at large cannot. The point of this article is instead to try and observe: Twitter LinkedIn how we got here; why people are building computers for your face and the many divergent paths this may lead us on. Content Overview XR: A Brief History Headsets Today: Code New Worlds The Next Frontier for Spatial Computers Requests for Startups XR: A Brief History For the uninitiated, XR refers to - a catch-all blend of (AR, e.g Pokemon Go), virtual reality (VR, e.g. Oculus Rift), and ‘capital R’ Reality (e.g. Google Street View). Mixed reality is another term that exists largely on the same dimension as AR and thus will mostly be ignored here. extended reality augmented reality The modern vision for XR begins around 1935. This was the year in which published . In this short story, protagonist Dan Burke is disillusioned with the reality in which he finds himself. To remedy this, he puts on a pair of glasses that transport him to a world of eternal youth and happiness. The only catch is that he obeys its rules (_T&C_s, in modern parlance). Stanley Weinbaum Pygmalion’s Spectacles Fast forward a few decades and we begin to see the first builds of XR in real environments. This begins with Morton Heilig’s ‘ ’ (excluded from the timeline below), a sort of 4D cinema experience that provided the viewer with smells and a vibrating chair in addition to the sounds & sight of the film. Sensorama Also excluded from the timeline below is the first military application of XR. In 1961, became the first headset with motion-tracking technology. Philco Headsight Though it has since been discontinued, the Oculus Rift was the first headset to gain real commercial appeal. Following Oculus, many of big tech’s household names joined the action. Curiously, most had different visions for who the primary customer base for these headsets would be. The dominant expected use case for most was in immersive entertainment applications, more specifically in gaming. This was the path followed by Oculus, Sony through their , and with the Meta Quest line of products. PlayStation VR Meta Others chose to focus predominantly on enterprise. Early pioneers in this segment included and These early movers focused their aims on things like enterprise collaboration platforms (much like some of the metaverse plays described below), emergency response training, and medical education. HTC Vive Magic Leap. Then, the hype cycle. Corporations the world over began to envision what it would look like if we took our lived environment but made it *virtual*. The vision was essentially , but this time with less porn and more corporate work stuff. metaverse SecondLife In the wake of COVID-19, the rise of remote work created a supposed need for collaborative digital environments. joined the party with its own industrial metaverse. Accenture launched a ‘metaverse services’ division. Facebook changed its name to Meta and began spending $1bn per month on this vision, culminating in this brilliant . Microsoft presentation I have had fun panning the metaverse, but the hype cycle around it was an important moment in raising awareness for the arrival of what Apple would refer to this week as spatial computing. As this space evolves, there is every chance that many of these ideas come back into vogue. Other use cases for commercial XR varied from the practical to the benign. Some of them are beginning to become commonplace in popular web back-ends. You take your pick as to which fits which description from the below: Virtual tourism Virtual home inspections Virtual clothing fitting (If you’re going to watch any of these videos, do yourself a favour and make it this one) Corporate training & education Surgical simulations Product design & prototyping Disney-fied sporting telecasts. Most of the existing devices listed above tried to achieve some sort of balance with the use cases they were targeting. The most successful to date by sales, the Meta Quest, did so by positioning hard as a gaming device. Normally, this would be a lesson. However, if anyone is to buck this trend it would be Apple. Apple has a broad suite of already ubiquitous products (1.6bn active iPhones, to name one) that can and will serve as secondary endpoints for VisionPro applications. While Apple’s inaugural demos for the VisionPro were all hosted within working environments, it is more than likely that it will end up being a general-purpose VR. I foresee its role as being a form of spatial augmentation for Apple’s existing ecosystem of products (e.g Maps, FaceTime, Siri et al) as its vision for computers shifts modalities. More on other potential applications later in this piece. Headsets Today: Code New Worlds Spatial Computing Rather than ride the wave of today’s tech buzzwords, Apple used its informational market-making power to stamp authority on a new one at WWDC. https://twitter.com/tim_cook/status/1665806600261763072?embedable=true While it sounds intuitive enough on the surface, let’s delve a little bit deeper into what this term may actually convey going forward. Besides the obvious branding benefits of coining a buzz phrase, the use of the term ‘computing’ feels extremely deliberate. It represents a logical next step from previous eras of ‘desktop computing’ and ‘mobile computing’. Just as Apple has done with the previous generations of personal computers, it will aim to make this concept and terminology ubiquitous. To quote Tim Cook from WWDC: “In the future, you’ll wonder how you led your life without augmented reality” Besides the change in hardware, how is spatial computing functionally different from its predecessors? Arguably the most important immediate shift will be from working in 2D to working in 3D. Tasks such as visualization, workspace navigation (i.e. clicking and moving things), and collaboration will all be enabled in ways not seen under previous modalities. In the longer term, and are more likely to prove the groundbreaking features of spatial computing. contextual awareness interconnectivity refers to the ability of the computer to; Contextual awareness i) adapt existing recommendations and notifications to a user’s sensory context (i.e recommendations based on location and movement patterns, automatic changes to screen presentations based on user habits, etc) and ii) provide customized (possibly agentic) assistance to users as they complete tasks in extended reality. refers to the user’s ability to extend extended reality beyond the device. Beyond just working and collaborating on holographic interfaces, users can control and interact with other devices in their environment. To take a boring example, users can change smart home settings from within the device. Interconnectivity In the longer run, interconnectivity may see Siri’s role as a virtual assistant evolve from voice commanded search aggregator to a living assistant that can help complete tasks from within the device based on the user’s previous habits and preferences. As the baseline capabilities of autonomous agents advance, the combination of Apple’s distribution network and the familiarity of Siri may make it the killer technology for bringing personal agents to the world en masse. Information Presentation As mentioned above, Apple’s demos of the VisionPro focused almost exclusively on working environments. Make no mistake, this first generation is very much geared toward information workers. How will they use it? Firstly, there is obvious and trodden ground. Apple will re-do and re-design all the enterprise XR solutions we have seen before. They will provide resources for institutions to train their employees, students, and leaders. They will provide a bevy of try-on solutions, gaming applications, and cloud TV solutions that we have seen before. One relatively novel example that was showcased as part of the launch was the ‘Minority Report’ style computer interfaces. Instead of having to boot up a physical device every time you want to work on a file, access it from anywhere in the cloud in an instant. But how does this make the core experience of personal computing itself different? Some early ideas: at any given point in time Unlimited monitors with information work (speech, physical movement etc) Multimodal forms of interacting Intelligent browser navigation through sensory (e.g eye) tracking Entirely automatically tailored to user preference custom 3D screen presentations (boring but important) 3D data visualization Each of these will present its own universe of custom ideas that will take the place of extensions and app stores before it. AppleLM This tweet from founder Sulaiman Omar feels prescient. Apple does indeed have a history of sacrificing first-mover advantage for the sake of developing a better product over a longer timeframe. The VisionPro is a perfect example of this, being released 13 years after the first commercially available Oculus Rift. Cognosys AI Integrating language models into its headsets will open up a number of possibilities for Apple that aren’t available to incumbents without their own proprietary LLMs (note: Meta’s LlaMa may put them in the same boat - Balaji Srinivasan has published a highlighting their potential role in the spatial computing race). great thread https://twitter.com/balajis/status/1666506719084843009?embedable=true&thread=true Among the possibilities that this may usher in: trained on users’ real-life habits. Virtual gaming experiences Immediate through the facial ‘browser’. queries built into the headset and trained on user habits. Agentic virtual assistants that recommend actions at a point in time. Spatial recommendation engines Real-time translation. allowing users to re-arrange objects in their space. Environmental augmentation that plays out immediately as the user speaks it. Generative content creation But could it be done more elegantly… The Next Frontier for Spatial Computers Many a commentator proclaimed the death of Google Glass upon seeing it. Who would want to wear a computer on the face? Why would you opt into looking a bit like a dork? Many have noted that not a single Apple executive even tried a VisionPro on as part of the launch. There has to be a better reason for strapping one on than “because everyone else is”. The key question in the long run then becomes: Why would anyone want to strap an XR computer to their face when you can have one inside your brain? The classic counter to this would be the idea of ‘ ’. When everyone is wearing an XR mask, we can reproject photons to make it look as though no one is. Because reprojection involves creating entirely new photons altogether, people can even make themselves look entirely different from what they actually do without the mask. This would become a killer app for catfishing. reprojection The techno-optimist’s concept of living in augmented reality is akin to living in a perpetual lucid dream. Sleep mask when you go to bed, extended reality mask when you get out of it. Examine exhibit A below: The current paradigm of XR relies on these exoskeletal aids for users to navigate alternative realities. The reality is that we are already so close to no longer needing to depend on these aids. Enter the brain-machine interface. BMIs In the long run, the envisioned reality is likely to trump any form of mixed reality that exists today. I use the term envisioned reality here to describe any form of environmental alteration that is; a) completely customizable to the user’s requirements or demands and b) requires no wearable aids. The most obvious conduit for envisioned reality today is (BMIs). For a longer primer, I cannot recommend Tim Urban’s piece on ‘ ’ from as far back as 2017. brain-machine interfaces Wizard hats for the brain BMIs offer many of the same ‘spatial computation’ benefits as tools like the VisionPro but without the need for strapping a weighty device to the face. existing prototypes for the first commercial BMIs involve invasive implants that, understandably, make many uncomfortable (even if the reward is superhuman intelligence and memory). What about when these become less invasive, more accessible, and thus more palatable to the human public? Neuralink’s Extended reality abounds. Requests for Startups Mentions of batteries were oddly absent from Apple’s inaugural launch of the VisionPro. It is less obvious for ‘in the moment’ demos and usage than other design features like interfaces, compute, and weight (as referenced in ) but equally important when used day-in-day-out. Better battery systems for today’s XR devices. this thread by Kyle Samani Not since first dates moved from cafés to online chat rooms has the core wiring of how humans interact with one another been so fundamentally rocked. How can behavior be moderated in a world where people can make themselves appear to be anything? (note: Apple already seems to be working on some form of protocol which may protect against this). What kind of content becomes more pervasive when the range of presentation methods expands so broadly? How do people come together online? Social infrastructure for an XR world. proof-of-identity SecondLife, Minecraft, and Roblox all created huger-than-expected businesses off the back of secondary marketplaces for digital assets. These marketplaces were enabled by creative tools baked into the very fabric of the platforms themselves. Arm the Rebels: Creative Tools for XR. While there are many developers today already working on designs for new experiences etc for headsets and metaverse-style projects, how can we make this process more accessible for the layman (more specifically the 8-year Roblocker) to create worlds of their own imagination? What will be the second coming of Minecraft for the augmented reality or ‘envisioned reality’ paradigm? A large part of the appeal of virtual/augmented/envisioned reality is the idea of living a different life from the one you have now. Much like Dan Burke in anyone will now be able to optimize their lived experiences within virtual worlds. Designs for Lived Experiences (i.e Qualia). Pygmalion’s Spectacles, In the event that such technologies do become ubiquitous, this creates a giant universal market for synthetic instances of subjective experience. The design scope is theoretically infinite. How many ways can you experience pure ecstasy? Nostalgia? The pool of rewards for those who can bring these kinds of experiences to the world through some form of extended reality are enormous. qualia - Also underappreciated may be the development of tools for users to capture or develop these qualia themselves. Is there a market for the ‘recording’ of one’s own lived experiences to share with others? How can people engineer new experiences to take to market and compose atop other open-source qualia? How can we provide guarantees of privacy for personal qualia if they begin to be used in recommendation engines? For the curious, check out the to learn more. Qualia Research Institute One thing that XR still has in common with other computing paradigms before it is its top-down nature. Users are at the whims of Apple for how they want to interact with their phones. If they want to protest this, they can select another one of the depressingly finite options on the market. To date, composability and customisability in hardware have been extremely limited. Open-Source Hardware & XR Software Marketplaces. But what if we were to open-source the development of headsets themselves through decentralized labs? Active participants could work to modify and iterate upon different hardware specifications to their preference. Modular designs could allow for customization at the layman's level. Just as importantly, these open-source labs would have no incentive to create closed-loop systems for software development. Developers could experiment with, ship, and deploy code that could be available to all headsets at the rate that they can build it. Such open-source development would represent a step change in the way people interact with both software and hardware. For some inspiration, check out the great work the team at is already doing. Auki Labs Linked to the point above, the top-down software ecosystem of today means that incentives built into software tend to be warped. Rather than going all out on user utility, market incentives corner developers into building sub-optimal applications that aim to maximize metrics like ‘average session time’ or ‘clickthrough rates’. Private User Agents. The opportunity that open-source development studios would have to outcompete legacy tech companies by building in private, natural language user agents that act as functional assistants to their users is immense. Also published here. The lead image for this article was generated by HackerNoon's AI Image Generator via the prompt "Augmented reality headsets"
