Apple’s Latest Tech Announcement, Augmented Reality, and Afterthoughts

It’s that time of year again, when all of modern nerdome comes together to eagerly anticipate the latest Apple announcements. What’s the new iPhone going to look like? What new features and innovations will it present us with? But most importantly how much will this enrich our lives and how much will it diminish our bank accounts.

In the last couple of years the announcements of new iPhones and mobile tech have been somewhat disappointing. They’ve been filled with the same bravado that always accompanies an Apple announcement, “the best [insert thing here] we’ve ever made”, but when you look beyond the polished exterior, you realize that these improvements and advancements are relatively minor, cosmetic and more recently, Emoji centric.

However, through the first part of today I have been anxious with the hope that this trend will have ended. It’s 10 years since the original iPhone announcement, and in those years, pretty much EVERY phone has become an iPhone.

Android users and Apple haters would likely have issue with that last statement, but it is true. Remember when we had flip phones and sliding phones and qwerty phones? A billion different designs, styles and features? well now we have different variations of the same iPhone-like interface with slightly different OS options and varying degrees of screen and bezel size.

Rant aside, for its 10th year anniversary there have been great expectations placed on the latest iPhone announcement with a lot of the rumor focusing on augmented reality, especially because of the ARKit announcement earlier in the year. Augmented reality was invented the same year as the internet (1968) but has only just this year exploded into consumer consciousness, with Snapchat, Facebook, Google and Microsoft all putting their faith, and money firmly in this “future” tech. It’s no wonder that Apple would also throw its hand in the ring and the iPhone 10 (or iPhone X as it is), was set to be the leap to change how we interacted with each other, digital information and the world around us. Just like the change the initial iPhone created a decade ago.

These augmented reality rumors were not idle ones either, three years ago Apple acquired Metaio, the foremost company dealing with augmented reality at the time. Metaio also happened to have a vast number of patents, including ones for hardware that would make augmented reality easier, and less of a resource drain for mobile devices. Apple has also employed quite a large number of augmented reality experts over the last year or so and snapped up all the AR patents they could. Including for AR glasses. Though, while it seems like all this has been put to great use, Apple’s announcement today around AR really felt like it was more of an afterthought than an innovation they wanted to showcase.

Apple’s AR Announcements And What They Mean

So on the back of today’s Apple announcements here is a list of the new features the iPhone 8/8 plus and X will be able to handle, and a little explanation on why I think they were important for the future of augmented reality.

With full transparency in mind, I’ve been working in the augmented reality field for over four years, and founding my AR company Happy Goon, the company that creates augmented reality greeting cards, that come to life. This means it’s likely some of the views present here will be skewed towards our application of augmented reality to some respect and not the infinitely vast possibilities augmented reality has the potential to encompass.

Added processor power and GPU

The new iPhone 8/8 plus will boast drastically increased processor power and a GPU, this is nothing but good for AR as both of these allow for richer, more dynamic experiences… especially 3D ones. The iPhone X goes one step further with its processing power, so no end of good things there as well.

2 cameras (depth sensor, of sorts)

For the 8/8 plus there are the 2 rear facing cameras which help with analyzing depth (like for portrait mode). This is useful for AR because it has been notoriously hard for the computer to figure out if a thing is very big and far away or very close but small. This is a far cry from Google’s Tango certified devices sensors, but it is something.

Facial Tracking and the True Depth Camera System

The facial tracking on the iPhone X looked fantastic. As consumers, we are really accustomed to face filters and effects thanks to snapchat, but what Apple demonstrated wasn’t just a 2D or 3D model or some funky effects placed on, and tracking to, a face. Their all too short demo showed a mask actually mapped ON Craig Federighi’s face. This was easily the highlight of all their AR demonstrations for me, but it did leave me wondering if there would be applications of it outside of novelty.

I’m not sure if I’m jaded here, as my head has been firmly inserted in the augmented reality world for over four years now, but I didn’t find Apple’s presentations overly impressive. The Emoji face tracking for instance, I met with a company doing exactly this thing 2 years ago. The iPhone X’s depth system and facial tracking ability makes this an amazingly improved experience and would definitely make AR applications fantastic, but it did lack that WOW factor. It wasn’t new… at least to me.

Jony Ive and the Animoji feature of the iPhone X

What I Would Have Hoped To See

Augmented reality inside the native camera app

This feature would be huge. One of the main problems with current augmented reality is that an individual app needs to be created to disseminate the experience/s. This is very difficult to do if you don’t have a large network of resources. What’s more, even if you’re able to get people to download your AR app, it would result in one phone (like mine is currently) having upwards of 10 to 20 augmented reality apps on it, all utilizing the iPhone’s camera to adapt and enhance the world the camera sees while needing to be individually opened to see each. Not the best user experience.

If Apple were to build augmented reality experiences into the native camera app, in a similar way to how they built app functionality into their native messenger app, it would mean all augmented reality experiences are housed in the same space. Even better than that, the camera app is an app that everybody uses on a near daily basis meaning more engagement all round. Apple would essentially take away one of the major barriers between people who create awesome augmented reality experiences and the people who want to use awesome augmented reality experiences.

Advancements on ARKit Functionality

ARKit is great, but it is still limited, especially when compared to the vast capabilities Metaio had when Apple purchased them 3 years ago, including a Graphic User Interface to build respectably complex AR experiences.

Without question, ARKit’s ability to determine what is a surface, or the ground, and almost seamlessly place a digital object in that space is fantastic, and an advancement on what Metaio could do at the time. But AR is more than just placing 3D models in real space. 2D and 3D object tracking is also important, as we interact with objects in our world a lot more than the spaces in our world. Currently object or tracker recognition is not an easy thing to do with ARKit.

Depth Sensors On The Back Camera Too

I know there is likely a reason for the absence of this hardware on the back camera setup. Maybe cost, maybe space, maybe battery life… But if Apple were to have more comprehensive depth sensors on the back camera, it could be used to enhance the capability of ARKit to an incredible new level. Digitally placed things could interact with objects and be obscured by them. Think of a digital robot, or alien, or Pokemon hiding behind a real life tree and peaking out playfully at you. That level of interaction between physical space and digital experience is definitely the AR of the future and one that needs widespread hardware support to achieve.

Final Thoughts

Overall, I was disappointed with today’s Apple announcements. I had expected a big augmented reality reveal that would shake their competitors to the bones. Instead I saw augmented reality as an afterthought in their minds, something that needed to be said to show they were keeping up with the trends, but nothing more. I am, however, still excited about the future of augmented reality and Apple augmented reality specifically. I have watched through enough Apple announcements to know they often leave the release of new technology until they are certain it is of a sufficiently high quality. After all, they weren’t the first touchscreen phone, but they were the defining touchscreen phone.

Besides, if they don’t make the next advancement in augmented reality tech or platform, there is still a thriving creative community, and a few giant tech companies that will.

Fingers crossed for next year.