How to redesign your AR app for persistent sessions. IKEA Place was one of the first ARKit app on the iOS app store One of the first ARKit apps to be released on the App Store was , a furniture shopping app, that let people virtually their own homes with IKEA’s furniture catalogue before making a purchase. IKEA Place design Since then, product visualization for furniture and home decor has become one of the most popular applications of AR on mobile phones and even big retailers like Lowe’s and Amazon have jumped in to experiment with the power of AR to influence furniture shopping behavior among consumers. If you’ve never tried one of these apps, check out IKEA Place or Housecraft . So what is the “Resume Session” problem? The “Resume Session” problem refers to a common problem faced by all AR interior design apps on the market today—It is that they have no way to save the positions of virtual furniture placed in a physical location from one app session to the next. That means every time a user accidentally or intentionally closes the app, they lose all progress and need to start re-designing their space from scratch! This post is about why this is a problem with current apps and how you can potentially solve it using a new SDK available for ARKit — Placenote SDK . Placenote SDK enables the persistent placement of AR content in any physical space. To start, let’s first look at how a basic AR interior design app is built. How do AR Interior Design apps work? These apps basically work by letting you virtually place 3D models of furniture on flat surfaces around you in AR. So, if you’re thinking of buying a new couch, you can see what it will look like from different angles and visualize whether the color and size of the couch suit your room. To understand how these apps work at a technical level, you need to understand how to build a simple ARKit-based iOS app that can detect horizontal planes and place 3D objects on it. If you’re new to this, that will show you how to build it. here’s a great tutorial Once you can build an app that can place a 3D object on a flat surface, you can simply replace the 3D object with a model of a chair, . like this one Now, at a very basic level, every furniture app has two main components **1. An interface to place and move 3D models on a flat plane**ARKit can easily detect flat surfaces and let you place 3D objects on them. Screen record of furniture placement in Housecraft on the App Store **2. A catalogue of 3D models to choose from**3D models will generally be provided by the furniture retailers. For the sake of prototyping, you can use free 3D models from websites like , or . Sketchfab Google Poly Turbosquid 3D models from Google Poly The combination of these elements is the basis for all interior design apps available on the App Store today and while many of these are beautifully well designed and fun to use, there’s one big problem with all these apps today. The Elephant in the Room If you’ve tried one of these apps on an iPhone or iPad, this will sound familiar to you. You painstakingly add various items of furniture around your room, align them correctly and choose just the right colors that match your decor. After 15 minutes of virtually decorating your room, if you accidentally (or intentionally) close the app and open it again…and , your design has disappeared and you need to redo everything again. Not only that, but you can’t share these designs with anyone, except taking a picture or video of the AR scene. POOF From a UX perspective, this isn’t a great experience given that the two main functions of most visual design tools are the ability to save and resume progress and collaborate with others. So let’s take a technical dive into what the problem is. The Problem with ARKit If you’ve worked with ARKit you know that each time an AR app is opened, ARKit sets an arbitrary (0,0,0) point at the phone’s current position and places all content in the scene relative to that point. This is because, although ARKit detects visual features in the environment to track the phone’s motion, it has no ability to “remember” these features. This means that you cannot build a system to save and resume your design session if the locations of your objects relative to the real world are important. You need to realign your furniture every time you open the app. In most user flows, this is unwanted friction. So what is the problem? The issue with ARKit is that the visual features it uses to track motion are not “distinct” enough to be repeatably found in the same place every time it observes an area. In order to save digital content in specific areas of a house, we need a way to permanently fix the positions of AR content in a physical space, so that every time an app is opened, the origin point of your scene snaps to its correct absolute position in the real world. Let’s call this feature “ ”. persistence Persistence in AR is the ability to let digital objects permanently retain their position in the real world across multiple app sessions. One way to solve this problem, is Placenote SDK . The rest of this post will teach you how to build an interior design app that incorporates persistent sessions in its workflow. Solving the Persistence Problem To give digital objects a persistent position in the real world, your app will need some representation of the real world (i.e. a map) and a way to find it’s position on that map. Apps like Pokemon GO did this by using GPS to geolocate Pokemon near specific real world landmarks like shops and parks. This kind of geolocation is sufficient for rough positioning but since GPS is so inaccurate, the positions of content can be between 5 to 20 meters off target every time you open the app. Further, GPS doesn’t work indoors which is where most “interior” design happens. Pokemon go uses GPS for geolocation with low accuracy. Placenote SDK takes a different approach to solving this problem. Rather than GPS, it uses a camera to visually scan a physical space and turn it into a map for retrieval. It also gives a camera the ability to position itself in a map by comparing its current image with a pre-created map. This way, Placenote implements a visual positioning system that is self sufficient (doesn’t need external beacons) and above all, . future works indoors How Placenote Does Visual Positioning Placenote divides the positioning process into 2 main steps — The camera is used to visually scan a physical area. Once mapping is complete, Placenote saves and uploads the map to the cloud, and returns a mapID that can be used to reload or share a map with multiple devices. Mapping: A pre-constructed map is downloaded using a mapID and Placenote attempts to match the image currently seen by the camera with a portion of the map. When it finds a match, it returns a position estimate of the phone/ tablet within that map. Relocalization: If you’re a computer vision geek like me, this will sound familiar to you. (SLAM). This system, while mostly seamless, will likely change the design of your user workflow a bit. Let’s look at how you can redesign an interior design app to incorporate the visual positioning paradigm. Redesigning your App for Visual Positioning Here is a potential redesign for an interior design app that incorporates a visual positioning workflow into its UX. 1. Designing a Room (and Mapping Simultaneously) With Placenote, the design stage needs to be modified slightly to run the mapping procedure in the background. That means, as a user is placing items of furniture in the room, Placenote can start building a 3D map of the environment by running a parallel feature extraction and mapping process in the background. The good news is, this can be done without ay extra prompts to the user. It should seamlessly fit into any existing workflow. If you’re coding in , you can run this line at the end of your viewDidLoad() function in your View Controller. This will start the mapping process immediately when a user opens the app. Swift on Xcode LibPlacenote.instance. startSession() Also add a new function to send image frames to Placenote from ARKit. func (_ session: ARSession, didUpdate: ARFrame) { session let : CVPixelBuffer = didUpdate.capturedImagelet : matrix_float4x4 = didUpdate.camera.transform image pose LibPlacenote.instance. (image: image, pose: pose)} setFrame The design stage can run mapping in the background (green dots are feature points) 2. Saving a Design Session “Saving” a session might be the only additional step added to your workflow if you don’t already have a “save” or “share” button in your app. Since no interior design apps today can really save designs in a specific location, there hasn’t been a real need for a “save” button. However, if you have a “share” feature that lets you capture a screen shot or video of a designed space, you can link the save function to this share button using this code snippet: LibPlacenote.instance. ( saveMap : {(mapId: String?) -> Void in savedCb print("Saved Map ID: " + String(mapId))LibPlacenote.instance. ()}, stopSession : {(completed: Bool, faulted: Bool, percentage: Float) -> Void in uploadProgressCb (completed) {print("Uploaded map!")}}) if In the context of Placenote’s mapping process, the save function saves and uploads a map created during the design process. Once the map is uploaded you will receive a that can be used to reload the map or share it with other devices to make your app more social. mapID From a development perspective, you will need to save the positions of content in your app as well. This can be done locally or on your own cloud backend using a simple JSON file. Save a design session by saving a map and the positions of content in the map. 3. Resuming a Design Session Once a map is saved, you can reload it whenever you want to open a saved design in that area. One way to incorporate this into your UX is to check if there is a previously saved mapID each time the app is opened (in the viewDidLoad() function). If a mapID exists, load the latest map and attempt to localize the device within that map. As soon as Placenote finds a match, the scene can be loaded exactly where it was left off so that the user can continue designing. You can add this line to your function in your View Controller. viewDidLoad() // if you saved a mapID in user defaults, retrieve it like this guard let savedID = defaults.string(forKey: "MapID")else { return false } LibPlacenote.instance. (mapId: “Map ID”, loadMap : {(completed: Bool, faulted: Bool, percentage: Float) -> Void in downloadProgressCb if (completed) { LibPlacenote.instance.**startSession**() }}) The session can be resumed as soon as the app is reopened. And that’s it! With a few small changes you can add a entirely new dimension to your AR experience, making it persistent, more social and overall a better solution for the interior design application it’s built for. The code examples above were in for Native SceneKit development, but Placenote SDK is available as a plugin as well. To learn more about Placenote or try the sample apps, check out . Swift Unity Placenote.com Getting Started with Placenote SDK You can learn more about how Placenote helps with building v apps here. I will be releasing the sample code for the app example above in the next few days so if you’re interested, sign up for an API key and try Placenote SDK for Unity or SceneKit, using the install guide linked below. irtual furniture design and staging _Click the link below to open your Developer Portal. Sign up for an account and create a new API key. Then return to…_placenote.com Get API Key and Choose Platform | Placenote How to Get in Touch To get in touch with me about Placenote, ARKit or anything AR/VR related, feel free to reach out at: Email me: neil [at] vertical.ai Placenote Slack Community: Join our slack group
