Disclosure: , which provide real-time APIs for developers, has previously sponsored Hacker Noon. Pusher Augmented Reality (AR) has a lot of interesting and practical use cases. One of them is location. With iOS 11, the ability to use ARKit to create AR apps and combine them with multiple libraries has opened a lot of possibilities. In this tutorial, we’re going to combine the power of ARKit, CoreLocation, and Pusher to create a geolocation AR app. Let’s think of a taxi service. Some services allow you to track on a map the car that is going to pick you up, but wouldn’t be great to have an AR view to see the route of the car and how it gets closer to you? Something like this: As you can see, the information to position the car in the AR world is not always accurate, both on the CoreLocation side and on the ARKit side, however, for this use case, most of the time it will be enough. Here’s what you’ll need: A device with an A9 or later processor (iPhone 6s or better, iPhone SE, any iPad Pro, or the 2017 iPad) iOS 11 (or newer) Xcode 9.1 A 3D model of a car (in DAE format) You can find free 3D models on sites like , , or . Free3D Turbosquid Google’s Poly The most common format is OBJ (with the its textures defined in a MTL file), which can be converted to DAE with a program like Blender. For this project I chose , which it’s available in DAE format. this model The math for this project is a bit heavy. I’ll dedicate more time to explain the operations related to geolocation than the ones related to rotating and translating a model with ARKit. If you don’t know about transformation matrices or how to convert your 3D model to the DAE format, take a look at my previous . tutorial about ARKit For reference, the source code of this project is on . Git Hub Let’s start by setting up a Pusher app. Setting up Pusher If you haven’t already, create a free account at . Then, go to your Dashboard and create an app, choosing a name, the cluster closest to your location, and as your front-end technology: Pusher iOS This will give you some sample code to get started: Save your app id, key, secret and cluster values. We’ll need them later. Finally, go to the tab, check the option and click on : App Setting Enable client events Update Through this app, the drivers will send their locations as latitude/longitude coordinates along with the direction they’re heading (in degrees) as a client . event But let’s not get ahead of ourselves, let’s set up the Xcode project first. Setting up the project Open Xcode 9 and create a new : Single View App We’re choosing this option because we are going to manually set up an AR view along with other controls. Enter the project information, choosing as the language: Swift Create the project and close it. We’re going to use to install the project’s dependencies. Open a terminal window, go to the root directory of your project and, in case you don’t have CocoaPods installed (or if you want to update it), execute: CocoaPods sudo gem install cocoapods Once installed, create the file Podfile with the command: pod init Edit this file to set the platform to iOS 11 and add the as a dependency of the project: Pusher’s Swift library # Uncomment the next line to define a global platform for your project platform :ios, '11.0' target 'ARKitCarGeolocation' do # Comment the next line if you're not using Swift and don't want to use dynamic frameworks use_frameworks! # Pods for ARKitCarGeolocation pod 'PusherSwift', '~> 5.1.1' end Once you’ve edited the Podfile, execute the following command to install the dependency: pod install In case version 5.1.1 (or later) is not installed (the output of the installation will tell you the installed version), you can update your CocoaPod repository and install the latest version of the library with the command: pod install --repo-update Now open the Xcode workspace instead of the project file. The workspace has the dependency already configured: open ARKitCarGeolocation.xcworkspace If you build your project at this point, a couple of warnings may show up, but the operation should be successful. Next, select the file Info.plist, add a row of type ( ) and give it a description. This is required for ARKit to access the camera. Privacy — Camera Usage Description NCameraUsageDescription We’ll also need a row of type ( ) This is required to get the location from your device’s GPS (only when the app is being used, not all the time): Privacy — Location When In Use Usage Description NSLocationWhenInUseUsageDescription Finally, configure a team so you can run the app on your device: Now let’s build the user interface. Building the user interface Go to Main.storyboard and drag an to the view: ARKit SceneKit View Next, add constraints to all sides of this view so that it fills the entire screen. You do this by pressing the key while dragging a line from the to each side of the parent view and choosing leading, top, trailing, and bottom to the superview, with a value of : ctrl ARSCNView 0 Next, add a text view and disable its and behaviors in the Editable Selectable Attributes inspector: Change its background color (I chose a white color with opacity): 50% Add a height constraint with a value of and leading, top, and trailing constraints with the value so it remains fixed to the top of the screen: 90 0 In import ARKit: ViewController.swift import ARKit Then, create two s, one to the scene view and another one to the text view: IBOutlet You’re ready to start coding the app, but before that, let me explain what needs to be done. However, if you’re already familiar with geolocation concepts or if you’re not interested, feel free to skip the next section. Understanding how the app works Imagine you are standing at some point in the world. It doesn’t matter where or in what direction you’re looking at. Your location is given by two numbers, latitude and longitude. Latitude is the distance between the North or the South Pole and the equator (an imaginary circle around the Earth halfway between the poles). It goes from to for places to the north of the equator, and to for places to the south of the equator. 0º 90º 0º -90º Longitude is the distance from the prime meridian (an imaginary line running from north to south through Greenwich, England) to a point at the west or east. It goes from to for places to the east of the prime meridian, and to for places to the west of the prime meridian. 0 180º 0º -180º For example, if you’re in Brazil, your latitude and longitude will be negative because you are on the southwest side of the Earth: And if you’re in Japan, for example, your latitude and longitude will be positive because you are on the northeast side of the Earth: This app will take into account your position and the driver’s position in a latitude and longitude coordinate system: But if it’s easier to you, you can think of your position as the origin ( , ): 0 0 You need to calculate two things: The distance between you and the driver The angle between the north (or south) line of the Earth and the line connecting you and the driver, which is called . bearing The distance will tell you how far you have to position the 3D model in the AR world. The bearing will help you create a rotation transformation to position your model in the right direction at the above distance. If we were talking about a simple x and y coordinate system, we could get those calculations by applying the and some simple trigonometry, with and operations. Pythagorean theorem sine cosine But we are talking about latitudes and longitudes of the Earth. And as the Earth is not a flat plane, the math gets more complex. The distance is calculated by calling just a of the class . It uses the which, from two different latitude/longitude pairs of values, calculates the distance by tracing a line between them that follows the curvature of the Earth. method CLLocation Haversine Formula On the other hand, we have to calculate the bearing between two different latitude/longitude pairs of values manually. : This is the formula atan2 ( X, Y ) Where equals: X sin(long2 - long1) * cos(long2) And equals: Y cos(lat1) * sin(lat2) - sin(lat1) * cos(lat2) * cos(long2 - long1) Another thing to consider is that for the matrix transformation, you’ll have to use instead of degrees as angle units. As the length of an entire circumference is equal to 2π radians ( ), one radian is equal to 180/π degrees. radians 360º So this is the plan. Using Pusher, the drivers will publish their location and direction they’re heading in realtime. Using CoreLocation, the AR app is going to get your location. It will also listen to the driver’s location updates. When a location update is received, using the formulas explained above, the app will place a 3D model of a car in a position relative to your location inside the AR world, and it will orient the model to the same direction the driver is heading. The app is only going to get your location once, so it assumes your location is fixed (which is true most of the time). In addition, an arrow emoji (⬇️) will be shown on top of the model at all times so you can spot it easily, and the text view you added in the last section will show the status of the app and the distance between you and the car. Now that you know what to do, let’s get into the code. Building the app with ARKit and CoreLocation Let’s start by defining two extensions. One to provide conversion methods to radians and degrees to all floating point types. Create a new Swift file, , with the following content: FloatingPoint+Extension.swift And another extension to create an image from a string. Create another Swift file, , with the following content (taken from this ): String+Extension.swift StackOverflow answer You’ll use this extension to create an image out of the arrow emoji (a string). It creates a rectangle of width and height , with a transparent background, to draw the string inside of it with a font size of . 100 100 90 Next, open the dialog and scroll down to choose the type: New File Asset Catalog Enter as the file name (confirming the use of the extension ): art.scnassets scnassets Now copy your model to this folder: Open the Scene Graph View, select the main node of your model and, in the properties tab, give it a name, which you’ll use to reference it in the code: Back to , let’s add the statements we’ll need: ViewController.swift import And the delegates the controller will use: Next, let’s add some instance variables. First, a to request the user location and another variable to store it: CLLocationManager Then, a variable to store the direction the drivers are heading, the distance between them and the user, and the status of the app: Whenever a new value for the distance or the status is set, the text view will be updated. Notice that the distance is calculated in meters. Next, a variable to store the root node of the car model and the name of this node, which should be the same than the one you set at the SceneKit editor: You’ll also need the original (first) transformation of that node: Why? To calculate the orientation (rotation) of the model in the best possible way. Ideally, the driver’s device will always give you the correct heading so you can take the first received reading, rotate the model in that direction, and then calculate the next rotations relative to the first one. However, if the first reading is wrong (which happens sometimes), the next rotations will be wrong even if the rest of the readings are correct. So you always need to calculate the orientation as if it was the first time you rotate the model, because once you rotate the model a certain angle the following rotations will be done relative to that angle. Resetting the rotation to won’t work either because of the way transformations work (matrix multiplication). 0º Finally, you’ll need to store the Pusher object and channel to receive the updates: Notice the value of the option. authMethod You’ll be receiving the updates through a private channel. They need to be authenticated by a server. However, at development time, you can use the option to bypass the need to set up an auth endpoint as part of a server. inline You can learn more about the object’s options . If you need it, you can learn how to create an . here authentication endpoint on this page In the function, set up the SceneKit scene and the location service: viewDidLoad Next, configure the AR session: The option will set the y-axis to the direction of gravity as detected by the device, and the x- and z-axes to the longitude and latitude directions as measured by Location Services. gravityAndHeading For the users position, when they have authorized the use of the location services, you have to request the location (the method is used so the location is requested only once): requestLocation Once the user’s location is received, take the last element of the array, update the status, and connect to Pusher (it doesn’t make sense to connect to Pusher before having the users location because all the calculations will be wrong): In the method you subscribe to and, when a client-new-location event is received, extract the driver’s latitude, longitude, and heading and update the status and location of the 3D model with the method : connectToPusher private-channel updateLocation In , create a object to calculate the distance between the user and the driver. Remember that the is calculated in meters: updateLocation CLLocation distance If this is the first update received, will be , so you have to instantiate the model: self.modelNode nil Next, you need to move the of the model to its center in the y-axis, so it can be rotated without changing its position: pivot Save the model’s transform to calculate future rotations, position it, and add it to the scene: Notice that there’s no need to create an to add the node as a child of it. An ARAnchor gives you the ability to track positions and orientations of models relative to the camera. ARAnchor But in this case, it’s better to work with the child directly. Mostly because you cannot delete or change the position of the whole manually -only of its children. ARAnchor Finally, create the arrow from an emoji, position it on top of the car (using the y-axis, I got the value by trial and error), and add it as a child of the model (so it stays with it at all times): This is the definition of the method (taken from this , modifying the width and height of the plane so the arrow can be properly seen): makeBillboardNode StackOverflow answer Now, if this is not the first update, you just need to position the model, animating the movement so it looks nice: To position the model, you just need to rotate first, then translate it to the correct position and scale it: The order is important because of how matrix multiplication works ( is not the same than ). a*b b*a In ARKit, rotation in the y-axis is counterclockwise (and handled in radians), so we need to subtract and make the angle negative. This is the definition of the method : 180º rotateNode I scale the node in proportion to the distance. They are inversely proportional -the greater the distance, the less the scale. In my case, I just divide by the distance and don’t allow the value to be less than or great than : 1000 1.5 3 I got these values from trial and error. They will vary depending on the model you’re using. To translate the node, you have to calculate the transformation matrix and get the position values from that matrix (from its fourth column, referenced by a zero-based index): To calculate the transformation matrix: You use an identity matrix (you don’t have to use the matrix of the camera or something like that, the position and orientation of the driver are independent of your position and orientation. You have to calculate the bearing using the formula explained in the previous section: atan2 ( sin(long2 - long1) * cos(long2), cos(lat1) * sin(lat2) - sin(lat1) * cos(lat2) * cos(long2 - long1) ) Using an identity matrix, get a rotation matrix in the y-axis using that bearing. The distance is given by the z-axis, so create a four element vector with the distance in the z position to get a translation matrix. Multiply both matrices (remember, the order is important) to combine them. Get the final transformation by multiplying the result of the previous step with the matrix passed as an argument. All this is done with the following methods: About how to rotate in the y-axis, the method returns the inverse of the matrix because rotations in ARKit are counterclockwise. . Here’s an answer from Mathematics Stack Exchange that explains rotation matrices pretty well And that’s it, time to test the app. Testing the app The first time you run the app, you’ll have to give permissions to the camera: And to the location service: And wait for a few seconds so the app can get the location and connect to Pusher. To test it, you’ll need someone that publishes location events while driving. On , you can find an app for iOS that publishes location events. this GitHub repository It uses CoreLocation, and the code is pretty similar to the one shown in the previous section but it requests the location information every one or two seconds. As a note, for the heading measurement, it’s important to hold the device in the direction the driver is heading. For a quick test, you can use the to manually send some location coordinates (that you can get ) every two seconds: following Node.js script from this site Once you have Node.js installed, you just have to copy this script to a file, let’s say , create a file with the command: publish.js package.json npm init Install the with: Pusher Node.js library npm install --save pusher Enter your Pusher and location info and execute the script with: node publish.js Once the app starts receiving location events, the 3D model of the car will appear in the direction where it is in the real world (with a small size if it’s far from you): Conclusion You have learned how to combine the power of ARKit, CoreLocation and Pusher to create an AR app. You can add more features to make it more useful: Adding more information to the screen. For example, you can . convert the coordinates of the driver to an address Adding a map so, in addition to seeing the 3D model moving in the world, you can see in which street the car is at any given time. Add more car models. Change the mechanism to get the car orientation. For example, by using deltas of the location. However, keep in mind that the app depends on the quality of the information received. In my tests, for a few seconds after starting the driver’s app, the heading information was completely wrong, and overall, the position was off a few meters. ARKit occasionally gets confused too. Sometimes this can be a problem, and it is another area of improvement. However, we’re just at the beginning. Without a doubt, these frameworks will be improved over time. Remember that you can find the entire project on this and you can contact me if you have questions. GitHub repository Originally published on . Pusher’s blog
