Disclosure: Pusher, which provide real-time APIs for developers, has previously sponsored Hacker Noon.
Augmented Reality (AR) has a lot of interesting and practical use cases. One of them is location.
With iOS 11, the ability to use ARKit to create AR apps and combine them with multiple libraries has opened a lot of possibilities.
In this tutorial, we’re going to combine the power of ARKit, CoreLocation, and Pusher to create a geolocation AR app.
Let’s think of a taxi service. Some services allow you to track on a map the car that is going to pick you up, but wouldn’t be great to have an AR view to see the route of the car and how it gets closer to you?
Something like this:
As you can see, the information to position the car in the AR world is not always accurate, both on the CoreLocation side and on the ARKit side, however, for this use case, most of the time it will be enough.
Here’s what you’ll need:
The most common format is OBJ (with the its textures defined in a MTL file), which can be converted to DAE with a program like Blender.
For this project I chose this model, which it’s available in DAE format.
The math for this project is a bit heavy. I’ll dedicate more time to explain the operations related to geolocation than the ones related to rotating and translating a model with ARKit.
If you don’t know about transformation matrices or how to convert your 3D model to the DAE format, take a look at my previous tutorial about ARKit.
Let’s start by setting up a Pusher app.
If you haven’t already, create a free account at Pusher. Then, go to your Dashboard and create an app, choosing a name, the cluster closest to your location, and iOS as your front-end technology:
This will give you some sample code to get started:
Save your app id, key, secret and cluster values. We’ll need them later.
Finally, go to the App Setting tab, check the option Enable client events and click on Update:
Through this app, the drivers will send their locations as latitude/longitude coordinates along with the direction they’re heading (in degrees) as a client event.
But let’s not get ahead of ourselves, let’s set up the Xcode project first.
Open Xcode 9 and create a new Single View App:
We’re choosing this option because we are going to manually set up an AR view along with other controls.
Enter the project information, choosing Swift as the language:
Create the project and close it. We’re going to use CocoaPods to install the project’s dependencies. Open a terminal window, go to the root directory of your project and, in case you don’t have CocoaPods installed (or if you want to update it), execute:
sudo gem install cocoapods
Once installed, create the file Podfile with the command:
Edit this file to set the platform to iOS 11 and add the Pusher’s Swift library as a dependency of the project:
# Uncomment the next line to define a global platform for your project platform :ios, '11.0' target 'ARKitCarGeolocation' do # Comment the next line if you're not using Swift and don't want to use dynamic frameworks use_frameworks! # Pods for ARKitCarGeolocation pod 'PusherSwift', '~> 5.1.1' end
Once you’ve edited the Podfile, execute the following command to install the dependency:
In case version 5.1.1 (or later) is not installed (the output of the installation will tell you the installed version), you can update your CocoaPod repository and install the latest version of the library with the command:
pod install --repo-update
Now open the Xcode workspace instead of the project file. The workspace has the dependency already configured:
If you build your project at this point, a couple of warnings may show up, but the operation should be successful.
Next, select the file Info.plist, add a row of type Privacy — Camera Usage Description (
NCameraUsageDescription) and give it a description. This is required for ARKit to access the camera.
We’ll also need a row of type Privacy — Location When In Use Usage Description (
NSLocationWhenInUseUsageDescription) This is required to get the location from your device’s GPS (only when the app is being used, not all the time):
Finally, configure a team so you can run the app on your device:
Now let’s build the user interface.
Go to Main.storyboard and drag an ARKit SceneKit View to the view:
Next, add constraints to all sides of this view so that it fills the entire screen. You do this by pressing the
ctrlkey while dragging a line from the
ARSCNView to each side of the parent view and choosing leading, top, trailing, and bottom to the superview, with a value of
Next, add a text view and disable its Editable and Selectable behaviors in the Attributes inspector:
Change its background color (I chose a white color with
Add a height constraint with a value of
90and leading, top, and trailing constraints with the value
0so it remains fixed to the top of the screen:
ViewController.swift import ARKit:
Then, create two
IBOutlets, one to the scene view and another one to the text view:
You’re ready to start coding the app, but before that, let me explain what needs to be done. However, if you’re already familiar with geolocation concepts or if you’re not interested, feel free to skip the next section.
Imagine you are standing at some point in the world. It doesn’t matter where or in what direction you’re looking at.
Your location is given by two numbers, latitude and longitude.
Latitude is the distance between the North or the South Pole and the equator (an imaginary circle around the Earth halfway between the poles). It goes from
90ºfor places to the north of the equator, and
-90ºfor places to the south of the equator.
Longitude is the distance from the prime meridian (an imaginary line running from north to south through Greenwich, England) to a point at the west or east. It goes from
180º for places to the east of the prime meridian, and
-180ºfor places to the west of the prime meridian.
For example, if you’re in Brazil, your latitude and longitude will be negative because you are on the southwest side of the Earth:
And if you’re in Japan, for example, your latitude and longitude will be positive because you are on the northeast side of the Earth:
This app will take into account your position and the driver’s position in a latitude and longitude coordinate system:
But if it’s easier to you, you can think of your position as the origin (
You need to calculate two things:
The distance will tell you how far you have to position the 3D model in the AR world.
The bearing will help you create a rotation transformation to position your model in the right direction at the above distance.
But we are talking about latitudes and longitudes of the Earth. And as the Earth is not a flat plane, the math gets more complex.
The distance is calculated by calling just a method of the class CLLocation. It uses the Haversine Formula which, from two different latitude/longitude pairs of values, calculates the distance by tracing a line between them that follows the curvature of the Earth.
On the other hand, we have to calculate the bearing between two different latitude/longitude pairs of values manually. This is the formula:
atan2 ( X, Y )
sin(long2 - long1) * cos(long2)
cos(lat1) * sin(lat2) - sin(lat1) * cos(lat2) * cos(long2 - long1)
Another thing to consider is that for the matrix transformation, you’ll have to use radians instead of degrees as angle units. As the length of an entire circumference is equal to 2π radians (
360º), one radian is equal to 180/π degrees.
So this is the plan.
Using Pusher, the drivers will publish their location and direction they’re heading in realtime.
Using CoreLocation, the AR app is going to get your location. It will also listen to the driver’s location updates.
When a location update is received, using the formulas explained above, the app will place a 3D model of a car in a position relative to your location inside the AR world, and it will orient the model to the same direction the driver is heading.
The app is only going to get your location once, so it assumes your location is fixed (which is true most of the time).
In addition, an arrow emoji (⬇️) will be shown on top of the model at all times so you can spot it easily, and the text view you added in the last section will show the status of the app and the distance between you and the car.
Now that you know what to do, let’s get into the code.
Let’s start by defining two extensions.
One to provide conversion methods to radians and degrees to all floating point types. Create a new Swift file,
FloatingPoint+Extension.swift, with the following content:
And another extension to create an image from a string. Create another Swift file,
String+Extension.swift, with the following content (taken from this StackOverflow answer):
You’ll use this extension to create an image out of the arrow emoji (a string). It creates a rectangle of width
100 and height
100 , with a transparent background, to draw the string inside of it with a font size of
Next, open the New File dialog and scroll down to choose the Asset Catalog type:
art.scnassets as the file name (confirming the use of the extension
Now copy your model to this folder:
Open the Scene Graph View, select the main node of your model and, in the properties tab, give it a name, which you’ll use to reference it in the code:
ViewController.swift, let’s add the
import statements we’ll need:
And the delegates the controller will use:
Next, let’s add some instance variables.
CLLocationManager to request the user location and another variable to store it:
Then, a variable to store the direction the drivers are heading, the distance between them and the user, and the status of the app:
Whenever a new value for the distance or the status is set, the text view will be updated. Notice that the distance is calculated in meters.
Next, a variable to store the root node of the car model and the name of this node, which should be the same than the one you set at the SceneKit editor:
You’ll also need the original (first) transformation of that node:
To calculate the orientation (rotation) of the model in the best possible way.
Ideally, the driver’s device will always give you the correct heading so you can take the first received reading, rotate the model in that direction, and then calculate the next rotations relative to the first one.
However, if the first reading is wrong (which happens sometimes), the next rotations will be wrong even if the rest of the readings are correct.
So you always need to calculate the orientation as if it was the first time you rotate the model, because once you rotate the model a certain angle the following rotations will be done relative to that angle. Resetting the rotation to
0º won’t work either because of the way transformations work (matrix multiplication).
Finally, you’ll need to store the Pusher object and channel to receive the updates:
Notice the value of the
You’ll be receiving the updates through a private channel. They need to be authenticated by a server. However, at development time, you can use the
inline option to bypass the need to set up an auth endpoint as part of a server.
viewDidLoad function, set up the SceneKit scene and the location service:
Next, configure the AR session:
The option gravityAndHeading will set the y-axis to the direction of gravity as detected by the device, and the x- and z-axes to the longitude and latitude directions as measured by Location Services.
For the users position, when they have authorized the use of the location services, you have to request the location (the requestLocation method is used so the location is requested only once):
Once the user’s location is received, take the last element of the array, update the status, and connect to Pusher (it doesn’t make sense to connect to Pusher before having the users location because all the calculations will be wrong):
In the method
connectToPusher you subscribe to
private-channel and, when a client-new-location event is received, extract the driver’s latitude, longitude, and heading and update the status and location of the 3D model with the method
If this is the first update received,
self.modelNode will be
nil, so you have to instantiate the model:
Next, you need to move the pivot of the model to its center in the y-axis, so it can be rotated without changing its position:
Save the model’s transform to calculate future rotations, position it, and add it to the scene:
Notice that there’s no need to create an ARAnchor to add the node as a child of it. An ARAnchor gives you the ability to track positions and orientations of models relative to the camera.
But in this case, it’s better to work with the child directly. Mostly because you cannot delete or change the position of the whole
ARAnchor manually -only of its children.
Finally, create the arrow from an emoji, position it on top of the car (using the y-axis, I got the value by trial and error), and add it as a child of the model (so it stays with it at all times):
This is the definition of the
makeBillboardNode method (taken from this StackOverflow answer, modifying the width and height of the plane so the arrow can be properly seen):
Now, if this is not the first update, you just need to position the model, animating the movement so it looks nice:
To position the model, you just need to rotate first, then translate it to the correct position and scale it:
The order is important because of how matrix multiplication works (
a*b is not the same than
In ARKit, rotation in the y-axis is counterclockwise (and handled in radians), so we need to subtract
180º and make the angle negative. This is the definition of the method
I scale the node in proportion to the distance. They are inversely proportional -the greater the distance, the less the scale. In my case, I just divide
1000 by the distance and don’t allow the value to be less than
1.5 or great than
I got these values from trial and error. They will vary depending on the model you’re using.
To translate the node, you have to calculate the transformation matrix and get the position values from that matrix (from its fourth column, referenced by a zero-based index):
To calculate the transformation matrix:
You use an identity matrix (you don’t have to use the matrix of the camera or something like that, the position and orientation of the driver are independent of your position and orientation.
You have to calculate the bearing using the formula explained in the previous section:
sin(long2 - long1) * cos(long2),
cos(lat1) * sin(lat2) - sin(lat1) * cos(lat2) * cos(long2 - long1)
Using an identity matrix, get a rotation matrix in the y-axis using that bearing.
The distance is given by the z-axis, so create a four element vector with the distance in the z position to get a translation matrix.
Multiply both matrices (remember, the order is important) to combine them.
Get the final transformation by multiplying the result of the previous step with the matrix passed as an argument.
All this is done with the following methods:
About how to rotate in the y-axis, the method returns the inverse of the matrix because rotations in ARKit are counterclockwise. Here’s an answer from Mathematics Stack Exchange that explains rotation matrices pretty well.
And that’s it, time to test the app.
The first time you run the app, you’ll have to give permissions to the camera:
And to the location service:
And wait for a few seconds so the app can get the location and connect to Pusher.
To test it, you’ll need someone that publishes location events while driving.
On this GitHub repository, you can find an app for iOS that publishes location events.
It uses CoreLocation, and the code is pretty similar to the one shown in the previous section but it requests the location information every one or two seconds.
As a note, for the heading measurement, it’s important to hold the device in the direction the driver is heading.
For a quick test, you can use the to manually send some location coordinates (that you can get from this site) every two seconds:
Once you have Node.js installed, you just have to copy this script to a file, let’s say
publish.js, create a
package.json file with the command:
Install the Pusher Node.js library with:
npm install --save pusher
Enter your Pusher and location info and execute the script with:
Once the app starts receiving location events, the 3D model of the car will appear in the direction where it is in the real world (with a small size if it’s far from you):
You have learned how to combine the power of ARKit, CoreLocation and Pusher to create an AR app.
You can add more features to make it more useful:
However, keep in mind that the app depends on the quality of the information received.
In my tests, for a few seconds after starting the driver’s app, the heading information was completely wrong, and overall, the position was off a few meters.
ARKit occasionally gets confused too. Sometimes this can be a problem, and it is another area of improvement. However, we’re just at the beginning. Without a doubt, these frameworks will be improved over time.
Remember that you can find the entire project on this GitHub repository and you can contact me if you have questions.
Originally published on Pusher’s blog.