Build Shared Augmented Reality Experience for Android using Sceneform and ARCore Cloud Anchors

In Google I/O 2018, some of the most exciting features with respect to Augmented Reality were ARCore, Sceneform, and Cloud Anchors. Augmented Reality (AR) is essentially overlaying graphics on top of reality. In this post, we’ll build a project with which we can share AR experiences between multiple apps and devices. We can also retrieve these experiences for upto 24 hours after they are hosted. You can build this project either with a single phone or with multiple phones. The testing mechanism for both is mentioned towards the end of this post. The code for this project can be found at the . This project is a part of our course (url contains promotional coupon) on Udemy. More on this course can be found in the parting shot of this post. CloudAnchors git repo ARCore and Sceneform for Android Augmented Reality Background and Terminology ARCore ARCore is Google’s platform for building augmented reality experiences. Using different APIs, ARCore enables your phone to sense its environment, understand the world and interact with information. ARCore works using Motion Tracking, Environmental Understanding and Light Estimation. Sceneform Sceneform is a high level library over ARCore that allows Android developers to work with ARCore without learning 3D graphics and OpenGL. It helps with importing and editing models, detecting planes, handling User Experience and building the AR Scene. Cloud Anchors An Anchor can be created at a Trackable Position and Orientation in ARCore. The Cloud Anchors API allows us to store Anchors on the cloud and retrieve them at a later point. What We Will Build We will host a cloud anchor, then resolve it using the Anchor ID. After the Anchor is resolved, we will recreate the AR scene by placing the 3D model on top of the retrieved anchor (position and orientation). Prerequisites Android Studio 3.1 or above ARCore Supported Device or Emulator Methodology The complete project can be found at the . CloudAnchors git repo Step 1: Setup Project Create a new Android Studio Project, Select API Level 24: Android 7.0 or higher and add an Empty Activity. After the project is built, setup the Gradle Scripts. Go to Gradle Scripts -> build.gradle(Module:app) Add support for Java 8 and Sceneform. Add compileOptions at the end of android section, right after buildTypes compileOptions { sourceCompatibility JavaVersion.VERSION_1_8 targetCompatibility JavaVersion.VERSION_1_8 } Add dependency for Sceneform at the end of dependencies section implementation "com.google.ar.sceneform.ux:sceneform-ux:1.4.0" Click Sync Now. After the gradle build finishes. go to app -> manifests -> AndroidManifest.xml. In the manifest section request permission for camera and internet. Add meta data to application section for AR so that the App Store will be able to recognize the app as an AR app To enable Cloud Anchor in your project, you will need to create an API key. You can create it here: ARCore Cloud Anchor API After you create the API Key, copy it. Add meta data for the cloud API in Android Manifest application section Next, we can acquire a 3D asset from Google Poly. You can choose any asset, but we used the this asset. Download the obj format 3D model. Arctic Fox Go to app -> New SampleData directory. Drag and drop the contents of the 3D model folder into sampledata directory. Right click on the obj file in the sampledata directory and click Import Sceneform Asset. Rename the sfa and sfb files as you like. The sfa file is used by Android Studio to display the model, while the sfb file is included in the APK (app installed on phone). Click Finish after assigning the paths. After the gradle build finishes, the sfb file should open. Go to model, and change the scale to an appropriate size for the model. We changed the scale for the Arctic fox from 0.188 to 0.059. Save and build again. Step 2: User Interface and Custom AR Fragment We need to configure the AR session for Cloud Anchors. We can do this by editing the session and configuration of the ArFragment class. Create a new java class called CustomArFragment. This class will extend ArFragment. Paste the following code in it. import com.google.ar.core.Config; import com.google.ar.core.Session; import com.google.ar.sceneform.ux.ArFragment; public class CustomArFragment extends ArFragment{ @Override protected Config getSessionConfiguration(Session session) { getPlaneDiscoveryController().setInstructionView(null); Config config = super.getSessionConfiguration(session); config.setCloudAnchorMode(Config.CloudAnchorMode.ENABLED); return config; } } We set the instruction view null as a step to disable the initial hand gesture. This is optional. In the user interface, we will add our CustomArFragment and two buttons: Clear and Resolve. Clear will be used to clear the anchor in the scene, and resolve to resolve cloud Anchor. Following is the code for activity_main.xml. The fragment name will change based on your package name: The design should look like this: Step 3: Import Snackbar helper Import the file . This is a class by Google that will help us print logs at the bottom of the screen. If you open this file, you will notice errors. To fix them, we need to add a dependency for snackbar. Go to build.gradle(Module:App) and in dependencies add: Snackbar Helper implementation 'com.android.support:design:27.1.1' Initialize the Snackbar Helper as a global in MainActivity: private SnackbarHelper snackbarHelper = new SnackbarHelper(); Step 4: Create an Anchor and Clear it In this step, we’ll create a mechanism where only one Anchor can be created in the scene and we will overlay a 3D model on it. Pressing the clear button should clear this anchor. Paste the functions placeObject and addNodeToScene in your MainActivity. private void placeObject(ArFragment fragment, Anchor anchor, Uri model) { ModelRenderable.builder() .setSource(fragment.getContext(), model) .build() .thenAccept(renderable -> addNodeToScene(fragment, anchor, renderable)) .exceptionally((throwable -> { AlertDialog.Builder builder = new AlertDialog.Builder(this); builder.setMessage(throwable.getMessage()) .setTitle("Error!"); AlertDialog dialog = builder.create(); dialog.show(); return null; })); } private void addNodeToScene(ArFragment fragment, Anchor anchor, Renderable renderable) { AnchorNode anchorNode = new AnchorNode(anchor); TransformableNode node = new TransformableNode(fragment.getTransformationSystem()); node.setRenderable(renderable); node.setParent(anchorNode); fragment.getArSceneView().getScene().addChild(anchorNode); node.select(); } The placeObject function builds a renderable model and passes the fragment, anchor and renderable to addNodetoScene. The addNodeToScene function creates an AnchorNode on the Anchor, and further a TransformableNode with the parent as AnchorNode. An AnchorNode is fixed position and orientation, whereas a Transformable node can be interacted with through gestures. The Transformable node can be translated, rotated and scaled. Next, create two globals: private CustomArFragment fragment; private Anchor cloudAnchor; Paste the function setCloudAnchor in your main activity. private void setCloudAnchor (Anchor newAnchor){ if (cloudAnchor != null){ cloudAnchor.detach(); } cloudAnchor = newAnchor; } This function will ensure that there is only one cloudAnchor in the activity at any point of time. In the onCreate function, we initialize CustomArFragment and clearButton. We will also hide plane dicovery controller to disable the hand gesture (optional). When the clearButton is clicked, we set the cloudAnchor to null. When an upward facing plane is tapped, we create an Anchor and pass it to the setCloudAnchor function. After which, we call placeObject on the cloudAnchor, placing our 3D model at the tapped point. protected void onCreate(Bundle savedInstanceState) { super.onCreate(savedInstanceState); setContentView(R.layout.activity_main); fragment = (CustomArFragment) getSupportFragmentManager().findFragmentById(R.id.sceneform_fragment); fragment.getPlaneDiscoveryController().hide(); Button clearButton = findViewById(R.id.clear_button); clearButton.setOnClickListener(new View.OnClickListener() { @Override public void onClick(View view) { setCloudAnchor(null); } }); fragment.setOnTapArPlaneListener( (HitResult hitResult, Plane plane, MotionEvent motionEvent) -> { if (plane.getType() != Plane.Type.HORIZONTAL_UPWARD_FACING){ return; } Anchor newAnchor = hitResult.createAnchor(); setCloudAnchor(newAnchor); placeObject(fragment, cloudAnchor, Uri.parse("Fox.sfb")); } ); } Now, we can run the app and test it. After running the app, Sceneform will take care of requesting permissions and detecting planes. After a horizontally upward plane is detected, tap on it. Tapping on it, will create an object at that point. Pressing clear should clear the anchor and the object. Host Cloud Anchor Create a global enum AppAnchorState in MainActivity that will contain the status of the cloudAnchor. It’ll be NONE by default, HOSTING when hosting the Anchor and HOSTED when the anchor is done hosting. private enum AppAnchorState { NONE, HOSTING, HOSTED } private AppAnchorState appAnchorState = AppAnchorState.NONE; Update the fragment.setOnTapArPlaneListner to the following code: fragment.setOnTapArPlaneListener( (HitResult hitResult, Plane plane, MotionEvent motionEvent) -> { if (plane.getType() != Plane.Type.HORIZONTAL_UPWARD_FACING || appAnchorState != AppAnchorState.NONE){ return; } Anchor newAnchor = fragment.getArSceneView().getSession().hostCloudAnchor(hitResult.createAnchor()); setCloudAnchor(newAnchor); appAnchorState = AppAnchorState.HOSTING; snackbarHelper.showMessage(this, "Now hosting anchor..."); placeObject(fragment, cloudAnchor, Uri.parse("Fox.sfb")); } ); This will start hosting the anchor if the appAnchorState is none and will then change the state to HOSTING. The hostCloudAnchor method is used to host the anchor. We check if the anchor has finished hosting every time a frame is updated. To do this, we can create a function onUpdateFrame. This function will call another function checkUpdatedAnchor which will check the state of the anchor and update appAnchorState. private void onUpdateFrame(FrameTime frameTime){ checkUpdatedAnchor(); } private synchronized void checkUpdatedAnchor(){ if (appAnchorState != AppAnchorState.HOSTING){ return; } Anchor.CloudAnchorState cloudState = cloudAnchor.getCloudAnchorState(); if (appAnchorState == AppAnchorState.HOSTING) { if (cloudState.isError()) { snackbarHelper.showMessageWithDismiss(this, "Error hosting anchor.. " + cloudState); appAnchorState = AppAnchorState.NONE; } else if (cloudState == Anchor.CloudAnchorState.SUCCESS) { snackbarHelper.showMessageWithDismiss(this, "Anchor hosted with id " + cloudAnchor.getCloudAnchorId()); appAnchorState = AppAnchorState.HOSTED; } } } In the OnCreate function, add the following line after the fragment is initialized: fragment.getArSceneView().getScene().setOnUpdateListener(this::onUpdateFrame); Set the appAnchorState to NONE in the setCloudAnchor function and update the fuction to the code shown below: private void setCloudAnchor (Anchor newAnchor){ if (cloudAnchor != null){ cloudAnchor.detach(); } cloudAnchor = newAnchor; appAnchorState = AppAnchorState.NONE; snackbarHelper.hide(this); } You can run the app to test it. After the object is placed on the plane, it should start hosting. After it is Hosted, the snackbar will display the Cloud Anchor ID. Resolve Cloud Anchor To resolve the Cloud Anchor from the same app, another app, or another device we will need the Cloud ID. However the ID can be super long and hard to remember. Thus, we can store it as a short code integer. This short code will be the key to the Anchor ID. We will store the Short code and Anchor ID in Firebase. If you want to use Cloud Anchors on the same device, you can store Short Code and Anchor ID locally in the app. This use case is also explored in the Udemy course. We can start by initializing Firebase in our project from Android Studio. Go to Tools -> Firebase -> Real Time Database -> Save and Retrieve Data Next, click on step 1: Connect to Firebase. Sign into you Google account. If you don’t have one, it is free to create. Click Allow for permissions and go back to Android Studio and click Connect to Firebase. If there is an error, click Connect to Firebase in the assistant again. After that click on Add the Realtime Database to your app, click apply changes, and wait for the build to complete. Next, open a browser and go to . Select the project that you created in Android Studio. In the Develop tab, click on Database. Firebase Console Create a Realtime Database, choose start in test mode and click enable. Next, import the and classes into your project. These are classes provided by Google. will help us store and retrieve the Short Code and Anchor ID. will help us create a Dialog where the user can enter the Short Code to resolve the anchor and press an OK button. The OK button has a listener that we will use to trigger the resolve. StoreManager ResolveDialogFragment StoreManager ResolveDialogFragment Update the appAnchorState enum and add the states RESOLVING and RESOLVED to the enum. private enum AppAnchorState { NONE, HOSTING, HOSTED, RESOLVING, RESOLVED } Add a global for storeManager and initialize at the end of the OnCreate function. private StorageManager storageManager; protected void onCreate(Bundle savedInstanceState) { ... storageManager = new StorageManager(this); } Add a function onResolveOkPressed to MainActivity private void onResolveOkPressed(String dialogValue){ int shortCode = Integer.parseInt(dialogValue); storageManager.getCloudAnchorID(shortCode,(cloudAnchorId) -> { Anchor resolvedAnchor = fragment.getArSceneView().getSession().resolveCloudAnchor(cloudAnchorId); setCloudAnchor(resolvedAnchor); placeObject(fragment, cloudAnchor, Uri.parse("Fox.sfb")); snackbarHelper.showMessage(this, "Now Resolving Anchor..."); appAnchorState = AppAnchorState.RESOLVING; }); } The String dialogValue will be the Short code. This function takes the shortCode as the input, retrieves the resolved anchor, and places our 3D object on the Resolved Anchor. It changes the appAnchorState to RESOLVING. Initialize the Resolve button and add onResolveOkPressed as the listener for the OK button in the ResolveDialogFragment. protected void onCreate(Bundle savedInstanceState) { ... Button resolveButton = findViewById(R.id.resolve_button); resolveButton.setOnClickListener(new View.OnClickListener() { @Override public void onClick(View view) { if (cloudAnchor != null){ snackbarHelper.showMessageWithDismiss(getParent(), "Please clear Anchor"); return; } ResolveDialogFragment dialog = new ResolveDialogFragment(); dialog.setOkListener(MainActivity.this::onResolveOkPressed); dialog.show(getSupportFragmentManager(), "Resolve"); } }); ... } Lastly update the checkUpdated Anchor function to detect when the anchor has been successfully resolved. private synchronized void checkUpdatedAnchor(){ if (appAnchorState != AppAnchorState.HOSTING && appAnchorState != AppAnchorState.RESOLVING){ return; } Anchor.CloudAnchorState cloudState = cloudAnchor.getCloudAnchorState(); if (appAnchorState == AppAnchorState.HOSTING) { if (cloudState.isError()) { snackbarHelper.showMessageWithDismiss(this, "Error hosting anchor.. " + cloudState); appAnchorState = AppAnchorState.NONE; } else if (cloudState == Anchor.CloudAnchorState.SUCCESS) { storageManager.nextShortCode((shortCode) -> { if (shortCode == null){ snackbarHelper.showMessageWithDismiss(this, "Could not get shortCode"); return; } storageManager.storeUsingShortCode(shortCode, cloudAnchor.getCloudAnchorId()); snackbarHelper.showMessageWithDismiss(this, "Anchor hosted! Cloud Short Code: " + shortCode); }); appAnchorState = AppAnchorState.HOSTED; } } else if (appAnchorState == AppAnchorState.RESOLVING){ if (cloudState.isError()) { snackbarHelper.showMessageWithDismiss(this, "Error resolving anchor.. " + cloudState); appAnchorState = AppAnchorState.NONE; } else if (cloudState == Anchor.CloudAnchorState.SUCCESS){ snackbarHelper.showMessageWithDismiss(this, "Anchor resolved successfully"); appAnchorState = AppAnchorState.RESOLVED; } } } After this you can run the app to test it. Host the anchor by tapping on a plane, and remember the short code. Press Clear. Resolve the Anchor by clicking the Resolve button, entering the Short Code and then press OK. To test the cloud storage, if you have two or more android phones, load the app on both phones. Host the anchor using one, and resolve the anchor using another phone. If you have a single phone, you can test cloud anchors by hosting the anchor, deleting and reinstalling the app, and then resolving the anchor. Summing Up In this project, following were the main takeaways: Using Sceneform to import and overlay 3D objects in AR Creating Custom AR Fragment for Cloud Anchors Hosting and Resolving an Anchor on the Cloud Parting Shot We have released a course on ARCore and Sceneform for Android on Udemy. Here is a coupon: https://www.udemy.com/arcore-and-sceneform-for-android-ar/?couponCode=ARCORE-10 The course covers projects such as building a furniture (IKEA type) AR app using Sceneform, Augmented Images i.e. detecting images in the real world and overlaying AR graphics on top of them, and Cloud Anchors. We will be regularly updating this course with more content based on student requests and ARCore Features. Work in progress modules include Working with Points and Point Cloud, and building interactive AR experiences. Hope you enroll in the course and join us on the AR Learning Adventure! :)