When Sceneform was discontinued and archived back in 2020, it made creating ARCore apps on Android more difficult. Inspired by the original Sceneform APIs and based on the ARCore samples, I have created AugmentedFaceFragment and AugmentedFaceListener interface to be able to easily create ARCore AugmentedFaces features on Android. I will be releasing 3 articles building different AugmentedFaces features using AugmentedFaceFragment and AugmentedFaceListener interface. What you will build Part 1 of the series will cover the basics of the AugmentedFaceFragment and AugmentedFaceListener  as well as the overview of all the helper classes. At the end of this article you will build a simple demo by writing a few lines of code. Project setup Download the Starter Project Clone the repository: git clone https://github.com/droid-girl/arfaces_labs.git What’s our starting point? Our starting point is a modified version of ARCore SDK sample for Augmented Faces. The code has been modified in a way that we can now easily add different textures and objects to a face object. Project structure Helpers - original Java files from sample code
Rendering - original Java files from sample code that handles rendering of AR objects and background Additional files for the repository: AugmentedFaceFragment.kt - handles rendering and tracking of Augmented Faces
AugmentedFaceListener.kt - handles Add and Update Augmented Faces callbacks
AugmentedFaceNode.kt - handles the rendering of a face including a texture and associated 3D models
AugmentedFaceRenderer.kt - renders face texture
FaceRegion.kt - renders 3D model located on a defined FaceLandmark MainActivity setup <LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:tools="http://schemas.android.com/tools"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    tools:context=".MainActivity">

    <androidx.fragment.app.FragmentContainerView android:name="com.ar.arfaces.arface.AugmentedFaceFragment"
        android:id="@+id/face_view"
        android:layout_width="match_parent"
        android:layout_height="match_parent"
        android:layout_gravity="top" />

</LinearLayout> class MainActivity : AppCompatActivity(), AugmentedFaceListener {

    private lateinit var binding: ActivityMainBinding

    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        binding = ActivityMainBinding.inflate(layoutInflater)
        binding.faceView.getFragment<AugmentedFaceFragment>().setAugmentedFaceListener(this)
        setContentView(binding.root)
    }

    override fun onFaceAdded(face: AugmentedFaceNode) {}

    override fun onFaceUpdate(face: AugmentedFaceNode) {}

} In activity_main.xml we add AugmentedFaceFragment as a main view and to be able to receive events from this fragment, we need to define and set AugmentedFaceListener to our fragment. onFaceAdded method will be called when ARCore detects a new face and onFaceUpdate will be called on each frame update. Let’s add a face texture as our next step. Add face texture We will use ARCore sample assets for our first face mask. You can find assets/models folder in your project. Let’s add freckles.png as face texture: In MainActivity.kt modify onFaceAdded method as follows: override fun onFaceAdded(face: AugmentedFaceNode) {
   face.setFaceMeshTexture("models/freckles.png")
} Add 3D models Introduction to AugmentedFaceNode AugmentedFace class uses the face mesh and center pose to identify face regions. These regions are: Left forehead (LEFT_FOREHEAD)
Right forehead (RIGHT_FOREHEAD)
Tip of the node (NOSE_TIP) This project includes the AugmentedFaceNode class. Inspired by Sceneform, it is a node that will render visual effects on a detected face using ARCore. AugmentedFaceNode defines same face regions as AugmentedFace in a companion object: companion object {
   enum class FaceLandmark {
       FOREHEAD_RIGHT,
       FOREHEAD_LEFT,
       NOSE_TIP
   }
} Later in this tutorial, we will extend FaceLandmark enum class and add our own face regions. AugmentedFaceNode includes faceLandmarks HashMapthat connects FaceLandmark with a 3D model. The source code for this project can be found here. When Sceneform was discontinued and archived back in 2020, it made creating ARCore apps on Android more difficult. Inspired by the original Sceneform APIs and based on the ARCore samples, I have created AugmentedFaceFragment and AugmentedFaceListener interface to be able to easily create ARCore AugmentedFaces features on Android. Sceneform AugmentedFaceFragment AugmentedFaceListener I will be releasing 3 articles building different AugmentedFaces features using AugmentedFaceFragment and AugmentedFaceListener interface. AugmentedFaceFragment AugmentedFaceListener What you will build Part 1 of the series will cover the basics of the AugmentedFaceFragment and AugmentedFaceListener as well as the overview of all the helper classes. At the end of this article you will build a simple demo by writing a few lines of code. AugmentedFaceFragment AugmentedFaceListener Project setup Download the Starter Project Clone the repository: git clone https://github.com/droid-girl/arfaces_labs.git git clone https://github.com/droid-girl/arfaces_labs.git What’s our starting point? Our starting point is a modified version of ARCore SDK sample for Augmented Faces. The code has been modified in a way that we can now easily add different textures and objects to a face object. ARCore SDK sample Project structure Helpers - original Java files from sample code Rendering - original Java files from sample code that handles rendering of AR objects and background Helpers - original Java files from sample code Helpers Rendering - original Java files from sample code that handles rendering of AR objects and background Rendering Additional files for the repository: AugmentedFaceFragment.kt - handles rendering and tracking of Augmented Faces AugmentedFaceListener.kt - handles Add and Update Augmented Faces callbacks AugmentedFaceNode.kt - handles the rendering of a face including a texture and associated 3D models AugmentedFaceRenderer.kt - renders face texture FaceRegion.kt - renders 3D model located on a defined FaceLandmark AugmentedFaceFragment.kt - handles rendering and tracking of Augmented Faces AugmentedFaceFragment.kt AugmentedFaceListener.kt - handles Add and Update Augmented Faces callbacks AugmentedFaceListener.kt AugmentedFaceNode.kt - handles the rendering of a face including a texture and associated 3D models AugmentedFaceNode.kt AugmentedFaceRenderer.kt - renders face texture AugmentedFaceRenderer.kt FaceRegion.kt - renders 3D model located on a defined FaceLandmark FaceRegion.kt MainActivity setup <LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:tools="http://schemas.android.com/tools"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    tools:context=".MainActivity">

    <androidx.fragment.app.FragmentContainerView android:name="com.ar.arfaces.arface.AugmentedFaceFragment"
        android:id="@+id/face_view"
        android:layout_width="match_parent"
        android:layout_height="match_parent"
        android:layout_gravity="top" />

</LinearLayout> <LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:tools="http://schemas.android.com/tools"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    tools:context=".MainActivity">

    <androidx.fragment.app.FragmentContainerView android:name="com.ar.arfaces.arface.AugmentedFaceFragment"
        android:id="@+id/face_view"
        android:layout_width="match_parent"
        android:layout_height="match_parent"
        android:layout_gravity="top" />

</LinearLayout> class MainActivity : AppCompatActivity(), AugmentedFaceListener {

    private lateinit var binding: ActivityMainBinding

    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        binding = ActivityMainBinding.inflate(layoutInflater)
        binding.faceView.getFragment<AugmentedFaceFragment>().setAugmentedFaceListener(this)
        setContentView(binding.root)
    }

    override fun onFaceAdded(face: AugmentedFaceNode) {}

    override fun onFaceUpdate(face: AugmentedFaceNode) {}

} class MainActivity : AppCompatActivity(), AugmentedFaceListener {

    private lateinit var binding: ActivityMainBinding

    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        binding = ActivityMainBinding.inflate(layoutInflater)
        binding.faceView.getFragment<AugmentedFaceFragment>().setAugmentedFaceListener(this)
        setContentView(binding.root)
    }

    override fun onFaceAdded(face: AugmentedFaceNode) {}

    override fun onFaceUpdate(face: AugmentedFaceNode) {}

} In activity_main.xml we add AugmentedFaceFragment as a main view and to be able to receive events from this fragment, we need to define and set AugmentedFaceListener to our fragment. activity_main.xml AugmentedFaceFragment AugmentedFaceListener onFaceAdded method will be called when ARCore detects a new face and onFaceUpdate will be called on each frame update. onFaceAdded onFaceUpdate Let’s add a face texture as our next step. Add face texture We will use ARCore sample assets for our first face mask. You can find assets/models folder in your project. Let’s add freckles.png as face texture: assets/models freckles.png In MainActivity.kt modify onFaceAdded method as follows: MainActivity.kt onFaceAdded override fun onFaceAdded(face: AugmentedFaceNode) {
   face.setFaceMeshTexture("models/freckles.png")
} override fun onFaceAdded(face: AugmentedFaceNode) {
   face.setFaceMeshTexture("models/freckles.png")
} Add 3D models Introduction to AugmentedFaceNode AugmentedFace class uses the face mesh and center pose to identify face regions. These regions are: Left forehead (LEFT_FOREHEAD) Right forehead (RIGHT_FOREHEAD) Tip of the node (NOSE_TIP) Left forehead (LEFT_FOREHEAD) Right forehead (RIGHT_FOREHEAD) Tip of the node (NOSE_TIP) This project includes the AugmentedFaceNode class. Inspired by Sceneform, it is a node that will render visual effects on a detected face using ARCore. AugmentedFaceNode defines same face regions as AugmentedFace in a companion object: AugmentedFaceNode AugmentedFaceNode AugmentedFace companion object {
   enum class FaceLandmark {
       FOREHEAD_RIGHT,
       FOREHEAD_LEFT,
       NOSE_TIP
   }
} companion object {
   enum class FaceLandmark {
       FOREHEAD_RIGHT,
       FOREHEAD_LEFT,
       NOSE_TIP
   }
} Later in this tutorial, we will extend FaceLandmark enum class and add our own face regions. FaceLandmark enum class AugmentedFaceNode includes faceLandmarks HashMap that connects FaceLandmark with a 3D model. AugmentedFaceNode HashMap The source code for this project can be found here . here

The code in this story is for educational purposes. The readers are solely responsible for whatever they build with it.

After Google Gave Up on Sceneform, This Developer Built a Better Way to Bring AR Faces to Life

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

Java bits: 0xFF and 0xFFL

10 Things You Didn't Know You Could Do With Google Keep

10 Things to check on Android code reviews

10 Mistakes In Mobile App Development That Make You Look Dumb

10 good rules for bad App

Java bits: 0xFF and 0xFFL

10 Things You Didn't Know You Could Do With Google Keep

10 Things to check on Android code reviews

10 Mistakes In Mobile App Development That Make You Look Dumb

10 good rules for bad App

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps