How to Connect Flutter with Unity

Recently, we did a development for our partner BrainCo which was a real challenge. The company, which was born out of the Harvard Innovation Labs, develops cognitive training technology products and applies their expertise in machine learning, design, and neuroscience to create innovative cognitive-based applications.

One of their most successful products is FocusCalm, a brain-sensing headband and an app that helps users learn how to control their mindset in just a few minutes per day.

Our goal was to help BrainCo take the games available in FocusCalm to the next level. These games would be played both by using the neurofeedback captured by the headband and user input.  

But the big question was... Which framework should we build the games with?

There were plenty of options, so we decided to define first the desired requirements of that framework. It should: allow embedding games inside a Flutter app, ideally be cross platform, perform smoothly on widely used devices and last but not least, allow building games with complex physics easily. This led us to choose Unity, which is good for mobile projects, dev friendly, and suitable for small projects, and even though it has less visual detail than other alternatives like Unreal Engine, it was enough for our needs.

Now the big challenge was: How to integrate Unity and Flutter?

A key piece in the puzzle was the Flutter_unity_widget built by juicycleff which allows embedding Unity games inside Flutter widgets and also provides a communication mechanism between Flutter and Unity.  

If you want to try out how the Unity and Flutter integration works firsthand you can find a very simple yet helpful sample project at the widget's github page.

Yet, it wasn’t as easy as it sounds.

While working on the integration we faced some small bugs that luckily we were able to fix. To name a few, to build the app we needed to export the Unity module for iOS and for Android. The problem was that after building for Android, building for iOS would not work. We fixed this by removing the Flutter Package from the Unity project after building for Android and adding it again. Another bug we faced was a Unity - Flutter messaging issue. Upon loading of the Unity module on iOS the widget sent a message to the iOS code, which made the app crash. We were able to fix it by commenting on the method which was sending the message in the unity plugin. Of course, this is not ideal because you could eventually need that method, but that wasn’t our case.

We are pretty sure that some of these bugs were quickly fixed by the developer afterwards.

We are super proud about this project, and here I tell you some of the reasons why:

• We were able to have the app logic running smoothly while the mini game was running its own logic, and both components –Flutter and Unity– communicating with each other without any hassles.
• We are now able to build games with complex physics and graphics without sacrificing performance.
• A combo of Wearable + Flutter + Unity was integrated successfully into one single app.
• All major aspects of the app are built with cross-platform frameworks, reducing development efforts.

If you want to learn more, you can watch the whole Sendbird's CodeCafe episode where we showcased the development process here.