How I hacked Google Daydream controller (Part VI)

Building a better typing system for VR/AR using AI.

Previous chapters of “How I hacked Google Daydream controller”:(Part I) Reversing and making it compatible with Apple iOS devices.(Part II) Developing compatibility across multiple platforms.(Part III) What about using two controllers at same time?(Part IV) Doing things even more serious.(Part V) Ghetto positional tracking.

Wait, what is this?! More details below.

FOREWORD

“VR will be big but AR will be bigger and take longer.” Right now the market has actually launched and major tech players’ strategies well emerged. Anyway, mobile VR will still produce the dominant “Explorer” installed base for VR, with Google Daydream View moving things on from Cardboard days. At the same time, digital experiences started to be increasingly immersive, holding people in virtual worlds for much longer. And in all those areas where real life overlaps with the virtual one, people begun to have basics needs like: to write.

At date, meanwhile in other fields VR technologies are growing fast taking solid place in the scene, Inputs Systems seem to be stuck to the Stone Age.

But… Can something old be futurist?

THE BEGINNING

There’s a mystical word called nostalgia that bridges the gap between the rational and the creative side of my brain. And when I feel nostalgic, some kind of magic happens. Being a 90’s boy passionate about technology means to have had at least one handheld like this:

R.I.P. Palm Inc.

First models were nothing so far from a digital agenda. Weaponized with a nib for resistive screens they really played the old school. As owner, by browsing the manuals I discovered a peculiar feature called Graffiti.

Graffiti Alphabet stickers (on the left) | A Palm Pilot User Manual page (on the right).

Basically, drawing with the nib on the area placed at the bottom of the user interface (the so called “Graffiti text area”) it was possible to write characters and strokes.

BACK TO THE FUTURE

As well as games, also apps and a relevant part of the web is moving to VR. But… “How people really feel typing in a virtual environment?!”. Undoubtedly digital keyboards are cute and prediction algorithms help a lot although pointing-and-clicking on each letter to write a word can be frustrating. The other disadvantage is that they occupy space in the visible.

Maybe a secret conspiracy wants us all drummers…

Google’s Daydream Keyboard promises “easy typing” in Virtual Reality.

Often an idea starts with the right question but many times the answers are not totally in the present. Digging into memories of my past, I found Graffiti meanwhile the AI’s mosquito was buzzing in my ear.

The next step was easy to predict: give touchpad a brain to type smarter in virtual worlds.

A NEW BIRTH

As the alchemist who needs to prepare his potions, everything started by dusting the old grimoire. I put some “npm install” and some JavaScript juice into the cauldron then I mixed them with some math algorithms and dragon’s nails, obviously. Unbundling some functionalities from my previous hacks I poisoned the main driver script (that allowed me to communicate with Google Daydream controller via Bluetooth LE) to read raw touchpad data directly on my laptop.

Node.js Daydream Driver running on macOS.

Whereupon, I quickly set up a tool to store and classify all the touchpad interactions triggered by thumb gestures. Few more filtering to improve the quality of the recorded training dataset and I get start a supervised learning process. Some more socket.io bindings behind the scenes to make it alive and then, a touch of bootstrap styling to beautify it.

The final result was surprising and as a tradition, I decided to baptise this tool calling it Thumbfiti.

Thumbfiti v0.1 alpha.

Thumbfiti v0.1 alpha video demo.

(to avoid to become part of the debate between Palm and Xerox about the patent infringement afflicting the Graffiti’s strokes-map, I applied few edits to build up the first alphabet).

SOMETHING CONCRETE

Most of the times, to see a prototype take shape implies several iterations, tribulations and definitively, it can take a long period, but… Fortunately, I really like to put an “alpha” after the names of my projects, aha! (joking). Feeling inspired combined with ambition and a considerable dose of enthusiasm pushed Thumbfiti development forward.

Take a look here to see it in action and taste the potential.

That’s all folks!Thank you.

Previous chapters of “How I hacked Google Daydream controller”:(Part I) Reversing and making it compatible with Apple iOS devices.(Part II) Developing compatibility across multiple platforms.(Part III) What about using two controllers at same time?(Part IV) Doing things even more serious.(Part V) Ghetto positional tracking.