Building a better typing system for VR/AR using AI. Previous chapters of “ ”: How I hacked Google Daydream controller (Part I) Reversing and making it compatible with Apple iOS devices. (Part II) Developing compatibility across multiple platforms. (Part III) What about using two controllers at same time? Doing things even more serious. (Part IV) Ghetto positional tracking. (Part V) Wait, what is this?! More details below. FOREWORD “VR will be big but AR will be bigger and take longer.” Right now the market has actually launched and major tech players’ strategies well emerged. Anyway, mobile VR will still produce the dominant “Explorer” installed base for VR, with moving things on from days. At the same time, digital experiences started to be increasingly immersive, holding people in virtual worlds for much longer. And in all those areas where real life overlaps with the virtual one, people begun to have basics needs like: . Google Daydream View Cardboard to write At date, meanwhile in other fields VR technologies are growing fast taking solid place in the scene, seem to be stuck to the . Inputs Systems Stone Age But… Can something old be futurist? THE BEGINNING There’s a mystical word called that bridges the gap between the rational and the creative side of my brain. And when I feel nostalgic, some kind of magic happens. Being a 90’s boy passionate about technology means to have had at least one handheld like this: nostalgia R.I.P. . Palm Inc First models were nothing so far from a digital agenda. Weaponized with a nib for resistive screens . As owner, by browsing the manuals I discovered a peculiar feature called . they really played the old school Graffiti Graffiti Alphabet stickers (on the left) | A Palm Pilot User Manual page (on the right). Basically, drawing with the nib on the area placed at the bottom of the user interface (the so called “ text area”) it was possible to write characters and strokes. Graffiti BACK TO THE FUTURE As well as games, also apps and a relevant part of the web is moving to VR. But… “How people really feel typing in a virtual environment?!”. Undoubtedly digital keyboards are cute and prediction algorithms help a lot although on each letter to write a word can be frustrating. The other disadvantage is that they occupy space in the visible. pointing-and-clicking Maybe a secret conspiracy wants us all drummers… promises “easy typing” in Virtual Reality. Google’s Daydream Keyboard Often an idea starts with the right question but many times the answers are not totally in the present. Digging into memories of my past, I found meanwhile the ’s mosquito was buzzing in my ear. Graffiti AI The next step was easy to predict: . give touchpad a brain to type smarter in virtual worlds A NEW BIRTH As the alchemist who needs to prepare his potions, everything started by dusting the old . I put some “ ” and some juice into the cauldron then I mixed them with some and dragon’s nails, obviously. Unbundling some functionalities from my I poisoned the main driver script (that allowed me to communicate with controller via ) to read directly on my laptop. grimoire npm install JavaScript math algorithms previous hacks Google Daydream Bluetooth LE raw touchpad data Daydream Driver running on macOS. Node.js Whereupon, I quickly set up a tool to store and classify all the triggered by Few more filtering to improve the quality of the recorded and I get start a process. Some more bindings behind the scenes to make it alive and then, a touch of styling to beautify it. touchpad interactions thumb gestures. training dataset supervised learning socket.io bootstrap The final result was surprising and as a tradition, I decided to baptise this tool calling it . Thumbfiti Thumbfiti v0.1 alpha. Thumbfiti v0.1 alpha video demo. (to avoid to become part of the debate between and about the patent infringement afflicting the ’s strokes-map, I applied few edits to build up the first alphabet). Palm Xerox Graffiti SOMETHING CONCRETE Most of the times, to see a prototype take shape implies several iterations, tribulations and definitively, it can take a long period, but… Fortunately, I really like to put an “alpha” after the names of my projects, aha! (joking). Feeling inspired combined with ambition and a considerable dose of enthusiasm pushed development forward. Thumbfiti Take a look to see it in action and taste the potential. here That’s all folks!Thank you. Previous chapters of “ ”: How I hacked Google Daydream controller (Part I) Reversing and making it compatible with Apple iOS devices. (Part II) Developing compatibility across multiple platforms. (Part III) What about using two controllers at same time? Doing things even more serious. (Part IV) Ghetto positional tracking. (Part V)
