Using Sofi and WebSockets to command game engines A while back I wrote a post on as frontend interfaces. I want to enable rich interfaces in the Python ecosystem that are usable in virtual and augmented reality. Since then, I was able to build some basic concepts on top of and I’m here to share them. using with game engines Sofi The backend Sofi works as a WebSocket server that can issue commands and handle events from a client. It’s written to simplify the use of frontend web technologies as interfaces to a Python backend. You can even with a desktop look and feel. package it as a desktop app It functions by sending the user to a webpage with the basics needed to open a WebSocket client. This client then registers handlers that process server commands telling it how to alter the DOM or respond to events. All the logic resides with Python, the webpage is the user interface. I wrote it with enough modularity such that the command and event structure is reusable for any technology. So it wasn’t much of a stretch to think that I could replace the webpage portion with a game engine. It was only a matter of finding one with easy access to WebSocket clients. The game engine I looked at Unity3D, Unreal and CryEngine. Of the three, the first two have the most expansive communities, and of those, Unity seems to have the larger ecosystem. While usually a matter of taste, I’m not here to argue which one has better graphics. At this point, I want simplicity for a proof of concept, and this came in the form of . WebSocketSharp WebSocketSharp is a C# library available in Unity3D’s asset store. It provides client and server classes usable by any script in your project’s scene. I ran a few tests and it worked well enough for my needs. Mapping back to Sofi, this part performs the function of the JavaScript WebSocket client library that runs on your browser. Now, let’s wrap it so it understands how to talk to the Sofi WebSocket server, and make the equivalent of . sofi.js Making the frontend If you haven’t used Unity3D before, or any game engine for that matter, getting to “hello world” can be confusing. While not required to understand this implementation, I’ll go over some of the main concepts here. Please note that Unity has their own extensive available for more details. documentation First thing to do when making a “game” — which is what I’m doing for all intensive purposes — is to create a project. Think of this as a package that contains all the assets you’ll need for rendering a world and its interactions. Almost everything in a project is an asset: textures, shaders, scripts, objects, etc. Within a project there are scenes. Think of these like game levels, or scenes in a movie. It’s a collection of objects, scripts and camera views that render into an experience with a common objective. Once there’s a scene, I can add the WebSocketSharp library as an asset. This makes it possible to import its class into a script in which to do some magic. You can assign scripts to objects in the scene and trigger execution through the engine’s event loop. WebSocket Every scene has a default camera. Cameras, among other things, determine how to view the scene. This includes viewing angles, distance, render settings, lighting, etc. But since they’re objects themselves, they can also hold scripts. Which means I can stick a script on the camera that loads immediately with the scene. WebSocketClient Unity scripts can schedule handlers in the main event loop. There are many game events for physical collisions or user inputs detailed in the docs. But I’m interested in those that run at initialization and on a regular basis. In our case, I’m looking for and . The former runs when rendering every frame, while the latter executes after loading. I also register a handler for to clean things up when exiting. Update Start OnDestroy To configure the client, instantiate a during the engine’s Start event. This requires the address and port of the server, as well as its own set of event handlers. I use when a connection completes, when receiving new data, and for disconnections. WebSocket OnOpen OnMessage OnClose It’s important to think about the event loop when working with . Message processing cannot happen here because it can block the scene from loading. So I made a command queue in the form of a object that lives within the client class. The idea is to review that queue on every event — once every frame — and handle the commands then. Start List Update Besides message processing, the event is a good point at which to check for a connection. It’s easy to trigger a reconnect, allowing the scene to wait for a server to spin up if needed. Update The engine gathers user inputs and queues them up, much like the earlier socket messages. So I add checks for those here as well, using the method to communicate them back to the server. sendAsync() Communicating with Sofi To talk a little bit about protocol, Sofi’s communication mechanism is very extensible. The socket server listens on a particular port expecting JSON data. Event messages have an key set with the name which the server uses to check for callbacks. If one is present, execution passes to that function, which can in turn use the method to send a command. UI commands have a key to identify them and their other parameters. Except for the basics, I don’t hardcode any of this, so it’s very easy to add custom events and commands by making up new names without modifying Sofi itself. 'event' dispatch() 'command' The can handle commands for spawning new objects, updating object properties, removing objects and subscribing to events. The types of objects are the very basic shapes: cubes, spheres, cylinders, capsules, planes and text. There are quite a few settings for each, including color, size, position and rotation. WebSocketClient Packaging The other neat part is that Unity3D can produce binaries for all major platforms. This includes Windows, Linux, MacOS, Android, iOS, and otherx. So not only does this add 3D capabilities to Python, but it also includes multi-platform support. The build process itself is simple and directed by the game engine itself. Couple it with PyInstaller like in , and now you have only one executable for distribution, Python environment and all. my previous article What does it look like? For a demo of what it can do, have a look at my PyCaribbean presentation below. I run it at about 20:09. I’ve posted the Python code on GitHub under the repo. The implementation is located in the folder. It also includes the Unity3D project along with it. The full code will run through my slides for PyCaribbean and starts a listener for any tweet. sofi-unity3d WebSocketClient engine/Assets #python I also went off and built the binaries for , and , so you can download and play with it yourselves. When you run the engine, it will load an empty sky and grassy plane scene that sits waiting for a WebSocket server. This could be the Sofi instance in or something of your own making that listens on port 9000. Linux Windows MacOS server.py What’s next? Today we’ve explored the possibilities around creating game engine based “widget libraries”. Demonstrating that it’s very possible for a Python backend to provide the logic, while leaving the UI heavy lifting to the engine. It’s also portable cross-platform and distributable with little effort. The next step on the game engine side is to provide practical widgets and assets. With this framework we can spawn any pre-existing game objects, meshes and scripts. It’s up to us to define what makes for good reusable components and interactions, and package them with the built binaries. On the Python side, I’d like to provide an HTTP-based asset server. This allows folks to create custom assets in something like , which is also Python-enabled, and load them into any scene. No need to touch the game engine. Blender I also want to build a world with that enables VR or AR on hardware like the Rift, Vibe, Hololens, etc. I have a DK2 at home, but haven’t looked into the making any projects with it yet. For additional reading on how I made Sofi, have a look at the series. A Python Ate My GUI If you liked this article and want to keep up with what I’m working on, please recommend it, visit for more topics and . tryexecptpass.org follow me on Twitter
