Gamers Paid for Deep Learning and Self-Driving Cars Research

The gaming industry has a significant impact on deep learning and self-driving cars. GPUs fueled the progress in deep learning, as they enabled training much larger neural networks on large datasets.

For about 20 years, GPUs were mostly used by gamers. Every year NVIDIA was pouring billions of dollars into R&D to make better and faster GPUs to support better gaming experience. In the early 2000s, they started considering scientific computing and machine learning acceleration, or, more generally, general purpose GPU computation (GPGPU). They released the first version of CUDA, a parallel computing platform, in 2007. CUDA made it much easier to program for GPUs, which led to more experimentation by researchers.

In 2012, SuperVision group (Alex Krizhevsky, Geoffrey Hinton, and Ilya Sutskever) used CUDA to develop the AlexNet model that won the ImageNet Large Scale Visual Recognition Challenge. Perhaps, it was the event that brought the attention of machine learning community to the power of deep neural networks and GPU computation. Now NVIDIA dominates the market of hardware for training deep neural networks and is moving the space of hardware for inference. That’s interesting to think that gamers of 90’s and 2000’s paid for R&D of the deep learning hardware.

Simulation is another example of leveraging software developed by the gaming industry to fuel artificial intelligence progress. For example, when you develop self-driving car software, you need to test it on a large variety of situations. Most of the real driving is uneventful. Interesting events, like a road construction, a car going against traffic, or a mountain lion running into the road, are rare. If you collect data and test your software only in the real world, it will take very long time and cost a lot of money, because self-driving cars and time of test engineers are expensive. If some interesting case appears only once every 1000 hours of driving, you will have to drive a lot to collect enough data about it.

One way to make it more efficient is to simulate interesting situations in the real world. Companies like Waymo, Uber, Tesla set up test sites with roads, road signs, traffic lights, and have humans driving around to create traffic that looks real. Then they can set up situations like the car stopping suddenly ahead of you, pedestrians running into the road, etc. That also takes time and resources, but at least you can test exactly what you want to test and can repeat the test case as many times as you want.

What if you can make a software simulation of the world? Then you can control everything and run tests faster, as fast as your computation hardware allows it. You can scale computation by adding more GPUs or more servers. You can test any exotic situations with great precision. In the time that it takes a real car fleet to drive a thousand miles, your simulation can drive millions of virtual miles. Software Development Times published a piece on using game engines for self-driving car simulation.

There are challenges here, of course. You need a good physics engine. Maps should look real. Ideally, you would need to recreate good copies of real cities with same traffic signs, lane markings, buildings. You need to simulate signals for all kinds of sensors: cameras, lidars, radars, etc. A lot of interesting challenges, but in the end, you get much faster experimentation. Also, it gives you an ability to do regression testing of your software at scale. When you release a new version of the software, you can test it on millions of interesting test cases, including rare edge cases.

The good thing is that you probably don’t have to build everything from scratch. Gaming industry already built a lot of low-level functionality into game engines. For example, Udacity built a car simulator on top of Unity engine for its self-driving car nanodegree program (see details in an article by Naoki Shibuya). Others are using Unreal Engine 4 (e.g., UE4Sim simulator). Even Grand Theft Auto can be used to test simple algorithms. DeepDrive project used GTA V for simulation, but, unfortunately, it was shut down for legal reasons.

I couldn’t find information on the technology behind car simulation that self-driving car companies like Waymo, Drive.ai, Voyage, Tesla, Uber and others use. Though, there is some coverage on how they use simulation. The Atlantic published an article on Waymo’s testing operations, including a secret base “Castle” in the Central Valley, and a simulation software “Carcraft.”

“At any time, there are now 25,000 virtual self-driving cars making their way through fully modeled versions of Austin, Mountain View, and Phoenix, as well as test-track scenarios. Waymo might simulate driving down a particularly tricky road hundreds of thousands of times in a single day. Collectively, they now drive 8 million miles per day in the virtual world. In 2016, they logged 2.5 billion virtual miles versus a little over 3 million miles by Google’s IRL self-driving cars that run on public roads. And crucially, the virtual miles focus on what Waymo people invariably call “interesting” miles in which they might learn something new. These are not boring highway commuter miles.”

Waymo Carcraft simulation software. Source

There is even a company that focuses just on simulation. Right Hook develops simulation software for self-driving cars, aviation, and robotics. Listen to the episode 22 of Autonocast podcast to learn more about what they are doing and why they think simulation is important for the future of self-driving cars.

It’s a great synergy and reuse of technology between gaming and artificial intelligence industries. Perhaps, in the next 10–20 years, deep learning and automotive industries will pay for R&D that will benefit gamers of the future. Interesting times.