Table of Links ABSTRACT ABSTRACT ABSTRACT 1 INTRODUCTION 1 INTRODUCTION 1 INTRODUCTION 2 INTERACTIVE WORLD SIMULATION 2 INTERACTIVE WORLD SIMULATION 2 INTERACTIVE WORLD SIMULATION 3 GAMENGEN 3 GAMENGEN 3 GAMENGEN 3.1 DATA COLLECTION VIA AGENT PLAY 3.2 TRAINING THE GENERATIVE DIFFUSION MODEL 4 EXPERIMENTAL SETUP 4 EXPERIMENTAL SETUP 4 EXPERIMENTAL SETUP 4.1 AGENT TRAINING 4.2 GENERATIVE MODEL TRAINING 5 RESULTS 5 RESULTS 5 RESULTS 5.1 SIMULATION QUALITY 5.2 ABLATIONS 6 RELATED WORK 6 RELATED WORK 6 RELATED WORK 7 DISCUSSION, ACKNOWLEDGEMENTS AND REFERENCES 7 DISCUSSION, ACKNOWLEDGEMENTS AND REFERENCES 7 DISCUSSION, ACKNOWLEDGEMENTS AND REFERENCES 6 RELATED WORK 6 RELATED WORK Interactive 3D Simulation Simulating visual and physical processes of 2D and 3D environments and allowing interactive exploration of them is an extensively developed field in computer graphics (Akenine-Mller et al., 2018). Game Engines, such as Unreal and Unity, are software that processes representations of scene geometry and renders a stream of images in response to user interactions. The game engine is responsible for keeping track of all world state, e.g. the player position and movement, objects, character animation and lighting. It also tracks the game logic, e.g. points gained by accomplishing game objectives. Film and television productions use variants of raytracing (Shirley & Morley, 2008), which are too slow and compute-intensive for real time applications. In contrast, game engines must keep a very high frame rate (typically 30-60 FPS), and therefore rely on highly-optimized polygon rasterization, often accelerated by GPUs. Physical effects such as shadows, particles and lighting are often implemented using efficient heuristics rather than physically accurate simulation. Neural 3D Simulation  Neural methods for reconstructing 3D representations have made significant advances over the last years. NeRFs (Mildenhall et al., 2020) parameterize radiance fields using a deep neural network that is specifically optimized for a given scene from a set of images taken from various camera poses. Once trained, novel point of views of the scene can be sampled using volume rendering methods. Gaussian Splatting (Kerbl et al., 2023) approaches build on NeRFs but represent scenes using 3D Gaussians and adapted rasterization methods, unlocking faster training and rendering times. While demonstrating impressive reconstruction results and real-time interactivity, these methods are often limited to static scenes. Neural 3D Simulation Video Diffusion Models Diffusion models achieved state-of-the-art results in text-to-image generation (Saharia et al., 2022; Rombach et al., 2022; Ramesh et al., 2022; Podell et al., 2023), a line of work that has also been applied for text-to-video generation tasks (Ho et al., 2022; Blattmann et al., 2023b;a; Gupta et al., 2023; Girdhar et al., 2023; Bar-Tal et al., 2024). Despite impressive advancement in realism, text adherence and temporal consistency, video diffusion models remain too slow for real-time applications. Our work extends this line of work and adapts it for real-time generation conditioned auto regressively on a history of past observations and actions. Video Diffusion Models Game Simulation and World Models Several works attempted to train models for game simulation with actions inputs. Yang et al. (2023) build a diverse dataset of real-world and simulated videos and train a diffusion model to predict a continuation video given a previous video segment and a textual description of an action. Menapace et al. (2021) and Bruce et al. (2024) focus on unsupervised learning of actions from videos. Menapace et al. (2024) converts textual prompts to game states, which are later converted to a 3D representation using NeRF. Unlike these works, we focus on interactive playable real-time simulation, and demonstrate robustness over long-horizon trajectories. We leverage an RL agent to explore the game environment and create rollouts of observations and interactions for training our interactive game model. Game Simulation and World Models Another line of work explored learning a predictive model of the environment and using it for training an RL agent. Ha & Schmidhuber (2018) train a Variational Auto-Encoder (Kingma & Welling, 2014) to encode game frames into a latent vector, and then use an RNN to mimic the VizDoom game environment, training on random rollouts from a random policy (i.e. selecting an action at random). Then controller policy is learned by playing within the “hallucinated” environment. Hafner et al. (2020) demonstrate that an RL agent can be trained entirely on episodes generated by a learned world model in latent space. Also close to our work is Kim et al. (2020), that use an LSTM architecture for modeling the world state, coupled with a convolutional decoder for producing output frames and jointly trained under an adversarial objective. While this approach seems to produce reasonable results for simple games like PacMan, it struggles with simulating the complex environment of VizDoom and produces blurry samples. In contrast, GameNGen is able to generate samples comparable to those of the original game, see Figure 2. Finally, concurrently with our work, Alonso et al. (2024) train a diffusion world model to predict the next observation given observation history, and iteratively train the world model and an RL model on Atari games. DOOM When DOOM released in 1993 it revolutionized the gaming industry. Introducing groundbreaking 3D graphics technology, it became a cornerstone of the first-person shooter genre, influencing countless other games. DOOM was studied by numerous research works. It provides an open-source implementation and a native resolution that is low enough for small sized models to simulate, while being complex enough to be a challenging test case. Finally, the authors have spent countless youth hours with the game. It was a trivial choice to use it in this work. DOOM Authors:

Dani Valevski
Yaniv Leviathan
Moab Arar
Shlomi Fruchter Authors: Dani Valevski
Yaniv Leviathan
Moab Arar
Shlomi Fruchter Dani Valevski Dani Valevski Yaniv Leviathan Yaniv Leviathan Moab Arar Moab Arar Shlomi Fruchter Shlomi Fruchter This paper is available on arxiv under CC by 4.0 Deed (Attribution 4.0 International) license. This paper is available on arxiv under CC by 4.0 Deed (Attribution 4.0 International) license. available on arxiv available on arxiv

Part of HackerNoon's growing list of open-source research papers, promoting free access to academic material.

Researchers Push Real-Time AI Game Simulation Beyond Traditional Engines

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

A Simple Guide to Blockchain Queries

Diffusion Models Are Real-Time Game Engines

Researchers Use Diffusion Models to Simulate DOOM Gameplay in Real Time

How Reinforcement Learning and Stable Diffusion Are Being Combined to Simulate Game Worlds

Study Finds AI-Generated Game Simulations Nearly Indistinguishable From Real Play

Finding AI-Generated Faces in the Wild: Model

A Simple Guide to Blockchain Queries

Diffusion Models Are Real-Time Game Engines

Researchers Use Diffusion Models to Simulate DOOM Gameplay in Real Time

How Reinforcement Learning and Stable Diffusion Are Being Combined to Simulate Game Worlds

Study Finds AI-Generated Game Simulations Nearly Indistinguishable From Real Play

Finding AI-Generated Faces in the Wild: Model

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps