Table of Links Abstract and 1. Introduction and Related Work Abstract and 1. Introduction and Related Work Methodology
2.1 LLM-Sim Task
2.2 Data
2.3 Evaluation


Experiments


Results


Conclusion


Limitations and Ethical Concerns, Acknowledgements, and References Methodology
2.1 LLM-Sim Task
2.2 Data
2.3 Evaluation Methodology Methodology 2.1 LLM-Sim Task 2.1 LLM-Sim Task 2.2 Data 2.2 Data 2.3 Evaluation 2.3 Evaluation Experiments Experiments Experiments Results Results Results Conclusion Conclusion Conclusion Limitations and Ethical Concerns, Acknowledgements, and References Limitations and Ethical Concerns, Acknowledgements, and References Limitations and Ethical Concerns, Acknowledgements, and References A. Model details A. Model details B. Game transition examples B. Game transition examples C. Game rules generation C. Game rules generation D. Prompts D. Prompts E. GPT-3.5 results E. GPT-3.5 results F. Histograms F. Histograms A Model details For the GPT-3.5 model, we use the gpt-3.5-turbo-0125 model. For the GPT-4 model, we use the gpt-4-0125-preview model. For both models, the temperature is set to 0 to get deterministic results. We also turn on the JSON mode of both models, which ensures that the model gives a valid JSON response. Our experiments cost approximately $5,000 for OpenAI API usage. B Game transition examples We manually pick the wash-clothes game in BYTESIZED32 as the example game as it contains both state transitions driven by actions and game’s underlying dynamics. In tasks where the model predicts action transition, environment-driven transitions, or the game progress alone, we provide one corresponding in-context example. In the task that requires the model to predict everything, we offer two in-context examples in the prompt. The two examples are manually picked such that in one example the game state is changed directly by the action taken while in the other example the game state is changed by the game’s underlying dynamics. C Game rules generation C.1 LLM generated rules C.1 LLM generated rules For LLM generated rules, we manually check all of them to avoid misinformation and offensive content. We prompt GPT-4 (gpt-4-0125-preview) with the code of each object class to acquire the rules of each object. We also provide one in-context example. We ask GPT-4 to describe the meaning of each critical property (i.e. properties that do not inherit from parent) of the object and the tick function of the object (i.e. a function that defines how object properties may change at each time step regardless of the action taken). Below is an example of our prompt of object rule generation: For action rules generation, we prompt GPT-4 (gpt-4-0125-preview) with the code of the whole game, but unlike object rules, we do not offer any in-context example. We ask GPT-4 to describe each of the actions in the game. Below is an example of our prompt for action rule generation: Similar to action rules, we generate score rules by prompting GPT-4 (gpt-4-0125-preview) with the code of the game and ask GPT-4 to describe how the game can be won or lose and how rewards can be earned. Below is an example of our prompt for score rule generation: C.2 Human-Written Action Rules C.2 Human-Written Action Rules The action rules describe how each action can change the game states. The expert annotator reads the game description and source code for each game. They went through the list of available actions in the game and their corresponding functions in the game. Each action rule has three main parts: Action, Description, and Rules. The Action specifies the name of the action (e.g., action). The Description explains the general purpose of the action (e.g., connect two objects with input terminals). The Rules is an unordered list of rule descriptions that describe the constraints of the action when interacting with different objects (e.g., At least one of the objects should be a wire or a multimeter) or how the rule might function under different conditions (e.g., Disconnect terminal if the terminal is already connected to other objects). To ensure accuracy, the annotator plays through the game and checks if the written object rules were correctly reflected in the gameplay. C.3 Human-Written Object Rules C.3 Human-Written Object Rules The object rules describe the meaning of each object property (e.g., temperature, size, weight, etc.) and how they will be changed at each time step. The expert annotators read the game description and source code for each game. They went through the object classes in the code script and wrote the object rules. Each object rule has three main parts: Object, Description, and Properties. The Object specifies the name of the object. The Description explains the general purpose of the object (e.g., GarbageCan is a container that can hold garbage). In the Description, the inheritance of the object class has been noted. The Properties is an unordered list of property descriptions that describe each property of that object (e.g., A Mold has its shape.) and their default value (e.g., By default, a GameObject is not combustible.) if the object is an abstract class. For objects with tick function, there is another property describing how an object may change under each tick. To ensure accuracy, the annotator plays through the game and checks if the written object rules were correctly reflected in the gameplay. C.4 Human-Written Score Rules C.4 Human-Written Score Rules Score rules describe the conditions to win or lose the game and how rewards can be earned. An expert annotator (one of the BYTESIZED32 game authors) creates the rules by reading the game description and the code of the score function. Authors:
(1) Ruoyao Wang, University of Arizona (ruoyaowang@arizona.edu);
(2) Graham Todd, New York University (gdrtodd@nyu.edu);
(3) Ziang Xiao, Johns Hopkins University (ziang.xiao@jhu.edu);
(4) Xingdi Yuan, Microsoft Research Montréal (eric.yuan@microsoft.com);
(5) Marc-Alexandre Côté, Microsoft Research Montréal (macote@microsoft.com);
(6) Peter Clark, Allen Institute for AI (PeterC@allenai.org).;
(7) Peter Jansen, University of Arizona and Allen Institute for AI (pajansen@arizona.edu). Authors: Authors: (1) Ruoyao Wang, University of Arizona (ruoyaowang@arizona.edu); ruoyaowang@arizona.edu (2) Graham Todd, New York University (gdrtodd@nyu.edu); gdrtodd@nyu.edu (3) Ziang Xiao, Johns Hopkins University (ziang.xiao@jhu.edu); ziang.xiao@jhu.edu (4) Xingdi Yuan, Microsoft Research Montréal (eric.yuan@microsoft.com); eric.yuan@microsoft.com (5) Marc-Alexandre Côté, Microsoft Research Montréal (macote@microsoft.com); macote@microsoft.com (6) Peter Clark, Allen Institute for AI (PeterC@allenai.org).; PeterC@allenai.org (7) Peter Jansen, University of Arizona and Allen Institute for AI (pajansen@arizona.edu). pajansen@arizona.edu This paper is available on arxiv under CC BY 4.0 license. This paper is available on arxiv under CC BY 4.0 license. available on arxiv available on arxiv

Part of HackerNoon's growing list of open-source research papers, promoting free access to academic material.

OpenAI

Microsoft

AI Models Can't Be Trusted in High-Stakes Simulations Just Yet

AI Models on HackerNoon

Too Long; Didn't Read

Make resilience your competitive advantage

Testing GPT-4 on Game State Predictions

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

102 Languages, One Model: The Multimodal AI Breakthrough You Need to Know

Are Large Language Models the Future of Game State Simulation?

Markov Chains, Rewards & Rules

Why GPT-4 Struggles with Complex Game Scenarios

AI Models Can't Be Trusted in High-Stakes Simulations Just Yet

102 Languages, One Model: The Multimodal AI Breakthrough You Need to Know

Are Large Language Models the Future of Game State Simulation?

Markov Chains, Rewards & Rules

Why GPT-4 Struggles with Complex Game Scenarios

AI Models Can't Be Trusted in High-Stakes Simulations Just Yet

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps