Table of Links Abstract and 1. Introduction and Related Work Abstract and 1. Introduction and Related Work Methodology
2.1 LLM-Sim Task
2.2 Data
2.3 Evaluation


Experiments


Results


Conclusion


Limitations and Ethical Concerns, Acknowledgements, and References Methodology
2.1 LLM-Sim Task
2.2 Data
2.3 Evaluation Methodology Methodology 2.1 LLM-Sim Task 2.1 LLM-Sim Task 2.2 Data 2.2 Data 2.3 Evaluation 2.3 Evaluation Experiments Experiments Experiments Results Results Results Conclusion Conclusion Conclusion Limitations and Ethical Concerns, Acknowledgements, and References Limitations and Ethical Concerns, Acknowledgements, and References Limitations and Ethical Concerns, Acknowledgements, and References A. Model details A. Model details B. Game transition examples B. Game transition examples C. Game rules generation C. Game rules generation D. Prompts D. Prompts E. GPT-3.5 results E. GPT-3.5 results F. Histograms F. Histograms D Prompts The prompts introduced in this section includes game rules that can either be human written rules or LLM generated rules. For experiments without game rules, we simply remove the rules from the corresponding prompts. D.1 Prompt Example: Fact D.1 Prompt Example: Fact D.1 Prompt Example: Fact D.1 Prompt Example: Fact D.1.2 State Difference Prediction D.1.2 State Difference Prediction D.2 Prompt Example: Fenv D.2.1 Full State Prediction D.2.2 State Difference Prediction D.2.2 State Difference Prediction D.3 Prompt Example: FR (Game Progress) D.4 Prompt Example: F D.4.1 Full State Prediction D.4.1 Full State Prediction D.4.2 State Difference Prediction D.4.2 State Difference Prediction D.5 Other Examples Below is an example of the rule of an action: Below is an example of the rule of an object: Below is an example of the score rule: Below is an example of a game state: Below is an example of a JSON that describes the difference of two game states: E GPT-3.5 results Table 5 and Table 6 shows the performance of a GPT-3.5 simulator predicting objects properties and game progress respectively. There is a huge gap between the GPT-4 performance and GPT-3.5 performance, providing yet another example of how fast LLM develops in the two years. It is also worth notices that the performance difference is larger when no rules is provided, indicating that GPT-3.5 is especially weak at applying common sense knowledge to this few-shot world simulation task. F Histograms 1. In Figure 3, we show detailed experimental results on the full state prediction task performed by GPT-4. full state prediction task GPT-4. 2. In Figure 4, we show detailed experimental results on the state difference prediction task performed by GPT-4. state difference prediction task 3. In Figure 5, we show detailed experimental results on the full state prediction task performed by GPT-3.5. full state prediction task GPT-3.5. 4. In Figure 6, we show detailed experimental results on the state difference prediction task performed by GPT-3.5. state difference prediction task GPT-3.5. Figure 3: GPT-4 - Full State prediction from a) Human-generated rules, b) LLM-generated rules, and c) No rules. Figure 4: GPT-4 - Difference prediction from a) Human-generated rules, b) LLM-generated rules, and c) No rules. Figure 5: GPT-3.5 - Full State prediction from a) Human-generated rules, b) LLM-generated rules, and c) No rules. Figure 6: GPT-3.5 - Difference prediction from a) Human-generated rules, b) LLM-generated rules, and c) No rules. Authors:
(1) Ruoyao Wang, University of Arizona (ruoyaowang@arizona.edu);
(2) Graham Todd, New York University (gdrtodd@nyu.edu);
(3) Ziang Xiao, Johns Hopkins University (ziang.xiao@jhu.edu);
(4) Xingdi Yuan, Microsoft Research Montréal (eric.yuan@microsoft.com);
(5) Marc-Alexandre Côté, Microsoft Research Montréal (macote@microsoft.com);
(6) Peter Clark, Allen Institute for AI (PeterC@allenai.org).;
(7) Peter Jansen, University of Arizona and Allen Institute for AI (pajansen@arizona.edu). Authors: Authors: (1) Ruoyao Wang, University of Arizona (ruoyaowang@arizona.edu); ruoyaowang@arizona.edu (2) Graham Todd, New York University (gdrtodd@nyu.edu); gdrtodd@nyu.edu (3) Ziang Xiao, Johns Hopkins University (ziang.xiao@jhu.edu); ziang.xiao@jhu.edu (4) Xingdi Yuan, Microsoft Research Montréal (eric.yuan@microsoft.com); eric.yuan@microsoft.com (5) Marc-Alexandre Côté, Microsoft Research Montréal (macote@microsoft.com); macote@microsoft.com (6) Peter Clark, Allen Institute for AI (PeterC@allenai.org).; PeterC@allenai.org (7) Peter Jansen, University of Arizona and Allen Institute for AI (pajansen@arizona.edu). pajansen@arizona.edu This paper is available on arxiv under CC BY 4.0 license. This paper is available on arxiv under CC BY 4.0 license. available on arxiv available on arxiv

Part of HackerNoon's growing list of open-source research papers, promoting free access to academic material.

Testing GPT-4 on Game State Predictions

AI Models on HackerNoon

Too Long; Didn't Read

Make resilience your competitive advantage

GPT-4 vs GPT-3.5 Performance in Game Simulations

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

102 Languages, One Model: The Multimodal AI Breakthrough You Need to Know

Are Large Language Models the Future of Game State Simulation?

Markov Chains, Rewards & Rules

Why GPT-4 Struggles with Complex Game Scenarios

AI Models Can't Be Trusted in High-Stakes Simulations Just Yet

102 Languages, One Model: The Multimodal AI Breakthrough You Need to Know

Are Large Language Models the Future of Game State Simulation?

Markov Chains, Rewards & Rules

Why GPT-4 Struggles with Complex Game Scenarios

AI Models Can't Be Trusted in High-Stakes Simulations Just Yet

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps