Authors:
(1) Pham Hoang Van, Department of Economics, Baylor University Waco, TX, USA (Van Pham@baylor.edu);
(2) Scott Cunningham, Department of Economics, Baylor University Waco, TX, USA (Scott Cunningham@baylor.edu). Authors: Authors: (1) Pham Hoang Van, Department of Economics, Baylor University Waco, TX, USA (Van Pham@baylor.edu); (2) Scott Cunningham, Department of Economics, Baylor University Waco, TX, USA (Scott Cunningham@baylor.edu). Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 Direct vs Narrative Prediction 2 Direct vs Narrative Prediction 3 Prompting Methodology and Data Collection 3 Prompting Methodology and Data Collection 4 Results 4.1 Establishing the Training Data Limit with Falsifications 4.1 Establishing the Training Data Limit with Falsifications 4.2 Results of the 2022 Academy Awards Forecasts 4.2 Results of the 2022 Academy Awards Forecasts 5 Predicting Macroeconomic Variables 5 Predicting Macroeconomic Variables 5.1 Predicting Inflation with an Economics Professor 5.1 Predicting Inflation with an Economics Professor 5.2 Predicting Inflation with a Jerome Powell, Fed Chair 5.2 Predicting Inflation with a Jerome Powell, Fed Chair 5.3 Predicting Inflation with Jerome Powell and Prompting with Russia’s Invasion of Ukraine 5.3 Predicting Inflation with Jerome Powell and Prompting with Russia’s Invasion of Ukraine 5.4 Predicting Unemployment with an Economics Professor 5.4 Predicting Unemployment with an Economics Professor 6 Conjecture on ChatGPT-4’s Predictive Abilities in Narrative Form 6 Conjecture on ChatGPT-4’s Predictive Abilities in Narrative Form 7 Conclusion and Acknowledgments 7 Conclusion and Acknowledgments Appendix Appendix A. Distribution of Predicted Academy Award Winners A. Distribution of Predicted Academy Award Winners B. Distribution of Predicted Macroeconomic Variables B. Distribution of Predicted Macroeconomic Variables References References 5 Predicting Macroeconomic Variables The selection of the Academy Awards as a predictive outcome is independently interesting. We also chose it because we thought it had a high change of success given the ample amount of writing on these movies and lead and supporting actors and actresses throughout the year. But now we move to macroeconomic phenomena that are regularly the subject of policy making and prediction. The prediction of macroeconomic variables is important because it helps individual, firms and government actors not just better plan today in light of possible future positive or negative news. It also can inform Fed decisions to engage in open market operations and other tools at its disposal to ease or tighten the money supply. While predicting Best Actor and predicting the inflation rate several months ahead of time are topically similar in that both require predicting real but unknown future events, they differ in important ways—some obvious, and some not so obvious. First, the prediction of Best Actor had a 20% chance of success under guessing. It was selecting a categorial event, not a right skewed potentially unbounded number ranging from 0% to something massive under hyperinflation scenarios. Even if higher values are unlikely, they are possible with a large language model that hallucinates. It is unclear what is in the training data, or to what degree large language models round continuous variables, as OpenAI has been secretive about the training data and has not shared the source code for ChatGPT-4. We bring these issues up simply to highlight that shifting from the Academy Awards to macroeconomic variables, even if large language models are somehow aggregating from its training data, these two types of predictions differ in their chances of success. The second thing is that the two predictions occur at different time periods. The 2022 Academy Awards were held on March 22nd, 2022, which was six months out from the training cutoff date. By contrast, we asked ChatGPT-4 to make several predictions regarding macro variables that required a monthly prediction over 12 months from October 2021 to September 2022. But the third thing is that it seems more likely that the Academy Awards is insensitive to trends. If anything, it is determined by trends as there are several earlier awards ceremonies (e.g., Director Guild Award) that are historically highly predictive of the various “Best” awards at the Oscars. Even though none of those are in the training data, either, it is unclear why earlier wins or losses might shift voting preferences at the Academy Awards. This is not the case, though, for macroeconomic variables because the Federal Reserve, insofar as it follows rules like a Taylor Rule, will respond to changing economic conditions with its policy levers to contract or expand the economy through monetary policy. This makes prediction challenging, and even if large language models could predict exogenous events, it may suffer from a built-in Lucas Critique problem if its training data has beliefs that are not based on the Taylor Rule. There were after all major world events that occurred between September 2021 and March 2022, such as Russia’s invasion of Ukraine, or higher than expected inflation, both of which may have had unknown effects that could have impacted domestic inflation and unemployment leading to predictions that overshoot or under-predict because of the Fed’s reliance on endogenous rules or discretion. We explore this prediction problem in detail by, again, asking direct (naive) and narrative prompting for 100 trials for both ChatGPT3.5 and ChatGPT-4 using our two RAs to minimize cascading bias but this time we repeat the experiment a second time by prompting ChatGPT-3.5 and ChatGPT-4 with additional information about Russia’s invasion of Ukraine in early 2022. This allows us to see if large language models ever attempt to utilize ceteris paribus style reasoning when aggregating the training data information, as there is no obvious reason why it necessarily should. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

Part of HackerNoon's growing list of open-source research papers, promoting free access to academic material.

Abstract

Even

Guild

Limit

OpenAI

AI Tries (and Fumbles) at Inflation Forecasting

We Asked ChatGPT To Predict Oscar Winners—the Results Were…Interesting

Precedent Tech 

Too Long; Didn't Read

Make resilience your competitive advantage

Can AI Predict Inflation? Testing ChatGPT on Macroeconomic Forecasting

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

AI Tries (and Fumbles) at Inflation Forecasting

Technological Trends Reshaping the Restaurant Industry in 2024

The Role of AI in Forecasting Economic Trends

Can ChatGPT Predict the Future?

Doctor, Doctor! AI Won’t Diagnose—Unless It’s in a Play

AI Tries (and Fumbles) at Inflation Forecasting

Technological Trends Reshaping the Restaurant Industry in 2024

The Role of AI in Forecasting Economic Trends

Can ChatGPT Predict the Future?

Doctor, Doctor! AI Won’t Diagnose—Unless It’s in a Play

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps