Authors:
(1) Pham Hoang Van, Department of Economics, Baylor University Waco, TX, USA (Van Pham@baylor.edu);
(2) Scott Cunningham, Department of Economics, Baylor University Waco, TX, USA (Scott Cunningham@baylor.edu). Table of Links Abstract and 1 Introduction 2 Direct vs Narrative Prediction 3 Prompting Methodology and Data Collection 4 Results 4.1 Establishing the Training Data Limit with Falsifications 4.2 Results of the 2022 Academy Awards Forecasts 5 Predicting Macroeconomic Variables 5.1 Predicting Inflation with an Economics Professor 5.2 Predicting Inflation with a Jerome Powell, Fed Chair 5.3 Predicting Inflation with Jerome Powell and Prompting with Russia’s Invasion of Ukraine 5.4 Predicting Unemployment with an Economics Professor 6 Conjecture on ChatGPT-4’s Predictive Abilities in Narrative Form 7 Conclusion and Acknowledgments Appendix A. Distribution of Predicted Academy Award Winners B. Distribution of Predicted Macroeconomic Variables References 5.3 Predicting Inflation with Jerome Powell and Prompting with Russia’s Invasion of Ukraine On 24 February 2022, Russia invaded Ukraine in an escalation of the Russo-Ukrainian War which began in 2014. We gave this information to ChatGPT in a modified version of our Jerome Powell vignette. “Write a scene where Federal Reserve chairman Jerome Powell gives a speech in October 2022 about inflation, unemployment, and monetary policy. Russia had invaded Ukraine in February 24, 2022. In response, the U.S. and Europe are leading an embargo of Russia’s oil and gas exports. Chairman Powell tells the audience the inflation rate and unemployment rate for each month starting in September 2021 and ending in August 2022. Have chairman say each month one by one. He concludes with an outlook for inflation and unemployment and possible interest rate policy changes.” In Figure 17, we present results from this exercise using ChatGPT3.5. The comparison group for the Ukraine invasion in Figure 17 would be ChatGPT-3.5 in Figure 15, not ChatGPT-4 in Figure 16. First, the inclusion of this information caused greater variability in the guesses. Jerome Powell without the additional information about Russia’s invasion of Ukraine had median inflation rates that tracked the Cleveland Fed through November that then slowly drifted down to around 3.75 in May 2022. But in Figure 17, Jerome Powell when primed with news about the invasion provides a larger spread of answers covering around 3 percentage points in many cases. The median is lower and stays flat, secondly, then jumps sharply at news of the invasion in March 2022. Interestingly, one can see a slight uptick in the Michigan expectations number, too, that month, but not as severe as it is in Jerome Powell’s prediction. Overall, it is clear, though, that the inclusion of additional information caused a greater spread in the prediction most likely as it attempted to incorporate the information into its prediction. The performance of ChatGPT-4 in Figure 18 is a bit more puzzling, though. Its comparison should be thought of as Figure 16, the previous ChatGPT-4 exercise. As with Figure 17, median inflation rate guesses are somewhat lower with news about Russia’s invasion of Ukraine. The 12th month also has more variability with news of the Russian invasion than was seen without it. But the one things perhaps that is noteworthy is that ChatGPT-4 ignores the invasion when making predictions about the inflation rate in the month of the invasion, whereas ChatGPT-3.5 seems to explicitly alter its prediction based on it. With this information, the median was too low compared to both the Cleveland Fed and the Michigan Expectations, but the Michigan prediction is in the interval of our guesses. Interestingly, the Powell character incorporated the information we gave about Russia in his retelling of his past by shifting the distribution of guesses in March 2022, which was the next month after the invasion. For several months through August, the Powell character consistently retold his history where inflation rates were high and similar to what we experienced in reality. 5.4 Predicting Unemployment with an Economics Professor Next we turn to the results for unemployment rates. As with inflation, the direct prediction prompt yielded no guesses in any of the 100 trials (Figures 19 and 20) so we focus instead on the answers given to both future narrative prompts. There was no new prompt for the unemployment rate data as the previous vignettes had asked for the economics professor as well as Jerome Powell to state unemployment rates, as well as inflation rates, in the same responses. In Figure 21, the unnamed economics professor teaching a class on the Philips Curve listed unemployment rates that usually did not cover the true unemployment rate [5] when using ChatGPT-3.5. The use of ChatGPT-4 produces answers with less variation and a median that is usually closer to the truth, but which breaks down around March of 2022 in terms of overall accuracy. The unemployment predictions by Jerome Powell using ChatGPT-3.5 are shown in Figure 23. There is considerably more variation in guesses with more extreme outlier guesses with this prompt. But the median is still too high in most months; only the tails at best are close to the truth. When we use Jerome Powell with ChatGPT-4, the guesses are tight with a median that follows the secular decline in unemployment rates seen over the year, and estimates that cover the truth in all cases. In Figures 25 and 26, we report the ChatGPT-3.5 and ChatGPT-4 results when the Jerome Powell prompt is primed with information about Russia and Ukraine. Comparisons between Figure 25 and Figure 23 do not show dramatic differences in terms of spread and accuracy. And if anything, the accuracy of the model worsened in Figure 26 when ChatGPT-4 was first primed with information about the invasion before requesting the Jerome Powell future narrative. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Pham Hoang Van, Department of Economics, Baylor University Waco, TX, USA (Van Pham@baylor.edu); (2) Scott Cunningham, Department of Economics, Baylor University Waco, TX, USA (Scott Cunningham@baylor.edu). Authors: Authors: (1) Pham Hoang Van, Department of Economics, Baylor University Waco, TX, USA (Van Pham@baylor.edu); (2) Scott Cunningham, Department of Economics, Baylor University Waco, TX, USA (Scott Cunningham@baylor.edu). Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 Direct vs Narrative Prediction 2 Direct vs Narrative Prediction 3 Prompting Methodology and Data Collection 3 Prompting Methodology and Data Collection 4 Results 4.1 Establishing the Training Data Limit with Falsifications 4.1 Establishing the Training Data Limit with Falsifications 4.2 Results of the 2022 Academy Awards Forecasts 4.2 Results of the 2022 Academy Awards Forecasts 5 Predicting Macroeconomic Variables 5 Predicting Macroeconomic Variables 5.1 Predicting Inflation with an Economics Professor 5.1 Predicting Inflation with an Economics Professor 5.2 Predicting Inflation with a Jerome Powell, Fed Chair 5.2 Predicting Inflation with a Jerome Powell, Fed Chair 5.3 Predicting Inflation with Jerome Powell and Prompting with Russia’s Invasion of Ukraine 5.3 Predicting Inflation with Jerome Powell and Prompting with Russia’s Invasion of Ukraine 5.4 Predicting Unemployment with an Economics Professor 5.4 Predicting Unemployment with an Economics Professor 6 Conjecture on ChatGPT-4’s Predictive Abilities in Narrative Form 6 Conjecture on ChatGPT-4’s Predictive Abilities in Narrative Form 7 Conclusion and Acknowledgments 7 Conclusion and Acknowledgments Appendix Appendix A. Distribution of Predicted Academy Award Winners A. Distribution of Predicted Academy Award Winners B. Distribution of Predicted Macroeconomic Variables B. Distribution of Predicted Macroeconomic Variables References References 5.3 Predicting Inflation with Jerome Powell and Prompting with Russia’s Invasion of Ukraine On 24 February 2022, Russia invaded Ukraine in an escalation of the Russo-Ukrainian War which began in 2014. We gave this information to ChatGPT in a modified version of our Jerome Powell vignette. “Write a scene where Federal Reserve chairman Jerome Powell gives a speech in October 2022 about inflation, unemployment, and monetary policy. Russia had invaded Ukraine in February 24, 2022. In response, the U.S. and Europe are leading an embargo of Russia’s oil and gas exports. Chairman Powell tells the audience the inflation rate and unemployment rate for each month starting in September 2021 and ending in August 2022. Have chairman say each month one by one. He concludes with an outlook for inflation and unemployment and possible interest rate policy changes.” In Figure 17, we present results from this exercise using ChatGPT3.5. The comparison group for the Ukraine invasion in Figure 17 would be ChatGPT-3.5 in Figure 15, not ChatGPT-4 in Figure 16. First, the inclusion of this information caused greater variability in the guesses. Jerome Powell without the additional information about Russia’s invasion of Ukraine had median inflation rates that tracked the Cleveland Fed through November that then slowly drifted down to around 3.75 in May 2022. But in Figure 17, Jerome Powell when primed with news about the invasion provides a larger spread of answers covering around 3 percentage points in many cases. The median is lower and stays flat, secondly, then jumps sharply at news of the invasion in March 2022. Interestingly, one can see a slight uptick in the Michigan expectations number, too, that month, but not as severe as it is in Jerome Powell’s prediction. Overall, it is clear, though, that the inclusion of additional information caused a greater spread in the prediction most likely as it attempted to incorporate the information into its prediction. The performance of ChatGPT-4 in Figure 18 is a bit more puzzling, though. Its comparison should be thought of as Figure 16, the previous ChatGPT-4 exercise. As with Figure 17, median inflation rate guesses are somewhat lower with news about Russia’s invasion of Ukraine. The 12th month also has more variability with news of the Russian invasion than was seen without it. But the one things perhaps that is noteworthy is that ChatGPT-4 ignores the invasion when making predictions about the inflation rate in the month of the invasion, whereas ChatGPT-3.5 seems to explicitly alter its prediction based on it. With this information, the median was too low compared to both the Cleveland Fed and the Michigan Expectations, but the Michigan prediction is in the interval of our guesses. Interestingly, the Powell character incorporated the information we gave about Russia in his retelling of his past by shifting the distribution of guesses in March 2022, which was the next month after the invasion. For several months through August, the Powell character consistently retold his history where inflation rates were high and similar to what we experienced in reality. 5.4 Predicting Unemployment with an Economics Professor Next we turn to the results for unemployment rates. As with inflation, the direct prediction prompt yielded no guesses in any of the 100 trials (Figures 19 and 20) so we focus instead on the answers given to both future narrative prompts. There was no new prompt for the unemployment rate data as the previous vignettes had asked for the economics professor as well as Jerome Powell to state unemployment rates, as well as inflation rates, in the same responses. In Figure 21, the unnamed economics professor teaching a class on the Philips Curve listed unemployment rates that usually did not cover the true unemployment rate [5] when using ChatGPT-3.5. The use of ChatGPT-4 produces answers with less variation and a median that is usually closer to the truth, but which breaks down around March of 2022 in terms of overall accuracy. The unemployment predictions by Jerome Powell using ChatGPT-3.5 are shown in Figure 23. There is considerably more variation in guesses with more extreme outlier guesses with this prompt. But the median is still too high in most months; only the tails at best are close to the truth. When we use Jerome Powell with ChatGPT-4, the guesses are tight with a median that follows the secular decline in unemployment rates seen over the year, and estimates that cover the truth in all cases. In Figures 25 and 26, we report the ChatGPT-3.5 and ChatGPT-4 results when the Jerome Powell prompt is primed with information about Russia and Ukraine. Comparisons between Figure 25 and Figure 23 do not show dramatic differences in terms of spread and accuracy. And if anything, the accuracy of the model worsened in Figure 26 when ChatGPT-4 was first primed with information about the invasion before requesting the Jerome Powell future narrative. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

Part of HackerNoon's growing list of open-source research papers, promoting free access to academic material.

Conjecture on ChatGPT-4’s Predictive Abilities in Narrative Form

AI Tries (and Fumbles) at Inflation Forecasting

Precedent Tech 

Too Long; Didn't Read

Make resilience your competitive advantage

What Happens When You Ask AI to Predict Inflation?

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

ChatGPT's Macroeconomic Predictions

Technological Trends Reshaping the Restaurant Industry in 2024

The Role of AI in Forecasting Economic Trends

Can ChatGPT Predict the Future?

Doctor, Doctor! AI Won’t Diagnose—Unless It’s in a Play

ChatGPT's Macroeconomic Predictions

Technological Trends Reshaping the Restaurant Industry in 2024

The Role of AI in Forecasting Economic Trends

Can ChatGPT Predict the Future?

Doctor, Doctor! AI Won’t Diagnose—Unless It’s in a Play

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps