Can GPT-4 Outdo Scientists? Testing AI’s Skills at Writing Reader-Friendly Science Summaries

Table of Links

• Abstract
• The Benefits of Simple Writing
• The Current Work
• Study 1a: Method
• Study 1a: Results
• Study 1b: Method
• Study 1b: Results
• Study 2: Method
• Study 2: Results
• General Discussion, Acknowledgements, and References

Study 1b: Method

An a priori power analysis using a small effect size (Cohen’s d = 0.20) powered at 80% suggested 788 cases were needed to detect a difference between GPT significance statements and PNAS significance statements. Therefore, a random selection of 800 abstracts from Study 1a was used in this study. Using the OpenAI API, the large language model GPT-4 was fed each abstract individually and given the following prompt, which was drawn from descriptions of what PNAS authors should communicate in their significance statements (31):

The following text is an academic abstract from the journal Proceedings of the National Academy of Sciences. Based on this abstract, create a significance statement. This statement should provide enough context for the paper’s implications to be clear to readers. The statement should not contain references and should avoid numbers, measurements, and acronyms unless necessary. It should explain the significance of the research at a level understandable to an undergraduate-educated scientist outside their field of specialty. Finally, it should include no more than 120 words. Write the significance statement here:

The same text analytic process was performed on these data as Study 1a. Each GPT significance statement received scores based on common words (LIWC dictionary category), analytic writing (LIWC analytic writing category), and readability (Flesch Reading Ease).