Humans vs. AI: Who Writes Simpler Science—and Who Gets the Blame for Complexity?

Table of Links

• Abstract
• The Benefits of Simple Writing
• The Current Work
• Study 1a: Method
• Study 1a: Results
• Study 1b: Method
• Study 1b: Results
• Study 2: Method
• Study 2: Results
• General Discussion, Acknowledgements, and References

The Current Work

The current empirical package evaluates fluency effects in the context of science writing and has several aims. The first aim is to evaluate how lay summaries of scientific articles (called significance statements in many journals) are indeed linguistically simpler compared to scientific summaries of the same articles (abstracts). It is unclear if scientists are aware of how to effectively summarize their work for non-experts (30), making this effort worthwhile to empirically test if ideals of a journal like simple and approachable writing are being realized (Study 1a). The second aim is to evaluate if such lay summaries can be made simpler. Study 1b had generative Artificial Intelligence (AI) and a popular large language model (GPT-4) create lay summaries based on paper abstracts and compared the linguistic properties of such texts.

Finally, building on this progression of studies, Study 2 tested the causal impact of reading scientific writing generated by AI (versus reading scientific writing generated by humans) on perceptions of scientists. Participants were randomly assigned to AI or human versions of a scientific summary, and they made judgments about the credibility, trustworthiness, and intelligence of the authors. To foreshadow the results: people preferred the simple (AI) version of each summary compared to the complex (human) version, yet ironically, people believed that the complex version was more likely to be AI than human.