Benchmarking Long-Form Factuality in Large Language Models

Table of Links Abstract and 1 Introduction 2 LongFact: Using LLMs to generate a multi-topic benchmark for long-form factuality 3 Safe:LLM agents as factuality autoraters 4 LLMs agents can be better factuality annotators than humans 5 F1@k: Extending F1 with recall from human-preferred length 6 Larger LLMs are more factual 7 Related Work 8 Limitations 9 Conclusion, Acknowledgments, Author Contribution, and References Appendix A. Frequently asked questions B. LongFact details C. SAFE details D. Metric details E. Further analysis 9 CONCLUSION In this paper, we examined how to thoroughly benchmark long-form factuality in large language models. To do so, we first used GPT-4 to generate LongFact, a set of 2,280 prompts spanning 38 different topics. We then proposed using language-model agents to automatically evaluate the longform factuality of a model’s response. This method, which we call SAFE, utilizes a search-enabled large language model to split a long-form response into individual facts, revise individual facts to be self-contained, determine the relevance of each individual fact to answering the prompt, and check the factuality of each relevant fact by issuing Google Search queries. Furthermore, we extended recall into the long-form domain by utilizing a hyperparameter K as a proxy for human-preferred length of responses, and we combined this measurement of recall with precision in F1@K. Empirically, we demonstrated that SAFE outperforms crowdsourced human annotators by agreeing with 72% of human annotations and winning 76% of examples out of a set of 100 randomly-sampled disagreement cases. We also showed that SAFE is more than 20× cheaper than crowdsourced human annotators. Moreover, we benchmarked thirteen models from four model families (Gemini, GPT, Claude, PaLM-2) on LongFact and found that larger language models generally had better long-form factuality. We make all of our code available at https://github.com/google-deepmind/ long-form-factuality. Future research in this area can explore a wide range of directions. First, a key avenue of exploration is how to improve a language model’s long-form factuality via better pretraining/finetuning or by augmenting them with external tool use. There also exists areas of improvement for SAFE in terms of reliance on search-enabled language-model agents (we discuss this further in Appendix C.6). Furthermore, our work considers factuality (i.e., correctness of facts with respect to world knowledge), and so it is still unclear how to reliably measure hallucination (i.e., correctness of facts with respect to a model’s internal knowledge) in long-form settings. Through our benchmark, we aim to have demonstrated how reliable methods of obtaining datasets, evaluating models, and aggregating metrics can significantly improve our understanding of model capabilities in long-form settings. We hope that our work encourages further research into both measuring and improving language models in long-form domains. ACKNOWLEDGEMENTS We thank Chen Liang for suggestions on improving LongFact. We thank Balaji Lakshminarayanan, Jeremiah Liu, Clara Huiyi Hu, Kelvin Guu, Ice Pasupat, Zhuyun Dai, Harshal Godhia, Vered Cohen, Eran Ofek, Chun-Sung Ferng, Da-Cheng Juan, Hao Zhou, Chenjie Gu, Qijun Tan, Jasper Snoek, Daniel Balle, Petru Gurita, Zhichun Wu, Norbert Kalb, Yanyan Zheng, Qiuchen Guo, John Nham, and Fangtao Li for helpful discussion on factuality autoraters. We thank Albert Webson, Adams Yu, Chen Zhu, Heng-Tze Cheng, Chenkai Kuang, YaGuang Li, and Xinyun Chen for insightful feedback on our initial ideas. We thank Denny Zhou, Slav Petrov, Le Hou, and Andrew Dai for providing feedback on the paper. AUTHOR CONTRIBUTIONS Chengrun Yang and Jerry Wei proposed framing the paper in the long-form setting. Jerry Wei prototyped using an LLM to automatically generate prompts, and Chengrun Yang proposed and prototyped generating topic-specific prompts for LongFact. Xinying Song, Jie Huang, Cosmo Du, and Chengrun Yang came up with the idea of developing a search-augmented LLM autorater, which was prototyped by Chengrun Yang, Jerry Wei, Yifeng Lu, Xinying Song, Jie Huang, and Cosmo Du. Yifeng Lu prototyped a multi-step agent approach using LLMs. Jerry Wei and Xinying Song implemented the current system of SAFE. Jerry Wei proposed names for LongFact and SAFE, and Nathan Hu proposed the name for F1@K. Nathan Hu proposed to incorporate human-expected length into the recall for F1@K. Jerry Wei led implementation and experimentation that resulted in the current codebase. Chengrun Yang, Yifeng Lu, Xinying Song, Nathan Hu, and Quoc V. Le helped develop ideas for experimentation. Chengrun Yang contributed to the codebase and experiments and examined experiment data, which influenced technical direction. Jerry Wei led open-sourcing efforts with help from Da Huang, Dustin Tran, and Chengrun Yang. Jerry Wei, Chengrun Yang, and Nathan Hu led paper writing. Quoc V. Le and Yifeng Lu advised the project. Quoc V. Le proposed the direction to work on factuality and hallucination. All other authors helped contribute feedback on paper framing. REFERENCES Anthropic. Claude 2, 2023. URL https://www.anthropic.com/news/claude-2. Anthropic. Introducing the next generation of Claude, 2024. URL https://www.anthropic. com/news/claude-3-family. Isabelle Augenstein, Timothy Baldwin, Meeyoung Cha, Tanmoy Chakraborty, Giovanni Luca Ciampaglia, David Corney, Renee DiResta, Emilio Ferrara, Scott Hale, Alon Halevy, Eduard Hovy, Heng Ji, Filippo Menczer, Ruben Miguez, Preslav Nakov, Dietram Scheufele, Shivam Sharma, and Giovanni Zagni. Factuality challenges in the era of large language models, 2023. URL https://arxiv.org/abs/2310.05189. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional AI: Harmlessness from AI feedback, 2022. URL https://arxiv.org/abs/2212.08073. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Conference on Neural Information Processing Systems, 2020. URL https://arxiv.org/abs/2005.14165. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with GPT-4, 2023. URL https://arxiv.org/abs/2303.12712. Shiqi Chen, Yiran Zhao, Jinghan Zhang, I-Chun Chern, Siyang Gao, Pengfei Liu, and Junxian He. FELM: Benchmarking factuality evaluation of large language models. In Conference on Neural Information Processing Systems, 2023. URL https://arxiv.org/abs/2310.00741. Furui Cheng, Vilém Zouhar, Simran Arora, Mrinmaya Sachan, Hendrik Strobelt, and Mennatallah El-Assady. RELIC: Investigating large language model responses using self-consistency, 2023a. URL https://arxiv.org/abs/2311.16842. Qinyuan Cheng, Tianxiang Sun, Wenwei Zhang, Siyin Wang, Xiangyang Liu, Mozhi Zhang, Junliang He, Mianqiu Huang, Zhangyue Yin, Kai Chen, and Xipeng Qiu. Evaluating hallucinations in chinese large language models, 2023b. URL https://arxiv.org/abs/2310.03368. I-Chun Chern, Steffi Chern, Shiqi Chen, Weizhe Yuan, Kehua Feng, Chunting Zhou, Junxian He, Graham Neubig, and Pengfei Liu. FacTool: Factuality detection in generative AI – a tool augmented framework for multi-task and multi-domain scenarios, 2023. URL https://arxiv.org/ abs/2307.13528. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, et al. PaLM: Scaling language modeling with Pathways, 2022. URL https://arxiv.org/abs/2204.02311. Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Conference on Neural Information Processing Systems, 2017. URL https://arxiv.org/abs/1706.03741. Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, and Jason Weston. Chain-of-verification reduces hallucination in large language models, 2023. URL https://arxiv.org/abs/2309.11495. Shahul Es, Jithin James, Luis Espinosa-Anke, and Steven Schockaert. RAGAS: Automated evaluation of retrieval augmented generation, 2023. URL https://arxiv.org/abs/2309.15217. Kavita Ganesan. ROUGE 2.0: Updated and improved measures for evaluation of summarization tasks, 2018. URL http://arxiv.org/abs/1803.01937. Luyu Gao, Zhuyun Dai, Panupong Pasupat, Anthony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vincent Y. Zhao, Ni Lao, Hongrae Lee, Da-Cheng Juan, and Kelvin Guu. RARR: Researching and revising what language models say, using language models. In Conference of the Association for Computational Linguistics, 2023. URL https://arxiv.org/abs/2210.08726. Google Gemini Team. Gemini: A family of highly capable multimodal models, 2023. URL https://arxiv.org/abs/2312.11805. Google. PaLM 2 technical report, 2023. URL https://arxiv.org/abs/2305.10403. Jian Guan, Jesse Dodge, David Wadden, Minlie Huang, and Hao Peng. Language models hallucinate, but may excel at fact verification, 2023. URL https://arxiv.org/abs/2310.14564. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021. URL https://arxiv.org/abs/2009.03300. Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions, 2023. URL https://arxiv.org/abs/2311.05232. Ryan Koo, Minhwa Lee, Vipul Raheja, Jong Inn Park, Zae Myung Kim, and Dongyeop Kang. Benchmarking cognitive biases in large language models as evaluators, 2023. URL https: //arxiv.org/abs/2309.17012. Junyi Li, Xiaoxue Cheng, Wayne Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. HaluEval: A largescale hallucination evaluation benchmark for large language models. In Conference on Empirical Methods in Natural Language Processing, 2023. URL https://arxiv.org/abs/2305. 11747. Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with AlphaCode. Science, 2022. URL https://arxiv.org/abs/2203.07814. Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Conference of the Association for Computational Linguistics, 2004. URL https://aclanthology.org/ W04-1013. Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. In Conference of the Association for Computational Linguistics, 2022. URL https: //arxiv.org/abs/2109.07958. Chaitanya Malaviya, Subin Lee, Sihao Chen, Elizabeth Sieber, Mark Yatskar, and Dan Roth. ExpertQA: Expert-curated questions and attributed answers. In 2024 Annual Conference of the North American Chapter of the Association for Computational Linguistics, 2024. URL https://arxiv.org/abs/2309.07852. Potsawee Manakul, Adian Liusie, and Mark JF Gales. SelfCheckGPT: Zero-resource black-box hallucination detection for generative large language models. In Conference on Empirical Methods in Natural Language Processing, 2023. URL https://arxiv.org/abs/2303.08896. Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. FActScore: Fine-grained atomic evaluation of factual precision in long form text generation. In Conference on Empirical Methods in Natural Language Processing, 2023. URL https://arxiv.org/abs/2305.14251. Dor Muhlgay, Ori Ram, Inbal Magar, Yoav Levine, Nir Ratner, Yonatan Belinkov, Omri Abend, Kevin Leyton-Brown, Amnon Shashua, and Yoav Shoham. Generating benchmarks for factuality evaluation of language models. In Conference of the European Chapter of the Association for Computational Linguistics, 2023. URL https://arxiv.org/abs/2307.06908. OpenAI. Introducing ChatGPT, 2022. URL https://openai.com/blog/chatgpt. OpenAI. GPT-4 technical report, 2023. URL https://arxiv.org/abs/2303.08774. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Conference on Neural Information Processing Systems, 2022. URL https://arxiv.org/abs/2203. 02155. Xiao Pu, Mingqi Gao, and Xiaojun Wan. Summarization is (almost) dead, 2023. URL https: //arxiv.org/abs/2309.09558. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Conference on Empirical Methods in Natural Language Processing, 2016. URL https://arxiv.org/abs/1606.05250. Thibault Sellam, Dipanjan Das, and Ankur P. Parikh. BLEURT: Learning robust metrics for text generation. In Conference of the Association for Computational Linguistics, 2020. URL https: //arxiv.org/abs/2004.04696. Katherine Tian, Eric Mitchell, Huaxiu Yao, Christopher D Manning, and Chelsea Finn. Fine-tuning language models for factuality, 2023. URL https://arxiv.org/abs/2311.08401. Tu Vu, Mohit Iyyer, Xuezhi Wang, Noah Constant, Jerry Wei, Jason Wei, Chris Tar, Yun-Hsuan Sung, Denny Zhou, Quoc Le, and Thang Luong. FreshLLMs: Refreshing large language models with search engine augmentation, 2023. URL https://arxiv.org/abs/2310.03214. David Wadden, Kyle Lo, Bailey Kuehl, Arman Cohan, Iz Beltagy, Lucy Lu Wang, and Hannaneh Hajishirzi. SciFact-Open: Towards open-domain scientific claim verification. In Findings of the Association for Computational Linguistics: Conference on Empirical Methods in Natural Language Processing 2022, 2022. URL https://arxiv.org/abs/2210.13777. Yuxia Wang, Revanth Gangi Reddy, Zain Muhammad Mujahid, Arnav Arora, Aleksandr Rubashevskii, Jiahui Geng, Osama Mohammed Afzal, Liangming Pan, Nadav Borenstein, Aditya Pillai, Isabelle Augenstein, Iryna Gurevych, and Preslav Nakov. Factcheck-GPT: End-toend fine-grained document-level fact-checking and correction of LLM output, 2023. URL https://arxiv.org/abs/2311.09000. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Conference on Neural Information Processing Systems, 2023a. URL https://arxiv.org/ abs/2201.11903. Jerry Wei, Le Hou, Andrew Lampinen, Xiangning Chen, Da Huang, Yi Tay, Xinyun Chen, Yifeng Lu, Denny Zhou, Tengyu Ma, and Quoc V. Le. Symbol tuning improves in-context learning in language models. In Conference on Empirical Methods in Natural Language Processing, 2023b. URL https://arxiv.org/abs/2305.08298. Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, and Tengyu Ma. Larger language models do in-context learning differently, 2023c. URL https://arxiv.org/abs/2303.03846. John Wieting, Kevin Gimpel, Graham Neubig, and Taylor Berg-Kirkpatrick. Paraphrastic representations at scale. In Conference on Empirical Methods in Natural Language Processing, 2022. URL https://arxiv.org/abs/2104.15114. Wenda Xu, Guanglei Zhu, Xuandong Zhao, Liangming Pan, Lei Li, and William Yang Wang. Perils of self-feedback: Self-bias amplifies in large language models, 2024. URL https://arxiv. org/abs/2402.11436. Xuan Zhang and Wei Gao. Towards LLM-based fact verification on news claims with a hierarchical step-by-step prompting method. In Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics, 2023. URL https://arxiv.org/abs/2310.00305. Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, Longyue Wang, Anh Tuan Luu, Wei Bi, Freda Shi, and Shuming Shi. Siren’s song in the AI ocean: A survey on hallucination in large language models, 2023. URL https://arxiv.org/abs/2309.01219. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-Judge with MT-Bench and chatbot arena. In Conference on Neural Information Processing Systems, 2023. URL https://arxiv.org/abs/2306.05685. This paper is available on arxiv under CC by 4.0 Deed license. Authors: (1) Jerry Wei, Google DeepMind and a Lead contributors; (2) Chengrun Yang, Google DeepMind and a Lead contributors; (3) Xinying Song, Google DeepMind and a Lead contributors; (4) Yifeng Lu, Google DeepMind and a Lead contributors; (5) Nathan Hu, Google DeepMind and Stanford University; (6) Jie Huang, Google DeepMind and University of Illinois at Urbana-Champaign; (7) Dustin Tran, Google DeepMind; (8) Daiyi Peng, Google DeepMind; (9) Ruibo Liu, Google DeepMind; (10) Da Huang, Google DeepMind; (11) Cosmo Du, Google DeepMind; (12) Quoc V. Le, Google DeepMind. Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 LongFact: Using LLMs to generate a multi-topic benchmark for long-form factuality 2 LongFact: Using LLMs to generate a multi-topic benchmark for long-form factuality 3 Safe:LLM agents as factuality autoraters 3 Safe:LLM agents as factuality autoraters 4 LLMs agents can be better factuality annotators than humans 4 LLMs agents can be better factuality annotators than humans 5 F1@k: Extending F1 with recall from human-preferred length 5 F1@k: Extending F1 with recall from human-preferred length 6 Larger LLMs are more factual 6 Larger LLMs are more factual 7 Related Work 7 Related Work 8 Limitations 8 Limitations 9 Conclusion, Acknowledgments, Author Contribution, and References 9 Conclusion, Acknowledgments, Author Contribution, and References Appendix Appendix A. Frequently asked questions A. Frequently asked questions B. LongFact details B. LongFact details C. SAFE details C. SAFE details D. Metric details D. Metric details E. Further analysis E. Further analysis 9 CONCLUSION In this paper, we examined how to thoroughly benchmark long-form factuality in large language models. To do so, we first used GPT-4 to generate LongFact, a set of 2,280 prompts spanning 38 different topics. We then proposed using language-model agents to automatically evaluate the longform factuality of a model’s response. This method, which we call SAFE, utilizes a search-enabled large language model to split a long-form response into individual facts, revise individual facts to be self-contained, determine the relevance of each individual fact to answering the prompt, and check the factuality of each relevant fact by issuing Google Search queries. Furthermore, we extended recall into the long-form domain by utilizing a hyperparameter K as a proxy for human-preferred length of responses, and we combined this measurement of recall with precision in F1@K. Empirically, we demonstrated that SAFE outperforms crowdsourced human annotators by agreeing with 72% of human annotations and winning 76% of examples out of a set of 100 randomly-sampled disagreement cases. We also showed that SAFE is more than 20× cheaper than crowdsourced human annotators. Moreover, we benchmarked thirteen models from four model families (Gemini, GPT, Claude, PaLM-2) on LongFact and found that larger language models generally had better long-form factuality. We make all of our code available at https://github.com/google-deepmind/ long-form-factuality. Future research in this area can explore a wide range of directions. First, a key avenue of exploration is how to improve a language model’s long-form factuality via better pretraining/finetuning or by augmenting them with external tool use. There also exists areas of improvement for SAFE in terms of reliance on search-enabled language-model agents (we discuss this further in Appendix C.6). Furthermore, our work considers factuality (i.e., correctness of facts with respect to world knowledge), and so it is still unclear how to reliably measure hallucination (i.e., correctness of facts with respect to a model’s internal knowledge) in long-form settings. Through our benchmark, we aim to have demonstrated how reliable methods of obtaining datasets, evaluating models, and aggregating metrics can significantly improve our understanding of model capabilities in long-form settings. We hope that our work encourages further research into both measuring and improving language models in long-form domains. ACKNOWLEDGEMENTS We thank Chen Liang for suggestions on improving LongFact. We thank Balaji Lakshminarayanan, Jeremiah Liu, Clara Huiyi Hu, Kelvin Guu, Ice Pasupat, Zhuyun Dai, Harshal Godhia, Vered Cohen, Eran Ofek, Chun-Sung Ferng, Da-Cheng Juan, Hao Zhou, Chenjie Gu, Qijun Tan, Jasper Snoek, Daniel Balle, Petru Gurita, Zhichun Wu, Norbert Kalb, Yanyan Zheng, Qiuchen Guo, John Nham, and Fangtao Li for helpful discussion on factuality autoraters. We thank Albert Webson, Adams Yu, Chen Zhu, Heng-Tze Cheng, Chenkai Kuang, YaGuang Li, and Xinyun Chen for insightful feedback on our initial ideas. We thank Denny Zhou, Slav Petrov, Le Hou, and Andrew Dai for providing feedback on the paper. AUTHOR CONTRIBUTIONS Chengrun Yang and Jerry Wei proposed framing the paper in the long-form setting. Jerry Wei prototyped using an LLM to automatically generate prompts, and Chengrun Yang proposed and prototyped generating topic-specific prompts for LongFact. Xinying Song, Jie Huang, Cosmo Du, and Chengrun Yang came up with the idea of developing a search-augmented LLM autorater, which was prototyped by Chengrun Yang, Jerry Wei, Yifeng Lu, Xinying Song, Jie Huang, and Cosmo Du. Yifeng Lu prototyped a multi-step agent approach using LLMs. Jerry Wei and Xinying Song implemented the current system of SAFE. Jerry Wei proposed names for LongFact and SAFE, and Nathan Hu proposed the name for F1@K. Nathan Hu proposed to incorporate human-expected length into the recall for F1@K. Jerry Wei led implementation and experimentation that resulted in the current codebase. Chengrun Yang, Yifeng Lu, Xinying Song, Nathan Hu, and Quoc V. Le helped develop ideas for experimentation. Chengrun Yang contributed to the codebase and experiments and examined experiment data, which influenced technical direction. Jerry Wei led open-sourcing efforts with help from Da Huang, Dustin Tran, and Chengrun Yang. Jerry Wei, Chengrun Yang, and Nathan Hu led paper writing. Quoc V. Le and Yifeng Lu advised the project. Quoc V. Le proposed the direction to work on factuality and hallucination. All other authors helped contribute feedback on paper framing. REFERENCES Anthropic. Claude 2, 2023. URL https://www.anthropic.com/news/claude-2. Anthropic. Introducing the next generation of Claude, 2024. URL https://www.anthropic. com/news/claude-3-family. Isabelle Augenstein, Timothy Baldwin, Meeyoung Cha, Tanmoy Chakraborty, Giovanni Luca Ciampaglia, David Corney, Renee DiResta, Emilio Ferrara, Scott Hale, Alon Halevy, Eduard Hovy, Heng Ji, Filippo Menczer, Ruben Miguez, Preslav Nakov, Dietram Scheufele, Shivam Sharma, and Giovanni Zagni. Factuality challenges in the era of large language models, 2023. URL https://arxiv.org/abs/2310.05189. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional AI: Harmlessness from AI feedback, 2022. URL https://arxiv.org/abs/2212.08073. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Conference on Neural Information Processing Systems, 2020. URL https://arxiv.org/abs/2005.14165. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with GPT-4, 2023. URL https://arxiv.org/abs/2303.12712. Shiqi Chen, Yiran Zhao, Jinghan Zhang, I-Chun Chern, Siyang Gao, Pengfei Liu, and Junxian He. FELM: Benchmarking factuality evaluation of large language models. In Conference on Neural Information Processing Systems, 2023. URL https://arxiv.org/abs/2310.00741. Furui Cheng, Vilém Zouhar, Simran Arora, Mrinmaya Sachan, Hendrik Strobelt, and Mennatallah El-Assady. RELIC: Investigating large language model responses using self-consistency, 2023a. URL https://arxiv.org/abs/2311.16842. Qinyuan Cheng, Tianxiang Sun, Wenwei Zhang, Siyin Wang, Xiangyang Liu, Mozhi Zhang, Junliang He, Mianqiu Huang, Zhangyue Yin, Kai Chen, and Xipeng Qiu. Evaluating hallucinations in chinese large language models, 2023b. URL https://arxiv.org/abs/2310.03368. I-Chun Chern, Steffi Chern, Shiqi Chen, Weizhe Yuan, Kehua Feng, Chunting Zhou, Junxian He, Graham Neubig, and Pengfei Liu. FacTool: Factuality detection in generative AI – a tool augmented framework for multi-task and multi-domain scenarios, 2023. URL https://arxiv.org/ abs/2307.13528. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, et al. PaLM: Scaling language modeling with Pathways, 2022. URL https://arxiv.org/abs/2204.02311. Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Conference on Neural Information Processing Systems, 2017. URL https://arxiv.org/abs/1706.03741. Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, and Jason Weston. Chain-of-verification reduces hallucination in large language models, 2023. URL https://arxiv.org/abs/2309.11495. Shahul Es, Jithin James, Luis Espinosa-Anke, and Steven Schockaert. RAGAS: Automated evaluation of retrieval augmented generation, 2023. URL https://arxiv.org/abs/2309.15217. Kavita Ganesan. ROUGE 2.0: Updated and improved measures for evaluation of summarization tasks, 2018. URL http://arxiv.org/abs/1803.01937. Luyu Gao, Zhuyun Dai, Panupong Pasupat, Anthony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vincent Y. Zhao, Ni Lao, Hongrae Lee, Da-Cheng Juan, and Kelvin Guu. RARR: Researching and revising what language models say, using language models. In Conference of the Association for Computational Linguistics, 2023. URL https://arxiv.org/abs/2210.08726. Google Gemini Team. Gemini: A family of highly capable multimodal models, 2023. URL https://arxiv.org/abs/2312.11805. Google. PaLM 2 technical report, 2023. URL https://arxiv.org/abs/2305.10403. Jian Guan, Jesse Dodge, David Wadden, Minlie Huang, and Hao Peng. Language models hallucinate, but may excel at fact verification, 2023. URL https://arxiv.org/abs/2310.14564. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021. URL https://arxiv.org/abs/2009.03300. Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions, 2023. URL https://arxiv.org/abs/2311.05232. Ryan Koo, Minhwa Lee, Vipul Raheja, Jong Inn Park, Zae Myung Kim, and Dongyeop Kang. Benchmarking cognitive biases in large language models as evaluators, 2023. URL https: //arxiv.org/abs/2309.17012. Junyi Li, Xiaoxue Cheng, Wayne Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. HaluEval: A largescale hallucination evaluation benchmark for large language models. In Conference on Empirical Methods in Natural Language Processing, 2023. URL https://arxiv.org/abs/2305. 11747. Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with AlphaCode. Science, 2022. URL https://arxiv.org/abs/2203.07814. Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Conference of the Association for Computational Linguistics, 2004. URL https://aclanthology.org/ W04-1013. Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. In Conference of the Association for Computational Linguistics, 2022. URL https: //arxiv.org/abs/2109.07958. Chaitanya Malaviya, Subin Lee, Sihao Chen, Elizabeth Sieber, Mark Yatskar, and Dan Roth. ExpertQA: Expert-curated questions and attributed answers. In 2024 Annual Conference of the North American Chapter of the Association for Computational Linguistics, 2024. URL https://arxiv.org/abs/2309.07852. Potsawee Manakul, Adian Liusie, and Mark JF Gales. SelfCheckGPT: Zero-resource black-box hallucination detection for generative large language models. In Conference on Empirical Methods in Natural Language Processing, 2023. URL https://arxiv.org/abs/2303.08896. Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. FActScore: Fine-grained atomic evaluation of factual precision in long form text generation. In Conference on Empirical Methods in Natural Language Processing, 2023. URL https://arxiv.org/abs/2305.14251. Dor Muhlgay, Ori Ram, Inbal Magar, Yoav Levine, Nir Ratner, Yonatan Belinkov, Omri Abend, Kevin Leyton-Brown, Amnon Shashua, and Yoav Shoham. Generating benchmarks for factuality evaluation of language models. In Conference of the European Chapter of the Association for Computational Linguistics, 2023. URL https://arxiv.org/abs/2307.06908. OpenAI. Introducing ChatGPT, 2022. URL https://openai.com/blog/chatgpt. OpenAI. GPT-4 technical report, 2023. URL https://arxiv.org/abs/2303.08774. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Conference on Neural Information Processing Systems, 2022. URL https://arxiv.org/abs/2203. 02155. Xiao Pu, Mingqi Gao, and Xiaojun Wan. Summarization is (almost) dead, 2023. URL https: //arxiv.org/abs/2309.09558. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Conference on Empirical Methods in Natural Language Processing, 2016. URL https://arxiv.org/abs/1606.05250. Thibault Sellam, Dipanjan Das, and Ankur P. Parikh. BLEURT: Learning robust metrics for text generation. In Conference of the Association for Computational Linguistics, 2020. URL https: //arxiv.org/abs/2004.04696. Katherine Tian, Eric Mitchell, Huaxiu Yao, Christopher D Manning, and Chelsea Finn. Fine-tuning language models for factuality, 2023. URL https://arxiv.org/abs/2311.08401. Tu Vu, Mohit Iyyer, Xuezhi Wang, Noah Constant, Jerry Wei, Jason Wei, Chris Tar, Yun-Hsuan Sung, Denny Zhou, Quoc Le, and Thang Luong. FreshLLMs: Refreshing large language models with search engine augmentation, 2023. URL https://arxiv.org/abs/2310.03214. David Wadden, Kyle Lo, Bailey Kuehl, Arman Cohan, Iz Beltagy, Lucy Lu Wang, and Hannaneh Hajishirzi. SciFact-Open: Towards open-domain scientific claim verification. In Findings of the Association for Computational Linguistics: Conference on Empirical Methods in Natural Language Processing 2022, 2022. URL https://arxiv.org/abs/2210.13777. Yuxia Wang, Revanth Gangi Reddy, Zain Muhammad Mujahid, Arnav Arora, Aleksandr Rubashevskii, Jiahui Geng, Osama Mohammed Afzal, Liangming Pan, Nadav Borenstein, Aditya Pillai, Isabelle Augenstein, Iryna Gurevych, and Preslav Nakov. Factcheck-GPT: End-toend fine-grained document-level fact-checking and correction of LLM output, 2023. URL https://arxiv.org/abs/2311.09000. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Conference on Neural Information Processing Systems, 2023a. URL https://arxiv.org/ abs/2201.11903. Jerry Wei, Le Hou, Andrew Lampinen, Xiangning Chen, Da Huang, Yi Tay, Xinyun Chen, Yifeng Lu, Denny Zhou, Tengyu Ma, and Quoc V. Le. Symbol tuning improves in-context learning in language models. In Conference on Empirical Methods in Natural Language Processing, 2023b. URL https://arxiv.org/abs/2305.08298. Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, and Tengyu Ma. Larger language models do in-context learning differently, 2023c. URL https://arxiv.org/abs/2303.03846. John Wieting, Kevin Gimpel, Graham Neubig, and Taylor Berg-Kirkpatrick. Paraphrastic representations at scale. In Conference on Empirical Methods in Natural Language Processing, 2022. URL https://arxiv.org/abs/2104.15114. Wenda Xu, Guanglei Zhu, Xuandong Zhao, Liangming Pan, Lei Li, and William Yang Wang. Perils of self-feedback: Self-bias amplifies in large language models, 2024. URL https://arxiv. org/abs/2402.11436. Xuan Zhang and Wei Gao. Towards LLM-based fact verification on news claims with a hierarchical step-by-step prompting method. In Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics, 2023. URL https://arxiv.org/abs/2310.00305. Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, Longyue Wang, Anh Tuan Luu, Wei Bi, Freda Shi, and Shuming Shi. Siren’s song in the AI ocean: A survey on hallucination in large language models, 2023. URL https://arxiv.org/abs/2309.01219. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-Judge with MT-Bench and chatbot arena. In Conference on Neural Information Processing Systems, 2023. URL https://arxiv.org/abs/2306.05685. This paper is available on arxiv under CC by 4.0 Deed license. This paper is available on arxiv under CC by 4.0 Deed license. available on arxiv available on arxiv Authors: (1) Jerry Wei, Google DeepMind and a Lead contributors; (2) Chengrun Yang, Google DeepMind and a Lead contributors; (3) Xinying Song, Google DeepMind and a Lead contributors; (4) Yifeng Lu, Google DeepMind and a Lead contributors; (5) Nathan Hu, Google DeepMind and Stanford University; (6) Jie Huang, Google DeepMind and University of Illinois at Urbana-Champaign; (7) Dustin Tran, Google DeepMind; (8) Daiyi Peng, Google DeepMind; (9) Ruibo Liu, Google DeepMind; (10) Da Huang, Google DeepMind; (11) Cosmo Du, Google DeepMind; (12) Quoc V. Le, Google DeepMind. Authors: Authors: (1) Jerry Wei, Google DeepMind and a Lead contributors; (2) Chengrun Yang, Google DeepMind and a Lead contributors; (3) Xinying Song, Google DeepMind and a Lead contributors; (4) Yifeng Lu, Google DeepMind and a Lead contributors; (5) Nathan Hu, Google DeepMind and Stanford University; (6) Jie Huang, Google DeepMind and University of Illinois at Urbana-Champaign; (7) Dustin Tran, Google DeepMind; (8) Daiyi Peng, Google DeepMind; (9) Ruibo Liu, Google DeepMind; (10) Da Huang, Google DeepMind; (11) Cosmo Du, Google DeepMind; (12) Quoc V. Le, Google DeepMind.