Authors:
(1) PIOTR MIROWSKI and KORY W. MATHEWSON, DeepMind, United Kingdom and Both authors contributed equally to this research;
(2) JAYLEN PITTMAN, Stanford University, USA and Work done while at DeepMind;
(3) RICHARD EVANS, DeepMind, United Kingdom. Table of Links Abstract and Intro Storytelling, The Shape of Stories, and Log Lines The Use of Large Language Models for Creative Text Generation Evaluating Text Generated by Large Language Models Participant Interviews Participant Surveys Discussion and Future Work Conclusions, Acknowledgements, and References A. RELATED WORK ON AUTOMATED STORY GENERATION AND CONTROLLABLE STORY GENERATION B. ADDITIONAL DISCUSSION FROM PLAYS BY BOTS CREATIVE TEAM C. DETAILS OF QUANTITATIVE OBSERVATIONS D. SUPPLEMENTARY FIGURES E. FULL PROMPT PREFIXES FOR DRAMATRON F. RAW OUTPUT GENERATED BY DRAMATRON G. CO-WRITTEN SCRIPTS 4 EVALUATING TEXT GENERATED BY LARGE LANGUAGE MODELS In this section, we first review existing methods for evaluating LLM-generated text before presenting our approach. Echoing Celikyilmaz et al. [16], we split evaluation methods into automated or machine-learned metrics, and humancentric evaluation. Automated and machine-learned metrics (reviewed in Appendix A.6) typically calculate the similarity between generated and “ground truth” stories, the consistency between generated stories and their writing prompts, or the diversity of language in the generated output. These metrics were not designed for generated text of the length of a screenplay or theatre script. This motivates a focus on human-centric evaluation, which can be conducted with naïve crowdworkers or with experts. We now review the limitations of crowdsourced evaluation of the coherence of generated text, and explain why non-expert, crowdsourced evaluation faces crucial quality and bias issues. Inheriting from research standards for large-scale natural language processing tasks, the majority of studies assessing the quality of generations from LLMs evaluate model performance by collecting data from crowdworkers [56, 75, 77, 85, 88, 121]. For instance Yao et al. [121] recruit crowdworkers to evaluate fidelity, coherence, interestingness, and popularity, and Rashkin et al. [85] to evaluate narrative flow and ordering of events. That said, it has been shown that crowdworkers’ personal opinions, demographic characteristics and cognitive biases [35] can affect the quality of crowdsourced annotations in fact-checking [30] or in tasks involving subjective assessments [51]. These issues have led some researchers to try evaluating their models with experts. Karpinska et al. [54] highlight the perils of using crowdsourced workers for evaluating open-ended generated text, because crowdworkers do not read text as carefully as expert teachers. Some studies consulted with expert linguists [31], university students in the sciences [41] or humanities [120], amateur writers [2, 21, 60, 97, 122]. In one recent study, Calderwood et al. [13] interviewed 4 novelists about their usage of GPT-2 via Talk To Transformer and Write With Transformer tools (see https://app.inferkit.com/demo), uncovering usages of the “model as antagonist” (i.e., random), for “description creation”, as “constraint”, or for “unexpected” ideas. Given the issues discussed above, we believe that crowdsourcing are not an effective approach to evaluating screenplays and theatre scripts co-written with language models. Thus, in a departure from crowd-sourced evaluations, we engage 15 experts—theatre and film industry professionals—who have both an experience in using AI writing tools and who have worked in TV, film or theatre in one of these capacities: writer, actor, director or producer. These experts participate in a 2-hour session wherein they co-write a screenplay or theatre script alongside Dramatron. Most were able to co-write a full script and the open discussion interview in the allotted 2-hours; for the others, we slightly extended the interview session. Interviews are analysed in Section 5, and following the interactive sessions, the participants were asked a series of questions adapted from [122] and detailed in Section 6. Each question was answered on a 5-point Likert-type scale, using questions adapted from [22, 56, 77, 104, 122]. As another quantitative evaluation, we track writer modifications to generated sentences [88]. This allows for comparison of Dramatron generations pre- and post-human edits. We track absolute and relative word edit distance, to assess whether and how much the writer adds to or removes from the output suggestions. We also track a Jaccard similarity-based metric on words, to quantify how similar is the draft after edits to the original suggestion. Our objective is to assess whether Dramatron “contributes new ideas to writing”, or “merely expand[s] on [the writer’s] existing ideas” [60]. We do not evaluate Dramatron outputs for grammatical correctness as [77], as the few errors made by the LLM can be fixed by the human co-writer. We compensate experts for the co-writing and interview sessions at 100 GBP per hour. Our study design and data collection process was validated and approved by HuBREC (Human Behavioral Research Ethics Committee), which is a research ethics committee run within Deepmind which includes and is chaired by academics from outside the company. This paper is available on arxiv under CC 4.0 license. Authors: (1) PIOTR MIROWSKI and KORY W. MATHEWSON, DeepMind, United Kingdom and Both authors contributed equally to this research; (2) JAYLEN PITTMAN, Stanford University, USA and Work done while at DeepMind; (3) RICHARD EVANS, DeepMind, United Kingdom. Authors: Authors: (1) PIOTR MIROWSKI and KORY W. MATHEWSON, DeepMind, United Kingdom and Both authors contributed equally to this research; (2) JAYLEN PITTMAN, Stanford University, USA and Work done while at DeepMind; (3) RICHARD EVANS, DeepMind, United Kingdom. Table of Links Abstract and Intro Abstract and Intro Storytelling, The Shape of Stories, and Log Lines Storytelling, The Shape of Stories, and Log Lines The Use of Large Language Models for Creative Text Generation The Use of Large Language Models for Creative Text Generation Evaluating Text Generated by Large Language Models Evaluating Text Generated by Large Language Models Participant Interviews Participant Interviews Participant Surveys Participant Surveys Discussion and Future Work Discussion and Future Work Conclusions, Acknowledgements, and References Conclusions, Acknowledgements, and References A. RELATED WORK ON AUTOMATED STORY GENERATION AND CONTROLLABLE STORY GENERATION A. RELATED WORK ON AUTOMATED STORY GENERATION AND CONTROLLABLE STORY GENERATION B. ADDITIONAL DISCUSSION FROM PLAYS BY BOTS CREATIVE TEAM B. ADDITIONAL DISCUSSION FROM PLAYS BY BOTS CREATIVE TEAM C. DETAILS OF QUANTITATIVE OBSERVATIONS C. DETAILS OF QUANTITATIVE OBSERVATIONS D. SUPPLEMENTARY FIGURES D. SUPPLEMENTARY FIGURES E. FULL PROMPT PREFIXES FOR DRAMATRON E. FULL PROMPT PREFIXES FOR DRAMATRON F. RAW OUTPUT GENERATED BY DRAMATRON F. RAW OUTPUT GENERATED BY DRAMATRON G. CO-WRITTEN SCRIPTS G. CO-WRITTEN SCRIPTS 4 EVALUATING TEXT GENERATED BY LARGE LANGUAGE MODELS In this section, we first review existing methods for evaluating LLM-generated text before presenting our approach. Echoing Celikyilmaz et al. [16], we split evaluation methods into automated or machine-learned metrics, and humancentric evaluation. Automated and machine-learned metrics (reviewed in Appendix A.6) typically calculate the similarity between generated and “ground truth” stories, the consistency between generated stories and their writing prompts, or the diversity of language in the generated output. These metrics were not designed for generated text of the length of a screenplay or theatre script. This motivates a focus on human-centric evaluation, which can be conducted with naïve crowdworkers or with experts. We now review the limitations of crowdsourced evaluation of the coherence of generated text, and explain why non-expert, crowdsourced evaluation faces crucial quality and bias issues. Inheriting from research standards for large-scale natural language processing tasks, the majority of studies assessing the quality of generations from LLMs evaluate model performance by collecting data from crowdworkers [56, 75, 77, 85, 88, 121]. For instance Yao et al. [121] recruit crowdworkers to evaluate fidelity, coherence, interestingness, and popularity, and Rashkin et al. [85] to evaluate narrative flow and ordering of events. That said, it has been shown that crowdworkers’ personal opinions, demographic characteristics and cognitive biases [35] can affect the quality of crowdsourced annotations in fact-checking [30] or in tasks involving subjective assessments [51]. These issues have led some researchers to try evaluating their models with experts. Karpinska et al. [54] highlight the perils of using crowdsourced workers for evaluating open-ended generated text, because crowdworkers do not read text as carefully as expert teachers. Some studies consulted with expert linguists [31], university students in the sciences [41] or humanities [120], amateur writers [2, 21, 60, 97, 122]. In one recent study, Calderwood et al. [13] interviewed 4 novelists about their usage of GPT-2 via Talk To Transformer and Write With Transformer tools (see https://app.inferkit.com/demo), uncovering usages of the “model as antagonist” (i.e., random), for “description creation”, as “constraint”, or for “unexpected” ideas. Given the issues discussed above, we believe that crowdsourcing are not an effective approach to evaluating screenplays and theatre scripts co-written with language models. Thus, in a departure from crowd-sourced evaluations, we engage 15 experts—theatre and film industry professionals—who have both an experience in using AI writing tools and who have worked in TV, film or theatre in one of these capacities: writer, actor, director or producer. These experts participate in a 2-hour session wherein they co-write a screenplay or theatre script alongside Dramatron. Most were able to co-write a full script and the open discussion interview in the allotted 2-hours; for the others, we slightly extended the interview session. Interviews are analysed in Section 5, and following the interactive sessions, the participants were asked a series of questions adapted from [122] and detailed in Section 6. Each question was answered on a 5-point Likert-type scale, using questions adapted from [22, 56, 77, 104, 122]. As another quantitative evaluation, we track writer modifications to generated sentences [88]. This allows for comparison of Dramatron generations pre- and post-human edits. We track absolute and relative word edit distance, to assess whether and how much the writer adds to or removes from the output suggestions. We also track a Jaccard similarity-based metric on words, to quantify how similar is the draft after edits to the original suggestion. Our objective is to assess whether Dramatron “contributes new ideas to writing”, or “merely expand[s] on [the writer’s] existing ideas” [60]. We do not evaluate Dramatron outputs for grammatical correctness as [77], as the few errors made by the LLM can be fixed by the human co-writer. We compensate experts for the co-writing and interview sessions at 100 GBP per hour. Our study design and data collection process was validated and approved by HuBREC (Human Behavioral Research Ethics Committee), which is a research ethics committee run within Deepmind which includes and is chaired by academics from outside the company. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

Part of HackerNoon's growing list of open-source research papers, promoting free access to academic material.

Evaluating LLM-Generated Text: Methods, Limitations, and Human-Centric Approaches

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Advancing Coherent Story Generation: Future Directions for AI Writing Tools

Co-Writing Screenplays and Theatre Scripts with Language Models: Notes from Industrial Professionals

Storytelling, The Shape of Stories, and Log Lines

Understanding Statistical Language Models and Hierarchical Language Generation

Professional Screenwriters Share Pros and Cons of Co-Writing with AI

How Exactly Does Dramatron Cook Stories Up

Advancing Coherent Story Generation: Future Directions for AI Writing Tools

Co-Writing Screenplays and Theatre Scripts with Language Models: Notes from Industrial Professionals

Storytelling, The Shape of Stories, and Log Lines

Understanding Statistical Language Models and Hierarchical Language Generation

Professional Screenwriters Share Pros and Cons of Co-Writing with AI

How Exactly Does Dramatron Cook Stories Up

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps