Table of Links Abstract and 1 Introduction 2 COCOGEN: Representing Commonsense structures with code and 2.1 Converting (T,G) into Python code 2.2 Few-shot prompting for generating G 3 Evaluation and 3.1 Experimental setup 3.2 Script generation: PROSCRIPT 3.3 Entity state tracking: PROPARA 3.4 Argument graph generation: EXPLAGRAPHS 4 Analysis 5 Related work 6 Conclusion, Acknowledgments, Limitations, and References A Few-shot models size estimates B Dynamic prompt Creation C Human Evaluation D Dataset statistics E Sample outputs F Prompts G Designing Python class for a structured task H Impact of Model size I Variation in prompts C Human Evaluation Out of the four tasks used in this work, PROSCRIPT edge prediction and PROPARA have only one possible correct value. Thus, following prior work, we report the automated, standard metrics for these tasks. For EXPLAGRAPHS, we use model-based metrics proposed by Saha et al. (2021), which were found to have a high correlation with human judgments. For PROSCRIPT graph generation, we conducted an exhaustive automated evaluation that separately scores the correctness of the nodes and the correctness of the edges. However, automated metrics are limited in their ability to evaluate model-generated output. Thus, to further investigate the quality of outputs, we conduct a human evaluation to compare the outputs generated by COCOGEN and DAVINCI. We sampled 20 examples, and three of the authors performed the evaluation. Annotators were shown two graphs (generated by COCOGEN and DAVINCI) and were asked to select one they thought was better regarding relevance and correctness. The selection for each criterion was made independently: the same graph could The annotations were done separately: the same graph could have more relevant nodes (higher relevance) but may not be correct. The identity of the model that generated each graph (COCOGEN or DAVINCI) was shuffled and unknown to the evaluators. The results in Table 11 indicate that human evaluation is closely correlated with the automated metrics: for EXPLAGRAPHS, annotators found the graphs generated by COCOGEN to be more relevant and correct. We find that DAVINCI often fails to recover semantic relations between nodes in the argument graphs. For example, consider a belief (B) urbanization harms natural habitats for the animals in the world. We want to generate a graph that can counter this belief with the argument (A) urbanization causes increase in jobs. For the same prompt, COCOGEN generated (urbanization; causes; increase in jobs); (increase in jobs; has context; good); (good; not capable of; harms) whereas DAVINCI generated (jobs; not harms; natural habitats) → (natural habitats; not part of; animals). Note that DAVINCI successfully recovered relevant events (“natural habitat” “animals”) but arranged them in incorrect relations. For PROSCRIPT, the human evaluation shows that COCOGEN and DAVINCI have complementary strengths, while COCOGEN generally produces more relevant and correct outputs. D Dataset statistics Dataset statistics are shown in Table 12. The test split for EXPLAGRAPHS is not available, so we evaluate on the validation split. For PROSCRIPT, we obtained the test splits from the authors. E Sample outputs Sample outputs from COCOGEN for all the tasks are located at https://github.com/ madaan/CoCoGen/tree/main/outputs. Representative examples from each task are presented in Figure 5. Surprisingly, COCOGEN (CODEX with a Python prompt) generates syntactically valid Python graphs that are similar to the task graphs/tables in nearly 100% of the cases. F Prompts The prompts for each tasks are present at this anonymous URL: PROSCRIPT script-generation: https://github.com/madaan/CoCoGen/tree/main/data/proscript_ script_generation/prompt.txt


PROSCRIPT edge-prediction: https://github.com/madaan/CoCoGen/tree/main/data/proscript_edge_ prediction/prompt.txt


PROPARA: https://github.com/madaan/CoCoGen/tree/main/data/ explagraphs/prompt.txt


EXPLAGRAPHS: https://github.com/madaan/CoCoGen/tree/main/data/ explagraphs/prompt.txt These prompts are also present in the attached supplementary material, and can be found in the data folder under respective task sub-directories. This paper is available on arxiv under CC BY 4.0 DEED license. Authors:
(1) Aman Madaan, Language Technologies Institute, Carnegie Mellon University, USA (amadaan@cs.cmu.edu);
(2) Shuyan Zhou, Language Technologies Institute, Carnegie Mellon University, USA (shuyanzh@cs.cmu.edu);
(3) Uri Alon, Language Technologies Institute, Carnegie Mellon University, USA (ualong@cs.cmu.edu);
(4) Yiming Yang, Language Technologies Institute, Carnegie Mellon University, USA (yiming@cs.cmu.edu);
(5) Graham Neubig, Language Technologies Institute, Carnegie Mellon University, USA (gneubig@cs.cmu.edu). Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 COCOGEN: Representing Commonsense structures with code and 2.1 Converting (T,G) into Python code 2 COCOGEN: Representing Commonsense structures with code and 2.1 Converting (T,G) into Python code 2.2 Few-shot prompting for generating G 2.2 Few-shot prompting for generating G 3 Evaluation and 3.1 Experimental setup 3 Evaluation and 3.1 Experimental setup 3.2 Script generation: PROSCRIPT 3.2 Script generation: PROSCRIPT 3.3 Entity state tracking: PROPARA 3.3 Entity state tracking: PROPARA 3.4 Argument graph generation: EXPLAGRAPHS 3.4 Argument graph generation: EXPLAGRAPHS 4 Analysis 4 Analysis 5 Related work 5 Related work 6 Conclusion, Acknowledgments, Limitations, and References 6 Conclusion, Acknowledgments, Limitations, and References A Few-shot models size estimates A Few-shot models size estimates B Dynamic prompt Creation B Dynamic prompt Creation C Human Evaluation C Human Evaluation D Dataset statistics D Dataset statistics E Sample outputs E Sample outputs F Prompts F Prompts G Designing Python class for a structured task G Designing Python class for a structured task H Impact of Model size H Impact of Model size I Variation in prompts I Variation in prompts C Human Evaluation Out of the four tasks used in this work, PROSCRIPT edge prediction and PROPARA have only one possible correct value. Thus, following prior work, we report the automated, standard metrics for these tasks. For EXPLAGRAPHS, we use model-based metrics proposed by Saha et al. (2021), which were found to have a high correlation with human judgments. For PROSCRIPT graph generation, we conducted an exhaustive automated evaluation that separately scores the correctness of the nodes and the correctness of the edges. However, automated metrics are limited in their ability to evaluate model-generated output. Thus, to further investigate the quality of outputs, we conduct a human evaluation to compare the outputs generated by COCOGEN and DAVINCI. We sampled 20 examples, and three of the authors performed the evaluation. Annotators were shown two graphs (generated by COCOGEN and DAVINCI) and were asked to select one they thought was better regarding relevance and correctness. The selection for each criterion was made independently: the same graph could The annotations were done separately: the same graph could have more relevant nodes (higher relevance) but may not be correct. The identity of the model that generated each graph (COCOGEN or DAVINCI) was shuffled and unknown to the evaluators. The results in Table 11 indicate that human evaluation is closely correlated with the automated metrics: for EXPLAGRAPHS, annotators found the graphs generated by COCOGEN to be more relevant and correct. We find that DAVINCI often fails to recover semantic relations between nodes in the argument graphs. For example, consider a belief (B) urbanization harms natural habitats for the animals in the world. We want to generate a graph that can counter this belief with the argument (A) urbanization causes increase in jobs. For the same prompt, COCOGEN generated (urbanization; causes; increase in jobs); (increase in jobs; has context; good); (good; not capable of; harms) whereas DAVINCI generated (jobs; not harms; natural habitats) → (natural habitats; not part of; animals). Note that DAVINCI successfully recovered relevant events (“natural habitat” “animals”) but arranged them in incorrect relations. For PROSCRIPT, the human evaluation shows that COCOGEN and DAVINCI have complementary strengths, while COCOGEN generally produces more relevant and correct outputs. D Dataset statistics Dataset statistics are shown in Table 12. The test split for EXPLAGRAPHS is not available, so we evaluate on the validation split. For PROSCRIPT, we obtained the test splits from the authors. E Sample outputs Sample outputs from COCOGEN for all the tasks are located at https://github.com/ madaan/CoCoGen/tree/main/outputs. Representative examples from each task are presented in Figure 5. Surprisingly, COCOGEN (CODEX with a Python prompt) generates syntactically valid Python graphs that are similar to the task graphs/tables in nearly 100% of the cases. F Prompts The prompts for each tasks are present at this anonymous URL: PROSCRIPT script-generation: https://github.com/madaan/CoCoGen/tree/main/data/proscript_ script_generation/prompt.txt PROSCRIPT edge-prediction: https://github.com/madaan/CoCoGen/tree/main/data/proscript_edge_ prediction/prompt.txt PROPARA: https://github.com/madaan/CoCoGen/tree/main/data/ explagraphs/prompt.txt EXPLAGRAPHS: https://github.com/madaan/CoCoGen/tree/main/data/ explagraphs/prompt.txt PROSCRIPT script-generation: https://github.com/madaan/CoCoGen/tree/main/data/proscript_ script_generation/prompt.txt PROSCRIPT script-generation: https: //github.com/madaan/CoCoGen/ tree/main/data/proscript_ script_generation/prompt.txt //github.com/madaan/CoCoGen/ PROSCRIPT edge-prediction: https://github.com/madaan/CoCoGen/tree/main/data/proscript_edge_ prediction/prompt.txt PROSCRIPT edge-prediction: https: //github.com/madaan/CoCoGen/ tree/main/data/proscript_edge_ prediction/prompt.txt //github.com/madaan/CoCoGen/ PROPARA: https://github.com/madaan/CoCoGen/tree/main/data/ explagraphs/prompt.txt PROPARA: https://github.com/ madaan/CoCoGen/tree/main/data/ explagraphs/prompt.txt https://github.com/ EXPLAGRAPHS: https://github.com/madaan/CoCoGen/tree/main/data/ explagraphs/prompt.txt EXPLAGRAPHS: https://github.com/ madaan/CoCoGen/tree/main/data/ explagraphs/prompt.txt https://github.com/ These prompts are also present in the attached supplementary material, and can be found in the data folder under respective task sub-directories. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv Authors: (1) Aman Madaan, Language Technologies Institute, Carnegie Mellon University, USA (amadaan@cs.cmu.edu); (2) Shuyan Zhou, Language Technologies Institute, Carnegie Mellon University, USA (shuyanzh@cs.cmu.edu); (3) Uri Alon, Language Technologies Institute, Carnegie Mellon University, USA (ualong@cs.cmu.edu); (4) Yiming Yang, Language Technologies Institute, Carnegie Mellon University, USA (yiming@cs.cmu.edu); (5) Graham Neubig, Language Technologies Institute, Carnegie Mellon University, USA (gneubig@cs.cmu.edu). Authors: Authors: (1) Aman Madaan, Language Technologies Institute, Carnegie Mellon University, USA (amadaan@cs.cmu.edu); (2) Shuyan Zhou, Language Technologies Institute, Carnegie Mellon University, USA (shuyanzh@cs.cmu.edu); (3) Uri Alon, Language Technologies Institute, Carnegie Mellon University, USA (ualong@cs.cmu.edu); (4) Yiming Yang, Language Technologies Institute, Carnegie Mellon University, USA (yiming@cs.cmu.edu); (5) Graham Neubig, Language Technologies Institute, Carnegie Mellon University, USA (gneubig@cs.cmu.edu).

Part of HackerNoon's growing list of open-source research papers, promoting free access to academic material.

COCOGEN vs DAVINCI: A Human Evaluation of Structured Commonsense Graph Generation

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

A Close Look at Misalignment in Pretraining Datasets

Complexity NIMBYs and the Illusion of Transcending Software Complexity Bounds

5 Practical Ways AI Can Boost Productivity for Web Developers

On the Concerns of Developers When Using GitHub Copilot

GitHub Copilot in Practice: Empirical Insights into User Experiences and Practical Challenges

Study Reveals Common Issues Faced by GitHub Copilot Users

A Close Look at Misalignment in Pretraining Datasets

Complexity NIMBYs and the Illusion of Transcending Software Complexity Bounds

5 Practical Ways AI Can Boost Productivity for Web Developers

On the Concerns of Developers When Using GitHub Copilot

GitHub Copilot in Practice: Empirical Insights into User Experiences and Practical Challenges

Study Reveals Common Issues Faced by GitHub Copilot Users

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps