Table of Links Abstract and 1 Introduction 2 COCOGEN: Representing Commonsense structures with code and 2.1 Converting (T,G) into Python code 2.2 Few-shot prompting for generating G 3 Evaluation and 3.1 Experimental setup 3.2 Script generation: PROSCRIPT 3.3 Entity state tracking: PROPARA 3.4 Argument graph generation: EXPLAGRAPHS 4 Analysis 5 Related work 6 Conclusion, Acknowledgments, Limitations, and References A Few-shot models size estimates B Dynamic prompt Creation C Human Evaluation D Dataset statistics E Sample outputs F Prompts G Designing Python class for a structured task H Impact of Model size I Variation in prompts G Designing Python class for a structured task Figure 7 shows three different designs for Explagraphs. For PROSCRIPT, the various formats include representing proscript as a Networkx[8] class (8), DOT-like class 9, and as a Tree (10). H Impact of Model size The CODEX model released by OpenAI is available in two versions[9]: code-davinci-001 and code-davinci-002. While the exact sizes of the models are unknown because of their proprietary nature, OpenAI API states that code-davinci-002 is the Most capable Codex model Tables 16 and ?? compares COCOGEN +code-davinci-001 with COCOGEN +code-davinci-002. Note that both code-davinci-001 and code-davinci-002 can fit 4000 tokens, so the number of in-context examples was identical for the two settings. The results show that for identical prompts, COCOGEN +code-davinci-002 vastly outperforms COCOGEN +code-davinci-001, showing the importance of having a better underlying code generation model. Model size vs. sensitivity to the prompt In Table 14 shows the performance of CODEX-001 (smaller) and CODEX-002 (larger, also see Appendix A) on identical prompts. Our experiments show that as model size increases, the sensitivity of the model on the prompt design might get progressively easier. I Variation in prompts We run each experiment with 4 different random seeds, where the random seeds decide the order of examples in the prompt. We find minimal variance between runs using different fixed prompts between 3 runs. Further, as shown in the Table 18, 19, 20, and 21, all improvements of COCOGEN over DAVINCI are statistically (p-value < 0.001). This paper is available on arxiv under CC BY 4.0 DEED license. [9] as of June 2022 Authors:
(1) Aman Madaan, Language Technologies Institute, Carnegie Mellon University, USA (amadaan@cs.cmu.edu);
(2) Shuyan Zhou, Language Technologies Institute, Carnegie Mellon University, USA (shuyanzh@cs.cmu.edu);
(3) Uri Alon, Language Technologies Institute, Carnegie Mellon University, USA (ualong@cs.cmu.edu);
(4) Yiming Yang, Language Technologies Institute, Carnegie Mellon University, USA (yiming@cs.cmu.edu);
(5) Graham Neubig, Language Technologies Institute, Carnegie Mellon University, USA (gneubig@cs.cmu.edu). Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 COCOGEN: Representing Commonsense structures with code and 2.1 Converting (T,G) into Python code 2 COCOGEN: Representing Commonsense structures with code and 2.1 Converting (T,G) into Python code 2.2 Few-shot prompting for generating G 2.2 Few-shot prompting for generating G 3 Evaluation and 3.1 Experimental setup 3 Evaluation and 3.1 Experimental setup 3.2 Script generation: PROSCRIPT 3.2 Script generation: PROSCRIPT 3.3 Entity state tracking: PROPARA 3.3 Entity state tracking: PROPARA 3.4 Argument graph generation: EXPLAGRAPHS 3.4 Argument graph generation: EXPLAGRAPHS 4 Analysis 4 Analysis 5 Related work 5 Related work 6 Conclusion, Acknowledgments, Limitations, and References 6 Conclusion, Acknowledgments, Limitations, and References A Few-shot models size estimates A Few-shot models size estimates B Dynamic prompt Creation B Dynamic prompt Creation C Human Evaluation C Human Evaluation D Dataset statistics D Dataset statistics E Sample outputs E Sample outputs F Prompts F Prompts G Designing Python class for a structured task G Designing Python class for a structured task H Impact of Model size H Impact of Model size I Variation in prompts I Variation in prompts G Designing Python class for a structured task Figure 7 shows three different designs for Explagraphs. For PROSCRIPT, the various formats include representing proscript as a Networkx[8] class (8), DOT-like class 9, and as a Tree (10). H Impact of Model size The CODEX model released by OpenAI is available in two versions[9]: code-davinci-001 and code-davinci-002. While the exact sizes of the models are unknown because of their proprietary nature, OpenAI API states that code-davinci-002 is the Most capable Codex model Tables 16 and ?? compares COCOGEN +code-davinci-001 with COCOGEN +code-davinci-002. Note that both code-davinci-001 and code-davinci-002 can fit 4000 tokens, so the number of in-context examples was identical for the two settings. The results show that for identical prompts, COCOGEN +code-davinci-002 vastly outperforms COCOGEN +code-davinci-001, showing the importance of having a better underlying code generation model. Model size vs. sensitivity to the prompt In Table 14 shows the performance of CODEX-001 (smaller) and CODEX-002 (larger, also see Appendix A) on identical prompts. Our experiments show that as model size increases, the sensitivity of the model on the prompt design might get progressively easier. I Variation in prompts We run each experiment with 4 different random seeds, where the random seeds decide the order of examples in the prompt. We find minimal variance between runs using different fixed prompts between 3 runs. Further, as shown in the Table 18, 19, 20, and 21, all improvements of COCOGEN over DAVINCI are statistically (p-value < 0.001). This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv [9] as of June 2022 Authors: (1) Aman Madaan, Language Technologies Institute, Carnegie Mellon University, USA (amadaan@cs.cmu.edu); (2) Shuyan Zhou, Language Technologies Institute, Carnegie Mellon University, USA (shuyanzh@cs.cmu.edu); (3) Uri Alon, Language Technologies Institute, Carnegie Mellon University, USA (ualong@cs.cmu.edu); (4) Yiming Yang, Language Technologies Institute, Carnegie Mellon University, USA (yiming@cs.cmu.edu); (5) Graham Neubig, Language Technologies Institute, Carnegie Mellon University, USA (gneubig@cs.cmu.edu). Authors: Authors: (1) Aman Madaan, Language Technologies Institute, Carnegie Mellon University, USA (amadaan@cs.cmu.edu); (2) Shuyan Zhou, Language Technologies Institute, Carnegie Mellon University, USA (shuyanzh@cs.cmu.edu); (3) Uri Alon, Language Technologies Institute, Carnegie Mellon University, USA (ualong@cs.cmu.edu); (4) Yiming Yang, Language Technologies Institute, Carnegie Mellon University, USA (yiming@cs.cmu.edu); (5) Graham Neubig, Language Technologies Institute, Carnegie Mellon University, USA (gneubig@cs.cmu.edu).

Part of HackerNoon's growing list of open-source research papers, promoting free access to academic material.

Polkadot

Why Thousands of Examples Beat Dozens Every Time

COCOGEN vs DAVINCI: A Human Evaluation of Structured Commonsense Graph Generation

FEW SHOT .tech

Too Long; Didn't Read

Make resilience your competitive advantage

How CODEX Model Size Influences COCOGEN's Output Quality

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

What Happens When Your Prompt Is Too Obscure?

Complexity NIMBYs and the Illusion of Transcending Software Complexity Bounds

5 Practical Ways AI Can Boost Productivity for Web Developers

On the Concerns of Developers When Using GitHub Copilot

GitHub Copilot in Practice: Empirical Insights into User Experiences and Practical Challenges

What Happens When Your Prompt Is Too Obscure?

Complexity NIMBYs and the Illusion of Transcending Software Complexity Bounds

5 Practical Ways AI Can Boost Productivity for Web Developers

On the Concerns of Developers When Using GitHub Copilot

GitHub Copilot in Practice: Empirical Insights into User Experiences and Practical Challenges

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps