113 reads

Orca 2: Enhancing Reasoning in Smaller Language Models - Prompts used in Evaluation

by Writings, Papers and Blogs on Text ModelsMay 29th, 2024

Too Long; Didn't Read

This paper is available on arxiv(https://arxiv.org/pdf/2311.11045.pdf) under CC 4.0 license. Authors: Arindam Mitra, Luciano Del Corro, Shweti Mahajan, Andres Codas, Guoqing Zheng, Corby Rosset, Hamed Khanpour, Ahmed Awadall.

featured image - Orca 2: Enhancing Reasoning in Smaller Language Models - Prompts used in Evaluation

Read by Dr. One voice-avatar

Listen to this story

Authors:

(1) Arindam Mitra;

(2) Luciano Del Corro, work done while at Microsoft;

(3) Shweti Mahajan, work done while at Microsoft;

(4) Andres Codas, denote equal contributions;

(5) Clarisse Simoes, denote equal contributions;

(6) Sahaj Agarwal;

(7) Xuxi Chen, work done while at Microsoft;;

(8) Anastasia Razdaibiedina, work done while at Microsoft;

(9) Erik Jones, work done while at Microsoft;

(10) Kriti Aggarwal, work done while at Microsoft;

(11) Hamid Palangi;

(12) Guoqing Zheng;

(13) Corby Rosset;

(14) Hamed Khanpour;

(15) Ahmed Awadall.

Table of Links

Abstract and Introduction

Preliminaries

Teaching Orca 2 to be a Cautious Reasoner

Conclusions and References

A. AGIEval Subtask Metrics

B. BigBench-Hard Subtask Metrics

C. Evaluation of Grounding in Abstractive Summarization

D. Evaluation of Safety

E. Prompts used in Evaluation

F. Illustrative Example from Evaluation Benchmarks and Corresponding Model Outpu

E Prompts used in Evaluation

We provide a list of prompts used for evaluation below:

Table 15: Table describes the prompts used for evaluating all models with empty. The prompts are simple and only aim at giving the models hints about answer format to improve the parsing of model responses. For tasks, where the question were formatted as a prompt, the input is used as is. Examples from all datasets are shown in Appendix F