paint-brush
Orca 2: Enhancing Reasoning in Smaller Language Models - BigBench-Hard Subtask Metricsby@textmodels
101 reads

Orca 2: Enhancing Reasoning in Smaller Language Models - BigBench-Hard Subtask Metrics

tldt arrow

Too Long; Didn't Read

Teaching Orca 2 to be a Cautious Reasoner is available on arxiv.org under the CC-BY 4.0:11045 license. The paper is available online at: http://hackernoon.com/preview/jrEFMCvsvN9Qu9mXULgU.
featured image - Orca 2: Enhancing Reasoning in Smaller Language Models - BigBench-Hard Subtask Metrics
Writings, Papers and Blogs on Text Models HackerNoon profile picture

Authors:

(1) Arindam Mitra;

(2) Luciano Del Corro, work done while at Microsoft;

(3) Shweti Mahajan, work done while at Microsoft;

(4) Andres Codas, denote equal contributions;

(5) Clarisse Simoes, denote equal contributions;

(6) Sahaj Agarwal;

(7) Xuxi Chen, work done while at Microsoft;;

(8) Anastasia Razdaibiedina, work done while at Microsoft;

(9) Erik Jones, work done while at Microsoft;

(10) Kriti Aggarwal, work done while at Microsoft;

(11) Hamid Palangi;

(12) Guoqing Zheng;

(13) Corby Rosset;

(14) Hamed Khanpour;

(15) Ahmed Awadall.

Abstract and Introduction

Preliminaries

Teaching Orca 2 to be a Cautious Reasoner

Technical Details

Experimental Setup

Evaluation Results

Limitations

Conclusions and References

A. AGIEval Subtask Metrics

B. BigBench-Hard Subtask Metrics

C. Evaluation of Grounding in Abstractive Summarization

D. Evaluation of Safety

E. Prompts used in Evaluation

F. Illustrative Example from Evaluation Benchmarks and Corresponding Model Outpu

B BigBench-Hard Subtask Metrics

Table 7, 8, 9, and 10 showcase the zero-shot performance of Orca 2 and the baseline models on each BBH MCQ reasoning task, with accuracy being the metric used to evaluate performance.


Table 7: Zero-Shot performance of models on Tasks 1-6 within BBH benchmark.


Table 8: Zero-Shot performance of models on Tasks 7-14 within BBH benchmark.


Table 9: Zero-Shot performance of models on Tasks 15-20 within BBH benchmark.


Table 10: Zero-Shot performance of models on Tasks 21-24 within BBH benchmark.


This paper is available on arxiv under CC 4.0 license.