Authors:
(1) Zhan Ling, UC San Diego and equal contribution;
(2) Yunhao Fang, UC San Diego and equal contribution;
(3) Xuanlin Li, UC San Diego;
(4) Zhiao Huang, UC San Diego;
(5) Mingu Lee, Qualcomm AI Research and Qualcomm AI Research
(6) Roland Memisevic, Qualcomm AI Research;
(7) Hao Su, UC San Diego. Table of Links Abstract and Introduction Related work Motivation and Problem Formulation Deductively Verifiable Chain-of-Thought Reasoning Experiments Limitations Conclusion, Acknowledgements and References A Deductive Verification with Vicuna Models B More Discussion on Improvements of Deductive Verification Accuracy Versus Improvements on Final Answer Correctness C More Details on Answer Extraction D Prompts E More Deductive Verification Examples D Prompts D.1 Prompt for Direct Reasoning Chain Verification Without Natural Program Format For the results in Tab. 2 of the main paper, We use “Do you think the above reasoning process is correct? Let’s think step by step.” as the zero-shot prompt to verify an entire reasoning chain at once. We also design a two-shot prompt for reasoning chain verification as shown in Tab. 12, which covers one correct reasoning chain and one incorrect reasoning chain. D.2 Prompts for Reasoning Chain Generation in the Natural Program Format To instruct models to generate reasoning chains in the Natural Program format that facilitates step-by-step deductive verification, we have designed four distinct prompts to address different types of problems. These include: Math word problems, as illustrated in Tab. 13, covering GSM8K, MATH, and AddSub datasets.


Math word problems with multiple-choice options, illustrated in Tab. 14, covering the AQuA dataset.


Date-related problems, illustrated in Tab. 15, covering the Date dataset.


Last Letters problems, illustrated in Tab. 16, covering the Last Letters dataset. D.3 Prompt for Deductive Verification Following Natural Program Format and Step-by-Step Decomposition We have designed a general one-shot prompt for the deductive verification of a single reasoning step on different datasets, as shown in Tab. 17. This prompt serves to instruct language models to generate the deductive validity of each reasoning step as illustrated in Sec. 4.2 and the top-right box of Fig. 1 of the main paper. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Zhan Ling, UC San Diego and equal contribution; (2) Yunhao Fang, UC San Diego and equal contribution; (3) Xuanlin Li, UC San Diego; (4) Zhiao Huang, UC San Diego; (5) Mingu Lee, Qualcomm AI Research and Qualcomm AI Research (6) Roland Memisevic, Qualcomm AI Research; (7) Hao Su, UC San Diego. Authors: Authors: (1) Zhan Ling, UC San Diego and equal contribution; (2) Yunhao Fang, UC San Diego and equal contribution; (3) Xuanlin Li, UC San Diego; (4) Zhiao Huang, UC San Diego; (5) Mingu Lee, Qualcomm AI Research and Qualcomm AI Research (6) Roland Memisevic, Qualcomm AI Research; (7) Hao Su, UC San Diego. Table of Links Abstract and Introduction Abstract and Introduction Related work Related work Motivation and Problem Formulation Motivation and Problem Formulation Deductively Verifiable Chain-of-Thought Reasoning Deductively Verifiable Chain-of-Thought Reasoning Experiments Experiments Limitations Limitations Conclusion, Acknowledgements and References Conclusion, Acknowledgements and References A Deductive Verification with Vicuna Models A Deductive Verification with Vicuna Models B More Discussion on Improvements of Deductive Verification Accuracy Versus Improvements on Final Answer Correctness B More Discussion on Improvements of Deductive Verification Accuracy Versus Improvements on Final Answer Correctness C More Details on Answer Extraction C More Details on Answer Extraction D Prompts D Prompts E More Deductive Verification Examples E More Deductive Verification Examples D Prompts D.1 Prompt for Direct Reasoning Chain Verification Without Natural Program Format For the results in Tab. 2 of the main paper, We use “Do you think the above reasoning process is correct? Let’s think step by step.” as the zero-shot prompt to verify an entire reasoning chain at once. We also design a two-shot prompt for reasoning chain verification as shown in Tab. 12, which covers one correct reasoning chain and one incorrect reasoning chain. D.2 Prompts for Reasoning Chain Generation in the Natural Program Format To instruct models to generate reasoning chains in the Natural Program format that facilitates step-by-step deductive verification, we have designed four distinct prompts to address different types of problems. These include: Math word problems, as illustrated in Tab. 13, covering GSM8K, MATH, and AddSub datasets. Math word problems with multiple-choice options, illustrated in Tab. 14, covering the AQuA dataset. Date-related problems, illustrated in Tab. 15, covering the Date dataset. Last Letters problems, illustrated in Tab. 16, covering the Last Letters dataset. Math word problems, as illustrated in Tab. 13, covering GSM8K, MATH, and AddSub datasets. Math word problems, as illustrated in Tab. 13, covering GSM8K, MATH, and AddSub datasets. Math word problems with multiple-choice options, illustrated in Tab. 14, covering the AQuA dataset. Math word problems with multiple-choice options, illustrated in Tab. 14, covering the AQuA dataset. Date-related problems, illustrated in Tab. 15, covering the Date dataset. Date-related problems, illustrated in Tab. 15, covering the Date dataset. Last Letters problems, illustrated in Tab. 16, covering the Last Letters dataset. Last Letters problems, illustrated in Tab. 16, covering the Last Letters dataset. D.3 Prompt for Deductive Verification Following Natural Program Format and Step-by-Step Decomposition We have designed a general one-shot prompt for the deductive verification of a single reasoning step on different datasets, as shown in Tab. 17. This prompt serves to instruct language models to generate the deductive validity of each reasoning step as illustrated in Sec. 4.2 and the top-right box of Fig. 1 of the main paper. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

Part of HackerNoon's growing list of open-source research papers, promoting free access to academic material.

Deductive Verification with Natural Programs: Case Studies

Essential Prompts for Reasoning Chain Verification and Natural Program Generation

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

A New Framework for Trustworthy AI Deductive Reasoning

The Noonification: How Often Do NFTs Pass The Howey Test? (1/13/2023)

Darwin's Hybrid Intelligence to Align AI & Human Goals for Startups & VCs

The Noonification: White Man (11/26/2022)

The Noonification: The Metaverse is a Sh*tshow (11/2/2022)

100 Days of AI Day 1: From Newsletter to Podcast, Leveraging AI for Audio Transformation

A New Framework for Trustworthy AI Deductive Reasoning

The Noonification: How Often Do NFTs Pass The Howey Test? (1/13/2023)

Darwin's Hybrid Intelligence to Align AI & Human Goals for Startups & VCs

The Noonification: White Man (11/26/2022)

The Noonification: The Metaverse is a Sh*tshow (11/2/2022)

100 Days of AI Day 1: From Newsletter to Podcast, Leveraging AI for Audio Transformation

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps