Zero-shot Prompts for Logical Reasoning Tasks in Biological Pathways

Table of Links

1. Abstract and Introduction
2. SylloBio-NLI
3. Empirical Evaluation
4. Related Work
5. Conclusions
6. Limitations and References

A. Formalization of the SylloBio-NLI Resource Generation Process

B. Formalization of Tasks 1 and 2

C. Dictionary of gene and pathway membership

D. Domain-specific pipeline for creating NL instances and E Accessing LLMs

F. Experimental Details

G. Evaluation Metrics

H. Prompting LLMs - Zero-shot prompts

I. Prompting LLMs - Few-shot prompts

J. Results: Misaligned Instruction-Response

K. Results: Ambiguous Impact of Distractors on Reasoning

L. Results: Models Prioritize Contextual Knowledge Over Background Knowledge

M Supplementary Figures and N Supplementary Tables

H Prompting LLMs - Zero-shot prompts

H.1 TASK 1

prompt = Suppose you are a specialist with existing knowledge about a signaling and metabolic molecules and their relations organized into biological pathways and processes. Given premises marked with the letter P and the following number and the conclusion marked with the letter C, determine whether the conclusion logically follows from these premises. If the conclusion logically follows from the premises, you need to return ’True’. If the conclusion does not follow logically from the premises, you need to return ’False’. The output should be a single word <True> or <False>.

"P1: " + Premise 1

"P2: " + Premise 2

"C:" + Conclusion

H.2 TASK 2

prompt = Suppose you are a specialist with existing knowledge about a signaling and metabolic molecules and their relations organized into biological pathways and processes. Given premises marked with the letter P and the following number and the conclusion marked with the letter C, determine whether the conclusion logically follows from these premises. If the conclusion logically follows from the premises, you need to return ’True’. If the conclusion does not follow logically from the premises, you need to return ’False’. Specify the premises you used to determine whether the conclusion logically follows from the premises, and only these premises. The output should be a single word <True> or <False>and the numbers of the selected premises after the decimal point, like <True, P1, P2>.

"P1: " + Premise 1

"P2: " + Premise 2

"C:" + Conclusion