How to Test Logical Validity in Biomedical Research

Table of Links

1. Abstract and Introduction
2. SylloBio-NLI
3. Empirical Evaluation
4. Related Work
5. Conclusions
6. Limitations and References

A. Formalization of the SylloBio-NLI Resource Generation Process

B. Formalization of Tasks 1 and 2

C. Dictionary of gene and pathway membership

D. Domain-specific pipeline for creating NL instances and E Accessing LLMs

F. Experimental Details

G. Evaluation Metrics

H. Prompting LLMs - Zero-shot prompts

I. Prompting LLMs - Few-shot prompts

J. Results: Misaligned Instruction-Response

K. Results: Ambiguous Impact of Distractors on Reasoning

L. Results: Models Prioritize Contextual Knowledge Over Background Knowledge

M Supplementary Figures and N Supplementary Tables

B Formalization of Tasks 1 and 2

B.1 Task 1: Textual Inference Task

Given: A set of premises P = {P1, P2, . . . , Pn} and a conclusion C. Determine whether the logical entailment holds: P |= C.

such as (i) The premises and conclusion are instantiated from formal syllogistic schemes and (ii) arguments may be valid or invalid. The assessment is purely on logical validity, independent of the factual correctness of the premises or conclusion.

Input:

Premises: A set P = {P1, P2, . . . , Pn} of premises, where each Pi is a natural language sentence derived from syllogistic schemes.

Conclusion: A natural language sentence C, representing the conclusion.

Output:

Logical Validity Indicator: A binary output Output ∈ {True, False} such that:

Here, P |= C denotes that C is a logical consequence of P.

B.2 Task 2: Premise Selection Task

Objective: Identify the minimal subset of premises P ′ ⊆ P that are necessary and sufficient to logically derive the conclusion C.

• Given:

– A set of premises P = {P1, P2, . . . , Pn}.

– A conclusion C.

• Task Definition: Find the minimal subset P

Input:

Premises: A set P = {P1, P2, . . . , Pn} containing both relevant and irrelevant (distractor) premises.

Conclusion: A natural language sentence C.

Output:

• Selected Premises: A subset P ′ ⊆ P such that:

P ′ |= C,

and P ′ is minimal and necessary.