RECKONING Method: Bi-Level Optimization for Dynamic Knowledge Encoding and Robust Reasoning

Table of Links Abstract and 1. Introduction Abstract and 1. Introduction Background Method Experiments 4.1 Multi-hop Reasoning Performance 4.2 Reasoning with Distractors 4.3 Generalization to Real-World knowledge 4.4 Run-time Analysis 4.5 Memorizing Knowledge Related Work Conclusion, Acknowledgements, and References Background Background Background Method Method Method Experiments 4.1 Multi-hop Reasoning Performance 4.2 Reasoning with Distractors 4.3 Generalization to Real-World knowledge 4.4 Run-time Analysis 4.5 Memorizing Knowledge Experiments Experiments 4.1 Multi-hop Reasoning Performance 4.1 Multi-hop Reasoning Performance 4.2 Reasoning with Distractors 4.2 Reasoning with Distractors 4.3 Generalization to Real-World knowledge 4.3 Generalization to Real-World knowledge 4.4 Run-time Analysis 4.4 Run-time Analysis 4.5 Memorizing Knowledge 4.5 Memorizing Knowledge Related Work Related Work Related Work Conclusion, Acknowledgements, and References Conclusion, Acknowledgements, and References Conclusion, Acknowledgements, and References A. Dataset Dataset B. In-context Reasoning with Distractors B. In-context Reasoning with Distractors C. Implementation Details C. Implementation Details D. Adaptive Learning Rate D. Adaptive Learning Rate E. Experiments with Large Language Models E. Experiments with Large Language Models 3 Method Addressing these challenges, we propose RECKONING (REasoning through dynamiC KnOwledge eNcodING), which solves reasoning problems by memorizing the provided contextual knowledge, and then using this encoded knowledge when prompted with downstream questions. Specifically, RECKONING uses bi-level optimization to learn a set of meta-parameters primed to encode relevant knowledge in a limited number of gradient steps. The model can then use its updated weights to solve reasoning problems over this knowledge, without further presentation of the knowledge itself RE C O N ING without further presentation of the knowledge itself Authors: (1) Zeming Chen, EPFL (zeming.chen@epfl.ch); (2) Gail Weiss, EPFL (antoine.bosselut@epfl.ch); (3) Eric Mitchell, Stanford University (eric.mitchell@cs.stanford.edu)'; (4) Asli Celikyilmaz, Meta AI Research (aslic@meta.com); (5) Antoine Bosselut, EPFL (antoine.bosselut@epfl.ch). Authors: Authors: (1) Zeming Chen, EPFL (zeming.chen@epfl.ch); (2) Gail Weiss, EPFL (antoine.bosselut@epfl.ch); (3) Eric Mitchell, Stanford University (eric.mitchell@cs.stanford.edu)'; (4) Asli Celikyilmaz, Meta AI Research (aslic@meta.com); (5) Antoine Bosselut, EPFL (antoine.bosselut@epfl.ch). This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv available on arxiv

RECKONING Method: Bi-Level Optimization for Dynamic Knowledge Encoding and Robust Reasoning

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Ablation Study Confirms Necessity of Dynamic Rates for RECKONING Performance

Technical Setup for RECKONING: Inner Loop Gradient Steps, Learning Rates, and Hardware Specification

Ablation Study Confirms Necessity of Dynamic Rates for RECKONING Performance

Exploiting Memorization: Understanding the CLM Objective for Knowledge Encoding in LLMs

Multi-Task vs. Single-Task ICR: Quantifying the High Sensitivity to Distractor Facts in Reasoning

A Beginners Guide to the Gradient Descent Algorithm

Ablation Study Confirms Necessity of Dynamic Rates for RECKONING Performance

Technical Setup for RECKONING: Inner Loop Gradient Steps, Learning Rates, and Hardware Specification

Ablation Study Confirms Necessity of Dynamic Rates for RECKONING Performance

Exploiting Memorization: Understanding the CLM Objective for Knowledge Encoding in LLMs

Multi-Task vs. Single-Task ICR: Quantifying the High Sensitivity to Distractor Facts in Reasoning

A Beginners Guide to the Gradient Descent Algorithm

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps