Table of Links Abstract and 1. Introduction Abstract and 1. Introduction Background


Method


Experiments
4.1 Multi-hop Reasoning Performance
4.2 Reasoning with Distractors
4.3 Generalization to Real-World knowledge
4.4 Run-time Analysis
4.5 Memorizing Knowledge


Related Work


Conclusion, Acknowledgements, and References Background Background Background Method Method Method Experiments
4.1 Multi-hop Reasoning Performance
4.2 Reasoning with Distractors
4.3 Generalization to Real-World knowledge
4.4 Run-time Analysis
4.5 Memorizing Knowledge Experiments Experiments 4.1 Multi-hop Reasoning Performance 4.1 Multi-hop Reasoning Performance 4.2 Reasoning with Distractors 4.2 Reasoning with Distractors 4.3 Generalization to Real-World knowledge 4.3 Generalization to Real-World knowledge 4.4 Run-time Analysis 4.4 Run-time Analysis 4.5 Memorizing Knowledge 4.5 Memorizing Knowledge Related Work Related Work Related Work Conclusion, Acknowledgements, and References Conclusion, Acknowledgements, and References Conclusion, Acknowledgements, and References A. Dataset Dataset B. In-context Reasoning with Distractors B. In-context Reasoning with Distractors C. Implementation Details C. Implementation Details D. Adaptive Learning Rate D. Adaptive Learning Rate E. Experiments with Large Language Models E. Experiments with Large Language Models 2 Background Notation We use f : X × θ → Y to refer to parameterised functions in which X is the set of possible inputs and θ are their possible weights (parameters). We use fθ : x 7→ f(x, θ) to easily refer to any f with a given set of parameters θ. We describe reasoning problems using tuples (K, x, y∗ , Y ) such that y ∈ Y is the correct answer for the question x given facts K, and use D to refer to sets of such problems. When it is clear from context, we drop Y and use only (K, x, y∗ ). Language Modeling and Memorization In the causal language modeling (CLM) objective, a parameterized model fθ is trained to estimate the conditional probabilities of each token in a sequence given its predecessors: p(xt|x) Specifically, we train fθ to approximate p using the CLM loss: This training objective allows language models to memorize individual training examples [10, 11], and we will exploit this ability to memorize and draw on contextual knowledge in our work. Authors:
(1) Zeming Chen, EPFL (zeming.chen@epfl.ch);
(2) Gail Weiss, EPFL (antoine.bosselut@epfl.ch);
(3) Eric Mitchell, Stanford University (eric.mitchell@cs.stanford.edu)';
(4) Asli Celikyilmaz, Meta AI Research (aslic@meta.com);
(5) Antoine Bosselut, EPFL (antoine.bosselut@epfl.ch). Authors: Authors: (1) Zeming Chen, EPFL (zeming.chen@epfl.ch); (2) Gail Weiss, EPFL (antoine.bosselut@epfl.ch); (3) Eric Mitchell, Stanford University (eric.mitchell@cs.stanford.edu)'; (4) Asli Celikyilmaz, Meta AI Research (aslic@meta.com); (5) Antoine Bosselut, EPFL (antoine.bosselut@epfl.ch). This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv available on arxiv

Part of HackerNoon's growing list of open-source research papers, promoting free access to academic material.

Exploiting Memorization: Understanding the CLM Objective for Knowledge Encoding in LLMs

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

AI for All: Operationalizing Diversity and Inclusion Requirements for AI Systems

RECKONING Method: Bi-Level Optimization for Dynamic Knowledge Encoding and Robust Reasoning

It Is Okay If You Don't Know What You Like. We Do (feat. Deep Recommendation Algorithms)

10 Machine Learning, Data Science, and Deep Learning Courses for Programmers in 2020

10 Computer Vision Startups on Product Hunt with the Most Upvotes

AI for All: Operationalizing Diversity and Inclusion Requirements for AI Systems

RECKONING Method: Bi-Level Optimization for Dynamic Knowledge Encoding and Robust Reasoning

It Is Okay If You Don't Know What You Like. We Do (feat. Deep Recommendation Algorithms)

10 Machine Learning, Data Science, and Deep Learning Courses for Programmers in 2020

10 Computer Vision Startups on Product Hunt with the Most Upvotes

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps