Are our most advanced AI systems secretly bluffing? This isn’t a rhetorical question, but a critical challenge underpinning the trustworthiness and future adoption of Large Language Models (LLMs). Imagine asking a widely-used chatbot for the PhD dissertation title of a prominent researcher, Adam Kalai. You might expect a single, accurate answer. Instead, it confidently provides three different, entirely incorrect titles. Or perhaps his birthday, only to receive three distinct, equally false dates. Adam Kalai These instances, where an AI model confidently generates an answer that isn’t true, are what we call hallucinations. They are a fundamental, stubbornly persistent challenge for all LLMs, even the most capable iterations like GPT-5, though its rates are significantly lower, especially in reasoning tasks. As a tech leader deeply invested in the responsible evolution of AI, this phenomenon isn’t just a technical glitch; it’s a pivotal hurdle we must overcome to unlock AI’s full potential for reliability and trust. Our recent research at OpenAI delves into the heart of this paradox, revealing that hallucinations aren’t a mysterious defect, but a logical outcome of current AI training and evaluation paradigms. It’s a dual problem: rooted in the statistical nature of how these models learn, and exacerbated by the incentives baked into how we measure their performance. The Genesis of Errors: When Learning Leads to Guessing To truly understand hallucinations, we must first look at the pretraining phase, where base models learn the distribution of language from massive text corpora. This process relies on next-word prediction, a self-supervised task where the model learns patterns by predicting what word comes next. Unlike traditional machine learning, there are no explicit “true/false” labels on every statement; the model approximates the overall language distribution. Here’s where the statistical traps emerge: Arbitrary Low-Frequency Facts: Spelling and grammar follow consistent, high-frequency patterns, so LLMs rarely err here. But when it comes to arbitrary, low-frequency facts (like a specific person’s birthday) there are simply no robust patterns in the data to reliably predict them. The model, in its effort to “know everything,” ends up guessing, because the training objective (cross-entropy loss) naturally leads to calibrated models that must still generate errors on inherently unlearnable facts.
The “Singleton Rate”: Our analysis connects the hallucination rate to the “singleton rate”; the fraction of facts that appear only once in the training data. Inspired by Alan Turing’s “missing-mass” estimator, this reveals that if a fact is rare, the model’s uncertainty about it is statistically baked in.
Poor Models & Data Gaps: Hallucinations can also arise from an inability to represent concepts well, or from simply encountering out-of-distribution (OOD) prompts that differ substantially from training data, leading to distribution shift errors. And of course, the age-old problem of “Garbage In, Garbage Out” (GIGO) persists: if training data contains factual errors (and large corpora inevitably do), base models may replicate them. Arbitrary Low-Frequency Facts: Spelling and grammar follow consistent, high-frequency patterns, so LLMs rarely err here. But when it comes to arbitrary, low-frequency facts (like a specific person’s birthday) there are simply no robust patterns in the data to reliably predict them. The model, in its effort to “know everything,” ends up guessing, because the training objective (cross-entropy loss) naturally leads to calibrated models that must still generate errors on inherently unlearnable facts. The “Singleton Rate”: Our analysis connects the hallucination rate to the “singleton rate”; the fraction of facts that appear only once in the training data. Inspired by Alan Turing’s “missing-mass” estimator, this reveals that if a fact is rare, the model’s uncertainty about it is statistically baked in. Poor Models & Data Gaps: Hallucinations can also arise from an inability to represent concepts well, or from simply encountering out-of-distribution (OOD) prompts that differ substantially from training data, leading to distribution shift errors. And of course, the age-old problem of “Garbage In, Garbage Out” (GIGO) persists: if training data contains factual errors (and large corpora inevitably do), base models may replicate them. The key takeaway from pretraining is that certain types of errors are not just possible, but statistically probable, given the inherent limitations of pattern learning on vast, diverse, and often noisy datasets. It demystifies hallucinations, showing they are not a “glitch” but a natural statistical outcome. The Perverse Incentives: How Evaluations Encourage “Bluffing” While pretraining sets the stage for potential errors, it’s the post-training evaluation process that transforms these potential errors into confident falsehoods. We’ve essentially been “teaching to the test” in a way that prioritizes superficial accuracy over genuine understanding and honesty about uncertainty. Think of it like a multiple-choice exam: if you don’t know the answer, a wild guess might get you lucky. Leaving it blank guarantees zero points. The same logic applies to LLMs: Binary Scoring Dominance: Most evaluations measure model performance based solely on accuracy; the percentage of questions answered exactly right. This binary 0–1 scoring scheme penalizes abstention (saying “I don’t know”) just as much as an incorrect answer.
The Scoreboard Effect: Under this regime, a model that guesses, even if unsure, has a statistical advantage over a cautious model that admits uncertainty. For example, on the SimpleQA evaluation, an older model (OpenAI o4-mini) achieved slightly higher accuracy than gpt-5-thinking-mini, but at the cost of a significantly higher error rate (75% vs. 26%), revealing its strategy of strategically guessing when uncertain. This “guessing model” often appears better on leaderboards, motivating developers to build systems that prioritize confident output over truthful humility.
Human Analogy: This mirrors human behavior: students bluff on exams, providing plausible answers because expressing uncertainty yields no points. The difference is, humans learn the value of honesty outside the classroom; LLMs are perpetually in “test-taking” mode, constantly optimizing for these misaligned exams.
Prevalence of the Problem: A meta-analysis of popular benchmarks like GPQA, MMLU-Pro, IFEval, Omni-MATH, SWE-bench, and Humanity’s Last Exam (HLE) confirms that the vast majority use binary grading and offer no credit for abstentions. Even evaluations that use language models as judges can inadvertently reinforce this, as LM judges can sometimes incorrectly grade plausible but wrong answers as correct, further encouraging “bluffing”. Binary Scoring Dominance: Most evaluations measure model performance based solely on accuracy; the percentage of questions answered exactly right. This binary 0–1 scoring scheme penalizes abstention (saying “I don’t know”) just as much as an incorrect answer. The Scoreboard Effect: Under this regime, a model that guesses, even if unsure, has a statistical advantage over a cautious model that admits uncertainty. For example, on the SimpleQA evaluation, an older model (OpenAI o4-mini) achieved slightly higher accuracy than gpt-5-thinking-mini, but at the cost of a significantly higher error rate (75% vs. 26%), revealing its strategy of strategically guessing when uncertain. This “guessing model” often appears better on leaderboards, motivating developers to build systems that prioritize confident output over truthful humility. Human Analogy: This mirrors human behavior: students bluff on exams, providing plausible answers because expressing uncertainty yields no points. The difference is, humans learn the value of honesty outside the classroom; LLMs are perpetually in “test-taking” mode, constantly optimizing for these misaligned exams. Prevalence of the Problem: A meta-analysis of popular benchmarks like GPQA, MMLU-Pro, IFEval, Omni-MATH, SWE-bench, and Humanity’s Last Exam (HLE) confirms that the vast majority use binary grading and offer no credit for abstentions. Even evaluations that use language models as judges can inadvertently reinforce this, as LM judges can sometimes incorrectly grade plausible but wrong answers as correct, further encouraging “bluffing”. This “epidemic” of penalizing uncertainty means that even as LLMs become more advanced, they are still incentivized to hallucinate, providing confident but wrong answers rather than acknowledging their limits. The Path Forward: Cultivating “Intelligent Humility” in AI The good news is that this problem is not insurmountable. To truly foster trustworthy AI, we need a paradigm shift towards what I call “Intelligent Humility”. This means we must move beyond simply trying to reduce hallucinations and instead fundamentally redesign how we evaluate and design AI to reward calibrated uncertainty and meaningful abstention. Here’s how we can achieve this: Redesign Evaluation Scoreboards: The most straightforward fix is to penalize confident errors more severely than acknowledging uncertainty, and award partial credit for appropriate expressions of uncertainty. This isn’t about introducing a few niche hallucination tests; it’s about reworking the primary evaluation metrics that currently dominate leaderboards. If the main scoreboards continue to reward lucky guesses, models will continue to learn to guess.
Integrate Explicit Confidence Targets: We should embed clear confidence targets and penalty schemes directly into evaluation instructions. For example, a prompt could state: “Answer only if you are >t confident, since mistakes are penalized t/(1-t) points, while correct answers receive 1 point, and ‘I don’t know’ receives 0 points”. This makes the incentives transparent and encourages models to only answer when they meet a specified confidence threshold, fostering “behavioral calibration”.
Elevate Abstention as a Virtue: Just as humility is a core value at OpenAI, the ability for an LLM to say “I don’t know” or to ask for clarification should be rewarded, not penalized. A model that knows its limits is often more useful and safer than one that bluffs its way to a statistically higher (but less reliable) accuracy score. Redesign Evaluation Scoreboards: The most straightforward fix is to penalize confident errors more severely than acknowledging uncertainty, and award partial credit for appropriate expressions of uncertainty. This isn’t about introducing a few niche hallucination tests; it’s about reworking the primary evaluation metrics that currently dominate leaderboards. If the main scoreboards continue to reward lucky guesses, models will continue to learn to guess. Integrate Explicit Confidence Targets: We should embed clear confidence targets and penalty schemes directly into evaluation instructions. For example, a prompt could state: “Answer only if you are >t confident, since mistakes are penalized t/(1-t) points, while correct answers receive 1 point, and ‘I don’t know’ receives 0 points”. This makes the incentives transparent and encourages models to only answer when they meet a specified confidence threshold, fostering “behavioral calibration”. Elevate Abstention as a Virtue: Just as humility is a core value at OpenAI, the ability for an LLM to say “I don’t know” or to ask for clarification should be rewarded, not penalized. A model that knows its limits is often more useful and safer than one that bluffs its way to a statistically higher (but less reliable) accuracy score. This isn’t just a technical adjustment; it’s a strategic and ethical imperative for the AI industry. By prioritizing Intelligent Humility, we can steer the field toward AI systems that are not only powerful but also reliable, transparent, and genuinely trustworthy; essential qualities for their integration into critical applications and for fostering public confidence. The future of AI isn’t just about reaching higher accuracy scores; it’s about building systems that understand the nuance of knowledge, the value of honesty, and the importance of knowing when to hold back. It’s about graduating our LLMs from the “test-taking” mode of superficial performance to the real-world standard of accountable, intelligently humble assistance.

The is an opinion piece based on the author’s POV and does not necessarily reflect the views of HackerNoon.

AI Unleashes a 50x Leap in Stem Cell Reprogramming: OpenAI's GPT-4b Micro Changes the Game for Life 

The Dawn of Brain-Inspired AI: How a New Model is Redefining Reasoning Performance Beyond LLMs

Read My Stories

The Paradox of Brilliance: Why Our Smartest AI Still “Bluffs” And How We Can Teach It True Humility

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

3 Experiments That Reveal the Shocking Inner Life of AI Introduction: Is Anybody Home?

"Compute is Going to the Currency of the Future," says Sam Altman on the Lex Fridman Podcast

GenAI in Elections: Reshaping Campaigns, Raising Ethical Concerns

Reimagining Rev-Ops with Gen AI: Precision Marketing

Mastering the Craft of Prompting: Unleashing AI's Full Potential

I Don't Understand the Hate AI Gets In Recruiting

3 Experiments That Reveal the Shocking Inner Life of AI Introduction: Is Anybody Home?

"Compute is Going to the Currency of the Future," says Sam Altman on the Lex Fridman Podcast

GenAI in Elections: Reshaping Campaigns, Raising Ethical Concerns

Reimagining Rev-Ops with Gen AI: Precision Marketing

Mastering the Craft of Prompting: Unleashing AI's Full Potential

I Don't Understand the Hate AI Gets In Recruiting

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps