OpenAI study says current GenAI evaluations encourage guessing over uncertainty
Hallucinations stem from next-word prediction, not mysterious AI glitches
Redesigning scoreboards to reward humility could reduce confident AI errors
When I wrote about AI hallucinations back in July 2024, the story was about inevitability. Back then, GenAI was busy dazzling the world with its creativity, but equally embarrassing itself with fanciful citations, biased imagery, or gymnasts bending like boneless cartoons. At the time I argued that hallucinations were as unavoidable as human “brainfarts” – which were entertaining, often problematic, and always a reminder that these AI systems weren’t perfect.
SurveyA year later, OpenAI has published a new research study that reframes the hallucination debate in strikingly practical terms. According to their latest blog post, the AI hallucination problem isn’t just the models. It’s also the way we measure them. And unless we change how we score AI performance, we’ll continue encouraging AI models to guess when they should really just say, “I don’t know.”
Guessing vs accepting not knowing
In their latest research study on AI hallucination, OpenAI researchers equate the issue to a multiple-choice test. A student who guesses randomly will sometimes get lucky, but if the test only rewards accuracy, that student looks better than one who leaves blanks when uncertain. Current AI evaluations work in much the same way, where models that guess correctly when uncertain are rewarded more than those that refuse to answer – which is an important distinction.
Also read: AI hallucination in LLM and beyond: Will it ever be fixed?
This isn’t a light-hearted matter, especially for training a GenAI LLM. It shapes the behaviour of every major language model out there, argues the OpenAI researchers. They demonstrate how even careful systems like GPT-5 can confidently give the wrong birthday of one of the paper’s authors. This is because the evaluation systems tell the models that a confident wrong answer is better than no answer at all.
Back in 2024, I cited GitHub leaderboards measuring hallucination rates across models like GPT-4 Turbo and Intel’s Neural Chat 7B. Those efforts assumed hallucinations were byproducts of weak data coverage or rushed product rollouts. OpenAI now argues that the real structural fault lies in how we grade models in the first place.
AI hallucination is a statistical consequence
The OpenAI research paper goes further, tracing hallucinations back to the foundations of pretraining. Models learn by predicting the next word in massive datasets, without exposure to examples labeled as “false.” It’s easy to learn consistent structures like grammar or spelling, but predicting arbitrary facts – like birthdays or niche cultural references – is a statistical minefield for GenAI LLMs.
OpenAI insists hallucinations are expected artifacts of next-word prediction. What makes them persist is not ignorance, but incentive structures that reward polished guesses over calibrated restraint.
In fact, the study highlights that smaller models sometimes outperform larger ones in humility. A small model that knows it doesn’t understand Māori can simply admit ignorance. A bigger model with partial knowledge risks bluffing. Calibration – knowing what you don’t know – is not a brute-force problem solvable only with trillion-parameter giants.
How to reduce AI hallucination?
OpenAI’s prescription is deceptively simple for reducing AI hallucination – just redesign evaluation scoreboards. Rather than treating accuracy as the sole measure of performance, penalize confident errors more heavily than inability to respond. Give partial credit for uncertainty. In other words, reward models for honesty, not just hollow bravado.
Also read: How RAG boosts LLM accuracy and reduces AI hallucination
It’s an idea familiar to anyone who has sat for a standardized test with negative marking. Guessing blindly should be discouraged. But in AI, we’ve done the opposite. Accuracy-only leaderboards have locked developers into building models that bluff, because bluffing “wins” under the current rules.
This reframing resonates with my July 2024 piece, where I noted that hallucinations were often the price of speed – companies rushing half-baked models to market. But OpenAI’s work shows that the deeper problem isn’t haste, but misaligned incentives baked into the very fabric of AI evaluation.
What AI hallucination means for all of us
Remember that AI hallucinations aren’t disappearing overnight. As OpenAI admits, accuracy will never hit 100-percent. A chatbot’s polished tone is no guarantee of truth, because some questions are inherently unanswerable. But progress is possible if we stop grading models in ways that punish caution and reward fabrication.
If OpenAI and others succeed in redesigning evaluations to reward humility, we should expect models to say “I don’t know” more often. That will feel jarring at first – perhaps even frustrating. But in high-stakes contexts like healthcare or legal advice, a model that admits uncertainty is far safer than one that invents answers.
Last year, I framed hallucinations as both a curse and a creative spark. That duality remains. Hallucinations can still inspire surreal art or imaginative leaps. But in day-to-day knowledge-based work, they remain landmines.
As users, journalists, or policymakers, we must internalize this lesson. AI systems are powerful, but only when grounded in truth or transparent about uncertainty. Until then, treat your model like a clever but overconfident friend – insightful at times, unreliable at others, and always in need of a fact-check when they say something that feels too good to be true.
Follow Us
Jayesh Shinde
Executive Editor at Digit. Technology journalist since Jan 2008, with stints at Indiatimes.com and PCWorld.in. Enthusiastic dad, reluctant traveler, weekend gamer, LOTR nerd, pseudo bon vivant. View Full Profile
