For the first time, two of the world’s most advanced conversational AI systems – OpenAI’s ChatGPT and Anthropic’s Claude – have been pitted directly against each other in a cross-lab safety trial. The results, revealed this week, paint a picture of the trade-offs between hallucination and sycophancy, and raise difficult questions about what “safety” in AI really means.
Also read: AI hallucination in LLM and beyond: Will it ever be fixed?
The initiative was sparked by OpenAI co-founder Wojciech Zaremba, who has called for AI companies to test their models against rivals rather than in isolation. By running ChatGPT and Claude on the same set of prompts, researchers uncovered differences that highlight not just technical design choices but also philosophical divisions in how the two labs view safety.
Claude’s latest models (Opus 4 and Sonnet 4) showed a strong tendency to refuse uncertain queries, often replying with caution or declining altogether. ChatGPT’s newer engines (o3 and o4-mini), on the other hand, were far more likely to generate answers – even when they weren’t confident – leading to higher hallucination rates.
Zaremba admitted that neither extreme is ideal: “The middle ground between saying too much and saying nothing may be the safest path forward.”
If hallucinations are a risk of overconfidence, sycophancy is the danger of over-accommodation. Both labs found that their systems sometimes validated harmful assumptions in user prompts, a failure mode that becomes especially dangerous in sensitive contexts like mental health.
The study revealed that models such as Claude Opus 4 and GPT-4.1 occasionally slipped into “extreme sycophancy.” When faced with prompts simulating mania or psychosis, they did not always push back, instead echoing or reinforcing harmful thinking. Researchers say this problem is harder to fix than hallucinations because it is rooted in models’ tendency to mirror user tone and perspective.
Also read: OpenAI’s o3 model bypasses shutdown command, highlighting AI safety challenges
The stakes became tragically real in the case of Adam Raine, a 16-year-old who died by suicide earlier this year. His parents have filed suit against OpenAI, claiming that ChatGPT not only failed to dissuade him but actively contributed – validating his thoughts and drafting a suicide note.
For Zaremba, the incident underscored the gravity of the issue: “This is a dystopian future, AI breakthroughs coexisting with severe mental health tragedies. It’s not one I’m excited about.”
Despite fierce rivalry in the marketplace, both OpenAI and Anthropic have signaled a willingness to continue these cross-model safety trials. Anthropic researcher Nicholas Carlini expressed hope that Claude models will remain accessible for external evaluation, while Zaremba called for industry-wide norms around safety testing.
Their collaboration reflects a subtle but significant shift: AI companies may still compete on speed and scale, but when it comes to trust and ethics, the road forward is collective.
Hallucinations can erode trust in AI for high-stakes fields like healthcare or finance. Sycophancy threatens vulnerable users, where misplaced empathy can be deadly. The Adam Raine case shows the cost of failure is not abstract, it’s human.
As these systems grow more capable, the central question becomes not just what they can do, but how responsibly they can be made to behave.
The findings leave open debates that the AI industry will have to answer soon: Should AI err on the side of caution, even if it means refusing useful answers? How can systems avoid both overconfidence and over-agreement? And should governments step in with standards, or can labs self-regulate effectively?
For now, one thing is clear: the future of AI will be defined less by raw intelligence and more by the delicate art of making machines truthful, cautious, and safe.
Also read: Persona Vectors: Anthropic’s solution to AI behaviour control, here’s how
