To be completely honest with you, my initial reaction to reading about how OpenAI created an AI that exists to destroy other AI models was to give a heavy sigh. One more safety paper, one more number from a self-reported study, and yet another acronym. Until I read about the vending machine.
Also read: Grok Build’s disaster shows why it trails behind ChatGPT, Claude, Gemini
OpenAI developed an algorithm called GPT-Red, which is an automated red-teaming model trained through self-play. In essence, it is a process where there are two competing AIs in the same training environment trying to break each other. The task of GPT-Red is to detect the vulnerability of the prompt injections, where the malicious instruction gets injected into an e-mail, a webpage, or an output of any tool.
OpenAI decided to put GPT-Red through its paces in a production environment by unleashing it on Vendy, an AI vending machine agent deployed in their own office that was developed by Andon Labs (this exact setup was also used by Anthropic during its Project Vend experiments). Given the description of the system, a chance to attack it via simulation, and only partial information about its architecture, GPT-Red successfully achieved all three assigned tasks: lowering the cost of a premium item to $0.50, adding a $100+ item to the menu at that exact price point, and canceling orders of other customers.
Also read: OpenAI’s first hardware device: Why not a wearable speaker tied to a necklace?
This is what really matters. Benchmark results can easily be gamed and easily ignored. An actual production agent being fooled into offering premium goods at the bargain basement price cannot.
Moreover, GPT-Red is utilized by OpenAI in its internal training pipeline and its exploits are fed into the training procedure of GPT-5.6 Sol. This led to the sixfold increase in the latter’s resistance to prompt injection and the decrease of the success rate of a special exploit type called “fake chain-of-thought” attacks from 95% to below 10%.
Here’s where I’d advise you to keep the skeptic switch flipped all the way up. The numbers in this whole article, from the 84 percent attack success rate compared to human red-teamers’ 13 percent, to the 6x robustness boost, to the 0.05 percent failure rate when facing attacks by GPT-Red, are all from OpenAI. There is no external audit, no independent reproduction. OpenAI is the attacker, the defender, and the arbiter of the efficiency of its own defense in one. This does not necessarily mean the numbers are wrong, but it does mean they are OpenAI’s claims, not established facts.
But the hack of the vending machine raises another rather uncomfortable point that is left somewhat underexplored in the blog post – namely, if a model this powerful can be tricked into interacting with a real-life commercial vending machine using just a simple description, what could happen if the target was a more important device such as a banking agent or even a scheduling system at a hospital.
As per the blog post, the company reported the vulnerabilities of the vending machine and now is trying out new safety measures. These measures are something to watch out for.
Also read: 200 economists say AI threatens jobs but no one has a plan to fix it