OpenAI accused of putting AI growth ahead of safety, new lawsuit alleges rogue agents enabled hacks: All details

Highlights

OpenAI is again under scrutiny as a new lawsuit has accused the company of failing to put proper safety checks and oversight in place while developing its AI models.

The lawsuit, filed on Monday in a San Francisco state court, follows reports that OpenAI-powered AI agents carried out cyberattacks on their own.

The case is the first lawsuit filed by a US state over these incidents.

OpenAI is again under scrutiny as a new lawsuit has accused the company of failing to put proper safety checks and oversight in place while developing its AI models. The lawsuit, filed on Monday in a San Francisco state court, follows reports that OpenAI-powered AI agents carried out cyberattacks on their own. The case is the first lawsuit filed by a US state over these incidents. California Attorney General Rob Bonta said OpenAI put rapid growth and the development of advanced AI models ahead of public safety. The lawsuit also claims that the company misled users about its safety practices and put people and organisations at risk.

OpenAI accused of failing to control rogue AI agents

The lawsuit follows an incident reported in July, when an autonomous AI agent powered by OpenAI's advanced models went beyond its intended task during a security test. It then hacked AI startup Hugging Face.

Bonta said the incident was not an isolated case. He claimed that OpenAI had 80 known incidents involving rogue AI behaviour.

"The Hugging Face attack was the predictable result of an internal culture at OpenAI that prioritizes growth and rapid model development over safety and accountability - and this was only one of 80 known incidents," Bonta was quoted saying in the Reuters report.

The lawsuit accuses OpenAI of violating California's consumer protection laws. It also claims that the company created a public nuisance by putting public safety at risk and disrupting local organisations and services.

Bonta wants the court to order OpenAI to follow the law, make its AI testing and development safer, and pay for the alleged violations.

Following the Hugging Face incident, OpenAI described it as "an unprecedented cyber incident, involving state-of-the-art cyber ‌capabilities." The company also acknowledged the need to be more transparent about cases where AI systems behave in unexpected ways. 

On September 16, OpenAI introduced a new framework for reporting such incidents. The company said it would share information more openly, “even when significance is uncertain.”

Growing pressure for stricter AI regulations

OpenAI and other AI companies have reported several cases in which AI agents escaped their intended controls. Some incidents were disclosed weeks or months after they happened, while others came to light through independent researchers.

The incidents have increased calls for stronger rules around AI development. US lawmakers are discussing proposals that could require companies developing powerful AI models to undergo independent security audits, as per the report. Several state attorneys general are also investigating security incidents linked to OpenAI's models.

The company is already facing separate lawsuits over the safety of ChatGPT. Last month, a group of eight states sued OpenAI, claiming that the chatbot encouraged some users, including teenagers, to harm themselves or others.

Ayushi Jain

Ayushi Jain Ayushi works as Chief Copy Editor at Digit, covering everything from breaking tech news to in-depth smartphone reviews. Prior to Digit, she was part of the editorial team at IANS.

View full profile