OpenAI brings new tools and warnings for ChatGPT to crack down child abuse content
OpenAI has introduced an AI image classifier that can detect potential CSAM, including AI-generated images, for approved organisations.
ChatGPT will display warnings, remove flagged conversations and pause account access for 24 hours over attempts to find abusive material.
Repeated violations could result in longer account restrictions, permanent suspension or reports to authorities.
AI deployment company OpenAI is widely popular for its viral chat model, ChatGPT. In a recent update, the company is now introducing a new tool for detecting child sexual abuse material (CSAM) in chats for select organisations. The tool is aimed at restricting harmful behaviour and directing users to practical resources. The new image classifier detects potential CSAM content, including AI-generated media. According to OpenAI’s blog post, the classifier can scan “visual signs” of abuse and return scores to let reviewers decide which images require more attention.
OpenAI claims its new image classifier could reduce the number of images requiring review by 48 per cent while identifying roughly the same number of abusive images as its existing tool. The company also noted that flagged images were about twice as likely to contain CSAM abuse. While OpenAI already offers restrictions for CSAM content, new product warnings in ChatGPT can disrupt attempts to find abusive content. These warnings will also explain the consequences of such activity.
“The warning makes clear that sexualising children causes harm, including when the requested material is AI-generated,” OpenAI said. After the warning, the company said it will also remove flagged chats and pause activity on the ChatGPT account for 24 hours. OpenAI added that specialist support will be available for stopping such behaviour. Repeated violations of these guidelines may lead to even longer account restrictions or even suspension of the account completely and reporting the matter to authorities.
The AI company is providing access to the classifier to select companies through its API. “We’re offering access to the classifier to vetted organisations through the OpenAI API at no cost, making CSAM detection more accessible to child safety organisations and the broader industry, including those with teams of varying sizes,” OpenAI said.
“These efforts share one goal: preventing harm to children and strengthening the response when abuse occurs,” it added. The ChatGPT maker claims it is the first CSAM classifier with “broad availability” that can also flag whether the CSAM could be AI-generated. The tool was developed with assistance from the National Center for Missing and Exploited Children (NCMEC).