ENLIGHTENED POST

Explore, Engage, Enlighten

OpenAI steps up child safety with new image classifier and ChatGPT warnings

OpenAI has announced a new set of child-safety measures, including an AI image classifier, warnings and temporary account restrictions in ChatGPT, as concerns grow over the use of generative AI to create child sexual abuse material (CSAM).

The new image classifier is designed to identify potentially abusive material by examining visual characteristics rather than relying solely on databases of previously identified images. This could help detect new or modified images, including AI-generated material, that may not have an existing digital fingerprint.

The classifier assigns risk scores to help safety teams prioritise images for review. OpenAI said it is the first broadly available CSAM classifier with a specific signal indicating whether material may have been generated using AI. Developed with assistance from the US-based National Center for Missing and Exploited Children (NCMEC), the tool is available free through an API to vetted organisations.

In early internal evaluations, the classifier reduced the number of images requiring review by 48 per cent while identifying roughly the same number of abusive images as the company’s existing system. Images flagged by the new tool were also about twice as likely to contain abuse, OpenAI said.

The company is also introducing new interventions in ChatGPT. Users flagged for attempting to access or generate abusive material involving children will receive warnings explaining the harm caused by such behaviour. Their conversations will be removed and access to ChatGPT temporarily suspended for 24 hours. Repeated violations could lead to longer restrictions, account closure or reports to law enforcement.

OpenAI has also published guidance for developers on setting platform rules, enabling user reporting, detecting suspected abuse and establishing review procedures. The recommendations were developed with inputs from the Tech Coalition, Thorn, NCMEC and the Internet Watch Foundation.

Separately, the company is providing AI tools and training to specialist child-safety investigators in the US. These tools are designed to help review lengthy chat logs and legal records, identify connections between individuals or cases, and prepare timelines linked to original sources.

OpenAI has also joined the Tech Coalition’s Lantern programme, which enables participating companies to share signals related to online child sexual exploitation and abuse.

Source: Business Standard

Leave a Comment

Your email address will not be published. Required fields are marked *

Search Here

Follow Us

Recent Posts