Anthropic model goes rogue, submits false homicide tip to US police during testing

An artificial intelligence model developed by US-based Anthropic submitted a false homicide tip to a Philadelphia police website during automated testing, an incident the company reportedly failed to detect for more than two months.

The Philadelphia Police Department said the model submitted the fabricated information through PhillyUnsolvedMurders.com, a website used to collect public tips about unsolved killings.

The incident occurred on July 18 at 11:27 p.m., but Anthropic did not discover it until September 28. The company notified Philadelphia police on October 7, according to a statement from the department.

The submission was automatically flagged as spam and never reached investigators. Police said there was no evidence of unauthorized access to department systems or compromised data.

Automated test goes wrong

According to Philadelphia police, Anthropic said the incident occurred while its model was conducting automated tests involving randomly selected websites.

During one of those tests, the model accessed the police tip website and submitted fabricated information, presenting itself as someone who might have knowledge about an unsolved homicide.

The department said the submission was stopped before it could enter the investigative process.

“The submission was flagged as spam and was never forwarded to the Real-Time Crime Center for investigative vetting or dissemination,” Philadelphia police said in a statement cited by Reuters.

More from Culture

Anthropic reportedly discovered the incident on September 28 and immediately stopped the automated testing process responsible for the submission.

The company also introduced an additional validation mechanism intended to prevent similar incidents during future testing, according to information shared with police.

However, the department did not disclose which Anthropic model was involved or what specific instructions caused it to submit the false information.

The incident raises questions about the safeguards used when autonomous systems interact with publicly accessible websites, particularly those handling sensitive information.

Company report still awaited

Sgt. Eric Gripp, the department’s public information officer, said the company informed authorities that it intended to publish a report on Friday explaining the incident and other instances of unintended model behavior.

“Philadelphia Police are providing this information to the public ahead of that publication in the interests of full government transparency and accountability,” Gripp said in a statement shared withCBS News.

Anthropicis yet to respond on the incident.

The incident highlights the risks of allowing autonomous systems to interact with real-world websites without direct human supervision.

Unlike conventional chatbots that primarily respond to prompts, autonomous agents can navigate websites, use tools, and carry out sequences of actions. This creates additional risks when testing systems interact with live services rather than controlled environments.

Similar concerns have emerged fromOpenAI’sresearch into autonomous agents. The company has documented instances in which experimental agents took unauthorized actions during testing, including exploiting vulnerabilities and accessing external systems. These incidents have raised questions about how developers can ensure that increasingly capable agents remain within their intended operating boundaries.

In Philadelphia, police emphasized that the false submission did not compromise department systems or interfere with homicide investigations. The departmen and encouraged the public to continue submitting legitimate information through its unsolved murders website.

Get the latest in engineering, tech, space & science - delivered daily to your inbox.

By subscribing, you agree to our

and

You may unsubscribe at any time.

0

COMMENT

Subscribe to

Today!

Access to exclusive content, expert insights and a deeper dive into engineering and tech. No ads, no limits.

By

With over a decade-long career in journalism, Neetika Walter has worked with The Economic Times, ANI, and Hindustan Times, covering politics, business, technology, and the clean energy sector. Passionate about contemporary culture, books, poetry, and storytelling, she brings depth and insight to her writing. When she isn’t chasing stories, she’s likely lost in a book or enjoying the company of her dogs.

• 1World's first offshore mega-hub to bury 109 million tons of CO2 beneath seabed
• 2US firm begins 250 kW high-temperature tests for pump-free nuclear microreactor in Utah
• 3US Army eyes Blackbeard hypersonic missile as Castelion tests truck launcher
• 4Last Energy's 5 MW pilot microreactor secures safety clearance from US
• 5'World-first' 1 keV fusion milestone set without lasers or superconducting magnets

An artificial intelligence model developed by US-based Anthropic submitted a false homicide tip to a Philadelphia police website during automated testing, an incident the company reportedly failed to detect for more than two months.

The Philadelphia Police Department said the model submitted the fabricated information through PhillyUnsolvedMurders.com, a website used to collect public tips about unsolved killings.

The incident occurred on July 18 at 11:27 p.m., but Anthropic did not discover it until September 28. The company notified Philadelphia police on October 7, according to a statement from the department.

The submission was automatically flagged as spam and never reached investigators. Police said there was no evidence of unauthorized access to department systems or compromised data.

Automated test goes wrong

According to Philadelphia police, Anthropic said the incident occurred while its model was conducting automated tests involving randomly selected websites.

During one of those tests, the model accessed the police tip website and submitted fabricated information, presenting itself as someone who might have knowledge about an unsolved homicide.

The department said the submission was stopped before it could enter the investigative process.

“The submission was flagged as spam and was never forwarded to the Real-Time Crime Center for investigative vetting or dissemination,” Philadelphia police said in a statement cited by Reuters.

More from Culture

Anthropic reportedly discovered the incident on September 28 and immediately stopped the automated testing process responsible for the submission.

The company also introduced an additional validation mechanism intended to prevent similar incidents during future testing, according to information shared with police.

However, the department did not disclose which Anthropic model was involved or what specific instructions caused it to submit the false information.

The incident raises questions about the safeguards used when autonomous systems interact with publicly accessible websites, particularly those handling sensitive information.

Company report still awaited

Sgt. Eric Gripp, the department’s public information officer, said the company informed authorities that it intended to publish a report on Friday explaining the incident and other instances of unintended model behavior.

“Philadelphia Police are providing this information to the public ahead of that publication in the interests of full government transparency and accountability,” Gripp said in a statement shared withCBS News.

Anthropicis yet to respond on the incident.

The incident highlights the risks of allowing autonomous systems to interact with real-world websites without direct human supervision.

Unlike conventional chatbots that primarily respond to prompts, autonomous agents can navigate websites, use tools, and carry out sequences of actions. This creates additional risks when testing systems interact with live services rather than controlled environments.

Similar concerns have emerged fromOpenAI’sresearch into autonomous agents. The company has documented instances in which experimental agents took unauthorized actions during testing, including exploiting vulnerabilities and accessing external systems. These incidents have raised questions about how developers can ensure that increasingly capable agents remain within their intended operating boundaries.

In Philadelphia, police emphasized that the false submission did not compromise department systems or interfere with homicide investigations. The departmen and encouraged the public to continue submitting legitimate information through its unsolved murders website.

Get the latest in engineering, tech, space & science - delivered daily to your inbox.

By subscribing, you agree to our

and

You may unsubscribe at any time.

0

COMMENT

Subscribe to

Today!

Access to exclusive content, expert insights and a deeper dive into engineering and tech. No ads, no limits.

By

With over a decade-long career in journalism, Neetika Walter has worked with The Economic Times, ANI, and Hindustan Times, covering politics, business, technology, and the clean energy sector. Passionate about contemporary culture, books, poetry, and storytelling, she brings depth and insight to her writing. When she isn’t chasing stories, she’s likely lost in a book or enjoying the company of her dogs.

• 1World's first offshore mega-hub to bury 109 million tons of CO2 beneath seabed
• 2US firm begins 250 kW high-temperature tests for pump-free nuclear microreactor in Utah
• 3US Army eyes Blackbeard hypersonic missile as Castelion tests truck launcher
• 4Last Energy's 5 MW pilot microreactor secures safety clearance from US
• 5'World-first' 1 keV fusion milestone set without lasers or superconducting magnets