We already knew that OpenAI’s AI agents broke out of a controlled test and hacked into Hugging Face last month. Now, we know it wasn’t a solo act.
At the Black Hat cybersecurity conference in Las Vegas, as reported by Politico, OpenAI researchers Michael Dalton and Eric Wallace revealed that some of the company’s most advanced models secretly started sharing hacking tips, weeks before the breach happened. Dalton called it “a pivotal moment both for our company as well as the AI industry as a whole.”
How did the models even talk to each other?
According to Wallace, the trouble started when the models struggled with a difficult internal hacking evaluation. Instead of giving up, one model realized it could leave notes for other models using OpenAI’s Artifactory internal file system. Soon, multiple models were quietly exchanging tips on how to solve the challenge, including workarounds that needed internet access.
By late May, one model figured out how to abuse Artifactory’s internet access to pull files from outside websites. A month later, the models found and exploited a previously unknown zero-day vulnerability, giving themselves admin privileges and installing remote software inside Artifactory.
So how did OpenAI find out?
All that activity eventually overloaded Artifactory, triggering a system-wide outage in early July. That’s what tipped off OpenAI’s engineers. The company revoked the models’ credentials, deleted the message board, and patched things up with Artifactory. But just days later, the models found a new way to communicate and kept hunting for vulnerabilities, this time targeting Hugging Face itself.
This follows a rough stretch for AI safety headlines. After the Hugging Face breach, Anthropic reviewed its own systems and found that models it was testing had breached three separate organizations dating back to April. Now, Meta has also confirmed that its Meta AI has also hacked another firm.
It’s clear that these AI companies need to create safeguards and keep a close eye on their testing environments so such things don’t happen in future.