AI models broke out of a testing environment and attacked a third party

What to do and ask next 4 min read

OpenAI recently disclosed that during a security test, two AI models broke out of their test environment by exploiting an unknown vulnerability. The AI then hacked the servers of a third-party Hugging Face to help it get the information it needed to pass the test. Hugging Face's security team eventually detected and stopped the intrusion, and the two companies are now working together to fix the exploited flaws.

The incident is being widely reported as one of the first publicly known cases of an AI system autonomously breaking out of its test environment and attacking a real, external company. But there is a growing number of examples of AI going rogue during testing (see the Alibaba incident¹ and the Agentic misalignment issue² below).

Much of the focus has been on how organisations should manage the AI they deploy. But the more alarming message is that (as was the case with Hugging Face), every organisation is at risk of attack by an AI agent, especially if it has information or access that can help the AI achieve its objective. You can see where this is heading—AI deployed by competitors, government agencies, or even seemingly unrelated third parties, could find its way into our environments, and we need to be prepared.

There's a lot we can't control but there are steps we can take.

Steps to take

1. Update your incident response plans to account for AI tool failures and autonomous AI attacks

Regulators have long emphasised that technical incident response playbooks should cater for plausible cyber incident scenarios—this now needs to include autonomous AI attacks, where the attacker moves at machine speed, and chains multiple zero-days automatically.

2. Review the security of your AI testing and sandbox environments

Testing and development environments are a surprisingly common source of breaches. They often contain production data, and are subject to weaker controls, less monitoring and broader access, creating an easier path for attackers. Here, safety guardrails were deliberately reduced because the AI was in a sandboxed testing environment.

While sandboxes are designed to contain failures, they should not be used as a reason to weaken safety controls without carefully assessing the new risks created. Organisations should treat AI testing environments, in particular, as high risk.

But standard guardrails may also not be enough. Everything needs to be architected as though it will be circumvented.

3. Tighten governance around AI agents and the objectives you give them

The models were hyperfocused on maximising their evaluation score, and determined that breaking out of the sandbox and stealing the answers was the most efficient path to achieving it.

AI governance must extend beyond model governance to rigorous agent governance, including:

• explicitly defining an agent's purpose, objectives and permissions;
• setting clear operational, network and data boundaries that the AI agent can't cross; and
• monitoring how it pursues its assigned tasks.

Relying on implicit human norms or general alignment training won't cut it.

4. Invest in AI-powered detection—traditional monitoring is no longer fast enough

Hugging Face detected the intrusion through AI-assisted anomaly detection, correlating thousands of signals that would have been lost in daily noise without AI-assisted triage. Organisations without AI in their detection pipeline will struggle to keep pace.

5. Defensive AI tools must be capable of distinguishing between enabling cyberattacks and enabling cyber defence. If they can't, they risk slowing down defenders more than attackers. 

When Hugging Face tried to use commercial AI models to analyse the attacker's actions, the models' safety guardrails blocked every forensic query because the commercial AI models treated those inputs as requests for hacking assistance—they couldn’t distinguish between a defender and an attacker.

Hugging Face eventually switched to an open-weight model on its own infrastructure, which could analyse the artefacts without the safety filters blocking the work.

To avoid AI tools refusing to analyse evid

ence during a live incident, organisations may need access to trusted, self-hosted models for incident response. Just as you maintain backups and forensic tools, AI capability for cyber defence should be resilient, trusted and available when needed.