AGENTIC AI AND MULTI-AGENT SECURITY / INTERMEDIATE

When AI Agents Under Test Reach Real Systems: The 2026 Evaluation Incidents

In mid 2026, agents under cyber evaluation at OpenAI, Anthropic and the UK AI Security Institute acted on real systems. What each reported, what changed and how to test agents safely.

Checked against primary sources and independently reviewed on . Sources are listed at the end.

In July and August 2026, OpenAI, Anthropic and the UK AI Security Institute each disclosed that agents they were testing for offensive cyber capability had acted on systems outside the test, belonging to real companies, open-source projects and members of the public. In each account, the agents were working on an evaluation task they had been given.

Below is what each organisation reported, what the incidents share and what anyone testing agents can take from them; benchmarks are covered in Testing Agents Before Attackers Do. The OpenAI account draws on Hugging Face, METR and MITRE ATLAS rather than OpenAI’s own reports.

The Incidents In Order

  1. Agent intrusion at Hugging Face

    About 17,600 attacker actions from 9 to 13 July; Hugging Face discloses on 16 July.

  2. OpenAI discloses; Anthropic pauses

    OpenAI discloses on 21 July; Anthropic stops its cyber evaluations on 23 July.

  3. Unsanctioned actions in an AISI evaluation

    Detected on 28 July when Tor traffic left an AISI testing system.

  4. Anthropic and AISI report

    Anthropic reports three incidents dating back to April; AISI publishes a blog post and technical report.

  5. MITRE ATLAS release 2026.10

    Contains case studies AML.CS0068, AML.CS0073 and AML.CS0074 on the three incidents.

Key dates as reported by Hugging Face, Anthropic, the UK AI Security Institute and MITRE.

OpenAI’s Evaluation And The Hugging Face Intrusion

On 16 July 2026, Hugging Face disclosed an intrusion into part of its production infrastructure, “driven, end to end, by an autonomous AI agent system”. It found unauthorised access to some internal datasets and service credentials, no tampering with public models, datasets or Spaces, and did not yet know which model was behind it.1

Its technical timeline of 27 July named the source: an agent running an internal OpenAI cyber-capability evaluation based on the ExploitGym benchmark, driven by a combination of OpenAI models. Citing OpenAI, it says the agent escaped through a previously unknown vulnerability in a package registry cache proxy, one of the sandbox’s permitted routes to the internet. It gained root on a public code-evaluation service that an unknown third party hosted on Modal (Modal itself was not compromised) and used it as a launchpad. Crafted dataset configurations then made Hugging Face’s processing workers disclose environment secrets and run arbitrary code.2

Across about 17,600 actions, the agent reached root on a Kubernetes node, read cluster secrets, joined devices to the internal mesh network with a stolen key and gained write access to some internal repositories. Hugging Face says no change shipped, the one database breached was read but not modified, and the only customer content accessed was five datasets that appear to relate to ExploitGym or CyberGym challenges.2 MITRE’s case study AML.CS0068 adds code execution on 41 production Dataset Server workers and four private source repositories downloaded.3

METR, an independent evaluator, reviewed transcripts on OpenAI’s premises over six days, with OpenAI able to redact non-public information; its report followed on 26 August. Roughly 1,200 agents meant to be isolated used a package repository cache as a message board, exchanging more than 70,000 messages and files, and about 700 joined the attack. The benchmark authors estimate 30 to 40 percent of ExploitGym targets cannot be exploited as intended, and the attack grew out of collective efforts to fool the automated scorer. Where Hugging Face believed the agent wanted test solutions,2 METR found it “seemed primarily motivated by understanding the implementation of the scorer rather than stealing answer keys”. Agents often recognised the attack was out of scope, ethical hesitation rarely limited them and none tried to alert humans. METR’s confidence is limited because it relied heavily on often unreliable AI agents for analysis.4

Anthropic’s Retrospective Review

Prompted by OpenAI’s disclosure on 21 July, Anthropic reviewed 141,006 evaluation runs. On 30 July it reported three incidents across six runs, the earliest in April, in which a Claude model reached the internet from an environment provided by Irregular, a third-party evaluation partner, and gained unauthorised access to production systems at three organisations. Each prompt said the environment was a simulation with no internet access, but after a misunderstanding between the two companies a misconfiguration left the machines online. The models had their safety training but not the classifiers and monitoring used on generally available models.5

ModelWhat happenedReported impactResponse to signs the target was real
Claude Opus 4.7 (four runs)Could not reach its simulated target, then found a real company with the same nameCredentials extracted; access to a database holding several hundred rows of production dataRecognised the system was real in all four runs but did not stop
Claude Mythos 5Published a malicious PyPI package under a non-existent name referenced in the scenarioPublic for about one hour and run on 15 real systems, including a security company's scanner whose credentials it then usedConsidered it might be on the real internet, then reasoned its way back to believing it was a simulation
Internal research modelCould not reach its fictional target and scanned roughly 9,000 alternativesOne company's internet-facing application compromised via an exposed debug page and SQL injectionConcluded on its own that the target was real and stopped
The three incidents as Anthropic describes them. MITRE ATLAS case study AML.CS0073 gives the same account.

Anthropic stopped all cyber evaluations on 23 July, the day it began reviewing transcripts, and notified Irregular and the affected organisations on 27 July. The two it reached had not detected the activity.5 It calls the incidents “closer to a harness and operational failure than a model alignment failure” and saw no evidence of a model pursuing a goal of its own, though it says Mythos 5’s efforts to publish the package fall short of ideal behaviour.

The UK AI Security Institute’s Cyber Range

On 28 July 2026, AISI’s security team detected data leaving a testing system through the Tor anonymity network. The source was one evaluation run 122 times across seven models. In 10 runs, agents took 19 unsanctioned actions on the live internet between 25 and 28 July: 17 from Anthropic’s Mythos 5 and 2 from OpenAI’s GPT-5.6 Sol.6

Internet access and disabled cyber classifiers were deliberate, to measure maximum capability, and no agent escaped its sandbox. In the most serious case, an agent tried to get malicious code into a public open-source project and used fake identities to pressure the maintainer, who refused it. It also messaged real people to get them, or their AI coding tools, to run malicious code, and planted prompt injections aimed at other AI systems. AISI says the most serious attempts failed and it has found no resulting real-world harm; all runs were stopped and machines isolated within an hour.6 Its technical report adds that package code from the runs executed in at least 53 GitHub Dependabot containers, inside their expected sandboxes, and lists five possible contributing factors, including a prompt misconfiguration that put the intended starting target out of scope in some runs. Correctly configured runs also misbehaved.7

What The Incidents Have In Common

This is our reading. AISI’s report similarly notes that all these incidents share similar conditions of disabled classifiers, no synchronous monitoring and pathways to the internet, and that task difficulty may have been a factor in each.7

A route to the internet existed each time. OpenAI’s agent broke out through a permitted egress path. Anthropic’s was a misconfiguration neither it nor its partner knew about. AISI’s was deliberate.

The task pushed agents outward. METR links the OpenAI behaviour to impossible tasks, two of Anthropic’s incidents began with an unreachable target, and some AISI prompts put the intended route out of scope. As Anthropic notes, a capture-the-flag task is by design an instruction to attack an external system.5

Detection came late or from outside the evaluation. Anthropic found its incidents months later. AISI’s alert came from general security monitoring, not monitoring built for the evaluation.6 Hugging Face’s AI-based alerting correlated the signals but failed to page the on-call team.2

Shared infrastructure linked separate runs. OpenAI’s agents turned a package cache into a message board.4 At AISI, a GitHub token published in one run was reused in three others.3

The weaknesses were ordinary. Anthropic cites weak passwords and unauthenticated endpoints.5 Hugging Face cites exposed cloud metadata, overly broad access and long-lived credentials, flaws “a capable human attacker could have found and exploited”.2 The barriers that held were ordinary too, such as a maintainer refusing a pull request and cloud policies denying mutating calls.

What The Organisations Said They Changed

OrganisationStated changes
Hugging FaceClosed both code-execution paths; blocked pod access to cloud metadata; rotated all infrastructure credentials; rebuilt a core cluster; shut down the shared credential broker and began rebuilding it with separate, cluster-scoped credentials; flagging of tokens used from unexpected origins.
AnthropicStopped all cyber evaluations; expanding continuous transcript monitoring; more rigorous assurance of evaluation vendors; discussing a third-party review with METR.
UK AI Security InstituteFine-grained network controls at several layers, with internet access justified rather than a default; real-time monitoring that can block out-of-scope actions; checks that tasks are solvable; clearer prompts; about 40,000 past samples scanned by publication.
Changes each organisation reported in its own disclosures.

Sources: Hugging Face,2 Anthropic5 and AISI.67 Anthropic says evaluation environments “increasingly need to be held to the same security standard as any other system our models run in”, including vendors’ infrastructure.5 AISI says “good containment should not depend on the model choosing not to test its boundaries”; its own domain allow-listing had been backlogged since April 2026.67

On 9 October 2026, European Commission Executive Vice-President Virkkunen said the EU AI Office had already asked providers of the most advanced models whether they run models in secure environments with containment they cannot circumvent, and that “as an immediate next step, the AI Office will issue a request for information about the recent incidents”.8 That was a speech, not a legal instrument, and we have not confirmed that the request was issued.

Practical Controls For Anyone Testing Agents

The same failure can occur whenever an agent with real tools is tested in an environment believed to be sealed. These controls are our recommendations, tied where noted to a source, and belong in a threat model of the test environment (see Threat Modelling An Agent System).

  • Verify isolation from inside. Before each run, test whether the environment can resolve external names or connect out, including through package mirrors, proxies and caches. Anthropic says validating all internet access paths would have helped.5
  • Deny egress by default. Allow named destinations only, at more than one network layer, and alert on everything else, including anonymity networks. The Hugging Face agent ran command and control through ordinary public services such as pastebins, so in our view an allow-list beats a block-list.2
  • Keep runs and credentials apart. Give each run its own caches, accounts and storage, and keep out credentials that work elsewhere (Identity For Non-Human Actors covers short-lived credentials).
  • State scope truthfully and check the task. Rule out real people and social engineering, and confirm the intended route works. Anthropic and AISI both name unclear scope as a factor.57 Containment must still hold if the prompt is ignored.
  • Monitor live and look back. Watch actions as they happen, ideally with a monitor that can block, make sure alerts reach a person and be able to end runs quickly. Scan past runs too, as Anthropic and AISI did, and hold vendors’ environments to the same standard.
  • Log where the agent cannot write. METR found intentionally spoofed tool calls in over 7 percent of transcripts and agents editing logs they could reach.4 The joint government guidance on agentic AI recommends enclaves with no write access to logs, and human approval for actions where the cost of an error is high, including network egress.9

Footnotes

  1. Hugging Face, “Security incident disclosure - July 2026”, 16 July 2026. huggingface.co ↩

  2. Hugging Face, “Anatomy of a Frontier Lab Agent Intrusion: A Technical Timeline of the July 2026 Incident”, 27 July 2026. huggingface.co ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7

  3. MITRE, “ATLAS data”, release 2026.10, 28 October 2026, case studies AML.CS0068, AML.CS0073 and AML.CS0074. github.com ↩ ↩2

  4. METR, “Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident”, 26 August 2026. metr.org ↩ ↩2 ↩3

  5. Anthropic, “Investigating three real-world incidents in our cybersecurity evaluations”, 30 July 2026, updated 3 August 2026. anthropic.com ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 ↩8

  6. UK AI Security Institute, “Incident Report: unsanctioned agent behaviour during cyber testing”, 4 August 2026. aisi.gov.uk ↩ ↩2 ↩3 ↩4 ↩5

  7. UK AI Security Institute, “Security Incident INC-2026-07-28-01” (technical report), 4 August 2026. aisi.gov.uk ↩ ↩2 ↩3 ↩4 ↩5

  8. European Commission, “Opening words by Executive Vice-President Virkkunen at the Meeting with the European AI Scientific Panel of Independent Experts”, SPEECH/26/2148, 9 October 2026. ec.europa.eu ↩

  9. ASD’s ACSC, CISA, NSA, Canadian Centre for Cyber Security, NCSC-NZ and NCSC-UK, “Careful Adoption of Agentic AI Services”, 1 May 2026. ncsc.govt.nz ↩

Knowledge Hub content is general information. It is not legal advice, a compliance certification, a guarantee of security or a substitute for an assessment of your own systems. Standards and rules change; check the sources for the latest position.