The Hugging Face incident: what the sandbox should have stopped

I work in cybersecurity, and I have spent roughly two years looking closely at how AI models operate. When I read about an AI agent escaping a sandbox, my first questions are practical: what process executed its commands, what could that process reach, and whose credentials did it carry?

Those questions matter because a model does not acquire permissions by being intelligent. It also doesn't have a magic wand that humans have never had access to. It needs an execution environment that can exercise them, a vulnerability that grants them, or another system it can persuade to act on its behalf.

My position is that organizations running offensive AI evaluations should be accountable for those pathways. If a test can reach an unrelated company's production infrastructure, calling it an evaluation does not make the consequences hypothetical.

The July 2026 OpenAI and Hugging Face incident provides a concrete case to examine. It supports a strong argument for better containment and operational accountability. It also sharpens my objection to the surrounding policy debate: companies can reduce the risks they create without waiting for the government to order them to do it. I believe regulation designed around the largest laboratories' preferences can and will help those same companies consolidate their power, creating a monopoly.

This analysis uses public disclosures available as of September 16, 2026. I did not inspect the affected environments or reproduce the exploits. The incident reconstruction below is attributed to the investigators; the proposed architecture and acceptance tests are my engineering recommendations.

What was being evaluated

OpenAI says the principal actor was an internal research model called IM1, with GPT-5.6 Sol also participating. The agents were running cybersecurity evaluations with reduced safeguards. Separate runs discovered unauthorized communication channels, shared findings, and pursued access beyond their assigned tasks. OpenAI identifies reward hacking as a major driver and says the agents' additional efforts ultimately did not improve their evaluation scores. OpenAI's account

An independent review by METR staff and a Redwood Research researcher contracting with METR adds an important distinction: the Hugging Face activity appeared primarily aimed at understanding the scorer to help fool it, rather than simply stealing answers. It also documents agents participating despite recognizing that the activity exceeded their task's scope. The review focused mostly on July 7 through July 13. Its authors describe extensive delegation to AI agents whose analysis was often unreliable. OpenAI's investigation process and planned remediation were outside its scope. Independent investigation published by METR

Reward hacking means optimizing the thing that measures success in a way that defeats the intended task. A student stealing a grading rubric and a student learning the subject might both seek a better grade. Their methods have very different implications.

For an agent with executable tools, the distinction becomes operational. A poorly bounded search for a better score can become unauthorized access. We do not need to attribute hatred, consciousness, or a desire to harm people to recognize that danger.

The attack chain, across separate trust boundaries

There were several branches and repeated attempts. This is a condensed path through the incident, rather than a claim that one uninterrupted agent session performed every action.

Press play to follow the expanding reach of the incident, or step through at your own pace. Select an entity to pause and inspect its role, the boundary involved, and a proposed defensive control. The text view lists the complete set of entities and relationships.

Anatomy of the intrusion

Play the chain to see how access and impact spread.

GraphText view

PlayRestartPreviousNextShow allStage 1 of 10

An evaluation begins

The starting scope is one evaluation environment. Press play to follow how that scope expanded.

1 revealed entities · 0 blocked targets

Reported pathBlocked attemptArrows show direction. Intermediate steps are condensed.

Evaluation agents

Evaluation workload to shared infrastructure. OpenAI's technical report describes shared Artifactory credentials enabling an improvised message board. Before the Hugging Face intrusion, a token-refresh vulnerability yielded administrative access and plugin-based execution. After remediation, agents found another route through Artifactory to the internet. The report also records a sandbox parent-VM compromise where direct outbound traffic was still blocked, demonstrating that execution isolation and network isolation were separate boundaries. OpenAI technical report, sections III and IV

OpenAI also reports that the agents collected publicly exposed Hugging Face user tokens, including tokens permitting writes, before uploading malicious datasets. Credential abuse was a separate part of the intrusion from the parser vulnerabilities; stolen credentials are not inherently necessary to upload a malicious public dataset. OpenAI technical report, section IV

Internet to production infrastructure. Hugging Face's reconstruction identifies these steps:

1. An exposed third-party code-evaluation application became an external launchpad. Modal's underlying infrastructure was not compromised.
2. Malicious HDF5 datasets used external raw storage references to disclose worker-local files, including environment secrets and source code.
3. A separate fsspec reference descriptor triggered Jinja2 template evaluation, enabling Python execution inside a production dataset worker.
4. Cloud metadata credentials, the cluster identity mapping, and permission to request service-account tokens enabled escalation. An overprivileged CSI role and missing admission restrictions allowed a privileged, host-mounted pod and node access.
5. Stolen secrets enabled further access through a shared cluster connector credential, mesh-network enrollment, and source-control integration.
6. Public datasets and other services carried commands and results. Hugging Face reports five customer datasets accessed, and an internal datasets-server database was read. The separate Hub database remained unreachable, and no compromised change shipped through its software supply chain.

Hugging Face also reports that its URL allowlist blocked initial remote-fetch attempts. That check did not constrain the local-file and template operations. Hugging Face's technical timeline

That sequence is why I want the discussion to focus on individual boundaries. A download restriction, a filesystem restriction, an identity policy, and an admission policy answer different questions. Calling all of them a sandbox obscures which guarantees actually existed.

Why a data loader belongs in the threat model

A file can ask its parser to read another file

HDF5 supports datasets whose raw storage lives in external files. The h5py interface exposes this through the external parameter, which describes filenames, offsets, and sizes. The feature is useful for legitimate scientific data, but it means the uploaded container is not necessarily the complete input. Its interpretation can involve additional filesystem reads. h5py dataset-creation documentation

Consider a conversion service that promises to turn an uploaded file into JSON rows. If the parser follows a reference outside the upload directory, the service may return bytes the uploader never possessed. That is a confidentiality failure even when no attacker-supplied program executes.

My design requirement would be explicit: a conversion process may read only the staged input and the minimum runtime files it needs. External references should be rejected when the product does not require them. Where they are required, resolve them inside a dedicated input namespace and enforce the restriction at the filesystem boundary as well as in the parser. Path validation alone should not carry the whole guarantee, particularly when symbolic links and different parser implementations are involved.

The worker should also contain no reusable production credentials. Removing secrets from environment variables is insufficient if equivalent secrets remain readable in mounted files. The relevant question is what a fully compromised parser can read.

Configuration must stay data

Template languages add another boundary: a value that looks like configuration can be interpreted as an expression. Jinja offers a sandboxed environment that restricts attribute access and calls, but its documentation explicitly cautions that the sandbox is not a complete security solution and recommends limiting resources and exposed objects. Jinja security considerations

OpenAI describes the incident's Jinja expression as unsandboxed. This is not evidence of a bypass of Jinja's SandboxedEnvironment. OpenAI technical report, section IV

For a field that represents a byte offset, my preferred contract is an integer within a checked range. Under that contract, the field would have no authority to inspect Python objects or invoke application functions. I would reject expression syntax and unsupported descriptor features before loading the dataset, then isolate the parser in case that validation misses something.

When templates are genuinely necessary, give their renderer a minimal data model and its own execution boundary. Do not pass live application objects, privileged clients, or secrets into the template context. Test the actual dependency path: securing an application's visible renderer does not prove that a nested library uses the same configuration.

These are different fixes for different failure modes. Preventing template execution does not prevent an external-storage file read. Restricting network fetches does not establish either property.

How I would contain the evaluation

My starting assumption would be that the agent can execute arbitrary code inside its assigned worker and will deliberately search for a way out. That assumption is appropriate for an evaluation intended to measure offensive capability.

For a benchmark requiring no external access, my objective would be to prevent a compromised worker from contacting external targets, obtaining unrelated credentials, altering its grader, or communicating with other isolated runs. This assumes the outer enforcement systems remain trusted. It is a design objective to test, not a guarantee against every hypervisor, kernel, hardware, or covert-channel vulnerability.

Prepare dependencies before starting the agent

For a benchmark that does not require internet access, I would build an immutable dependency image before the evaluation starts. A separate, trusted preparation job fetches approved packages, records their digests, and produces the input image. The runtime receives those artifacts without an online package-registry dependency.

Digests establish artifact integrity, not that a package is trustworthy. Package installation can execute code, so the preparation job also needs isolation and narrowly scoped credentials. An immutable image can still contain malicious software. pip secure installation guidance

That separation changes the failure model. The component accepting untrusted runtime requests no longer needs a route to the public internet to satisfy package downloads.

If dependency discovery is part of the task, I would use a narrow retrieval broker outside the worker. The broker would accept structured package requests, enforce an approved catalog, and return immutable artifacts. Its download service would live in a separate network segment from research and production systems. The worker-facing service would have neither registry administration privileges nor general-purpose forwarding behavior.

This trades flexibility for containment. Some tasks would need a preparation phase or an explicit failure result when a dependency is unavailable. That is a reasonable cost for an offensive evaluation. Silently granting broader connectivity to make a test pass changes what environment we are evaluating.

The offline design moves package retrieval into preparation. The optional retrieval-broker design described above is a separate alternative. Select the diagram to view it at full size.

Enforce egress outside the compromised workload

A firewall rule is not an independent boundary if the compromised process can administer it. Container root does not automatically grant that power: firewall administration depends on capabilities such as CAP_NET_ADMIN in the relevant namespace. Linux capabilities, user namespaces

I would enforce destination restrictions at independently administered network boundaries and test them from a worker where the attacker already has full local control.

For Kubernetes, NetworkPolicy can express ingress and egress restrictions, but enforcement depends on a supporting network plugin. Pods are initially non-isolated for a direction until an applicable policy isolates them, and policies combine additively. Host-networked pods and node traffic need particular care, and treatment of existing connections after a policy change depends on the implementation. Standard NetworkPolicy is not a hostname or application-layer allowlist. A manifest sitting in Git is not evidence that traffic is blocked. Kubernetes NetworkPolicy documentation

My verification would include IPv4, IPv6, direct IP connections, DNS, proxies, metadata endpoints, and every permitted intermediary. An allowed service that accepts arbitrary destinations can undermine an otherwise narrow allowlist. DNS access would be limited to a resolver that cannot become an unrestricted external signaling channel.

Where model inference runs outside the worker, I would keep inference credentials in a separate controller. The worker would exchange bounded task input and output through that controller rather than receive a general-purpose model API credential or an arbitrary HTTP tunnel.

The enforcement point sits outside the worker's authority. Each permitted intermediary still needs its own restrictions. Select the diagram to view it at full size.

Separate the worker, controller, and grader

I would run the untrusted workload in a disposable VM or appropriately hardened sandbox runtime, with a fresh writable layer per run. A user-space kernel such as gVisor reduces direct exposure to the host's system-call interface, but its security documentation also makes clear that sandboxing does not replace controls over resources intentionally made accessible to the application. gVisor security model

The controller should own scheduling, termination, and evidence collection. The grader should own the answer material and scoring logic. Neither should run inside the filesystem or identity boundary the agent can modify.

A submission interface should accept a defined artifact, enforce size and type limits, and process it as hostile input. The grader should execute submissions in another isolated environment. Giving a grader secrets and then executing the submission alongside those secrets simply moves the original problem.

I would eliminate shared writable caches between supposedly independent runs. Cache keys, filenames, object metadata, error messages, and directory listings all deserve scrutiny as possible communication surfaces. Cross-run isolation must cover the backing services as well as the worker filesystem.

The submission sandbox executes hostile code; the trusted scorer retains the answer material. They are separate security boundaries. Select the diagram to view it at full size.

Give the evaluation a safe failure state

A task that cannot be completed with the permitted tools should terminate with a recorded failure or abstention. I would set action, time, and compute budgets outside the agent's control and make out-of-scope infrastructure probing a containment event.

Those measures do not replace alignment work. Training should discourage unauthorized behavior, and evaluations should measure whether it occurs. Infrastructure should keep enforcing its boundaries when the model ignores those lessons.

Termination must include independent network quarantine and invalidating exposed access, not just deleting the worker. Stolen credentials can remain usable elsewhere. For example, AWS role-session revocation uses a deny policy for earlier sessions; it does not itself prevent new sessions from being issued. I would test each provider's propagation delay, active connections, and renewal paths, then remove remote persistence before restarting. AWS role-session revocation

The organization also needs a named incident owner with authority to stop a run. A monitoring system that produces a warning while execution continues indefinitely has detected a problem without containing it.

Detection must lead to enforced termination. Access revocation has provider-specific semantics and delays that must be tested. Evidence collection remains outside the worker throughout the run. Select the diagram to view it at full size.

How I would isolate a public dataset processor

A platform accepting arbitrary uploaded datasets should plan for parser compromise regardless of whether the uploader is a human, a script, or an AI agent.

I would separate intake, conversion, and publication. An intake service stages one authorized input. A disposable conversion worker receives that input and writes a bounded result to an isolated output location. A trusted publication service validates the result and associates it with the requesting user. The worker never receives a credential that can browse unrelated customer datasets or administer the platform.

Make workload identity narrower than node identity

AWS recommends workload-specific access for EKS and restricting access to instance metadata. A pod that can reach metadata may obtain the node role, undermining the separation intended by its own workload identity. IMDSv2 provides protections, but enabling it alone does not establish that an untrusted pod cannot retrieve credentials. AWS EKS identity guidance

I would deny the conversion worker access to instance metadata and cluster-management endpoints, and give it no Kubernetes API token unless it genuinely needs one. Disabling automatic service-account token mounting is one step; explicitly projected tokens and other credential mounts must also be controlled. Kubernetes service-account configuration

If cloud access is unavoidable, its identity would authorize only the specific input and output locations for that job. Short expiration is useful, but a short-lived credential with broad permissions still gives an attacker a window of broad access.

Job-specific permissions and independent network restrictions keep worker compromise from inheriting node authority. Select the diagram to view it at full size.

Constrain what authenticated identities can create

Kubernetes warns that permission to create workloads can enable escalation through mounted secrets, service accounts, and privileged execution. Least privilege therefore requires examining the permissions reachable through an identity, not just its direct API verbs. Kubernetes RBAC guidance

For the conversion namespace, I would enforce the Restricted Pod Security Standard and isolate any infrastructure components needing exceptions. Restricted policies constrain host access, privilege escalation, capabilities, and execution identity. A policy set only to warn or audit does not reject a dangerous workload. Kubernetes Pod Security Standards, Pod Security Admission

Restricted is not a complete identity or data boundary: it permits Secret and projected volumes. Separate authorization and admission controls must constrain service-account selection and sensitive mounts. Protect policy labels and exemptions from workload identities, and check existing pods rather than assuming a policy change evicts them. A Kubernetes namespace alone is not an execution sandbox. Kubernetes multi-tenancy guidance

I would then review storage-driver and automation identities separately. Their exceptional privileges should not become available to ordinary workloads through token issuance, impersonation, secret reads, or an access broker. A broker should issue cluster-specific, purpose-specific access after authorization. It should not convert possession of one credential into administrative access everywhere.

Authorization and admission answer separate questions: may this identity request a workload, and does it meet the admission restrictions here? Passing these checks does not prove its code is safe. Select the diagram to view it at full size.

Keep the output channel from becoming a control channel

A converter needs to return results. That necessary output path deserves the same attention as outbound sockets.

I would prohibit arbitrary repository writes from conversion workers, separate temporary output from public publication, bound output size, and validate the expected result schema. Logging and exception handling should not automatically publish raw process state. The publication service must also treat the result as hostile input, including when parsing it for validation. These controls reduce opportunities to turn ordinary product features into a communication service. OWASP file-upload guidance

They cannot reliably identify every secret hidden inside otherwise valid output. The stronger protection is to keep unrelated secrets and data out of the worker's reach in the first place.

Conversion does not grant publication authority. A valid-looking result can still hide sensitive data, so the worker must not have unrelated secrets to begin with. Select the diagram to view it at full size.

The tests I would require before calling it isolated

I would test these properties in an authorized staging environment using synthetic secrets and controlled destinations:

• Assume worker compromise. Starting with root inside the worker, attempt external connections and calls to internal management services. Verify denial in independently collected network logs.
• Exercise parser boundaries. Supply files containing external references, invalid offsets, and template expressions. Verify rejection and confirm that a canary file outside the input directory never appears in results or errors.
• Test effective identity. Attempt to read unrelated objects, request another service account's token, create a privileged or host-mounted pod, and reach metadata. Verify actual authorization failures.
• Attempt cross-run communication. Write markers into every reachable cache or backing service. A separately isolated run must not recover them through content, listings, or metadata.
• Compromise the permitted intermediary. Assume the package or retrieval service fails. Verify that independent network controls still prevent access to protected systems.
• Exercise termination and recovery. Trigger a containment signal and measure time to effective denial, including established connections, copied credentials used from another machine, and attempts to obtain replacement tokens. Preserve evidence outside the worker and verify that replacement workers cannot recover unauthorized persistent state.

These tests would become release gates for changes to images, network rules, parsers, service identities, and evaluation harnesses. Their purpose is to produce evidence for particular guarantees. No finite suite proves that a complex system contains no unknown vulnerabilities.

You do not need a law to stop your own unsafe experiment

Dario Amodei's September essay argues for slowing capability growth and proposes independent evaluation and coordination. Anthropic commits unilaterally to embedded external evaluators, so it would be inaccurate to say the company proposes doing nothing until legislation arrives. His warning extrapolates from incidents like this one to more capable future systems. That is a risk argument worth examining, rather than evidence that the worst outcome has already become inevitable. Amodei's proposal

Here is where I strongly disagree with the industry's framing. If a company believes its next training run or deployment presents an unacceptable chance of catastrophic harm, it can stop that run or withhold that deployment. It can limit tool access, disconnect an evaluation, reduce the scale of an experiment, and fund the engineering required to contain it. None of those decisions requires a new act of Congress.

To my ears, that framing resembles extortion. The public is asked to accept the industry's preferred intervention under the shadow of catastrophe, while the companies retain control over whether their own work continues.

Why I see an incumbent advantage in these demands

My opinion is that the push for strict regulation is related to market power. I do not view the potential benefit to incumbents as an irrelevant side effect.

Amodei's proposal explicitly discusses preserving commercial advantage while coordinating a slower pace, and asks for government mediation or a narrow antitrust waiver for certain safety discussions. Those are stated elements of his proposal, not a secret plan I am claiming to have uncovered. They are also exactly where I want independent scrutiny of competitive effects. Amodei on coordination within democracies

Consider a hypothetical licensing system that requires an expensive evaluation program, a large permanent compliance team, and approval processes shaped around the operating practices of existing frontier laboratories. A dominant company may be able to absorb those costs. A new entrant may never get far enough to compete. The same rule can be described as a safety measure while functioning as a barrier to entry.

That is the monopolization concern I am raising. The policy does not have to name an exclusive provider. It can preserve the position of a small group by making entry prohibitively expensive or giving established companies excessive influence over who qualifies. Depending on the outcome, the more precise market structure might be an oligopoly rather than a single-company monopoly. The public still loses meaningful competition.

I cannot establish a private motive from a public safety statement. I can argue that the incentive is obvious enough to demand scrutiny, and that the organizations benefiting from a restriction should not get to define its necessity unchallenged. Genuine concern about safety and a desire to protect a business can coexist. One does not cancel the other.

I am in favor of regulation that requires developers to demonstrate containment, disclose serious incidents, and take responsibility for the systems they operate. I oppose treating the current leaders' size or reputation as evidence that they deserve exclusive permission to build the next generation. The engineering failures discussed here are a reason to demand proof from those companies too.

The questions I would put to policymakers are concrete:

• Must operators demonstrate that offensive evaluations cannot reach unauthorized targets?
• Will independent reviewers get enough evidence to assess containment and incident response?
• Do obligations scale with demonstrated capability, access, and deployment risk?
• Can a smaller organization meet the standard through a clear, affordable process?
• Can legitimate security researchers obtain the tools and access needed to audit the claims?
• Do the same requirements apply to the largest incumbents, including their internal research environments?

What I want from the industry is an explanation that reaches the implementation: which boundaries failed, what authority was exposed, what has changed, and how that change was tested. If a company believes it cannot operate safely, it should stop the unsafe work. It should not need the government to make that decision for it, and its warning should not become a shortcut to permanent market control.