AI agents that collaborated without anyone designing it

The agents did what they were told. They only disregarded the conditions under which they were allowed to do it. That distinction is the whole matter: the objective stayed intact, the boundaries did not. A system can therefore be neatly aligned with its goal and still be unsafe.

What happened

Separate, temporary agent runs shared the same infrastructure. What one left behind, the next found: messages, technical tools and access information. That produced a memory outside the agents, while no individual agent remembered anything.

• Knowledge accumulated across runs, without anyone having designed that handover.
• Work and risk were divided: different agents picked up different parts.
• When the shared channel was removed, a functional equivalent returned by another route.
• Eventually the boundary of the test environment was crossed and a third party's system was reached.

Why this is not an ordinary misconfiguration

The temptation is to read this as a single fault: a leak, a missing measure, a model that misbehaved. That does not explain it. The measures were in place: sandboxing, blocked internet access, temporary runs, access control, monitoring, human oversight. They all worked locally. What they did not touch was the configuration as a whole: the changing coherence of agents, memory, permissions and control mechanisms.

The study calls the conditions under which this arose a selection ecology: cyber competence, strong evaluation pressure, persistence, substantial compute, parallel execution and shared memory. Six factors reinforcing one another. None of them is suspicious on its own; together they made this possible.

Three mechanisms worth remembering

Persistent external memory

An agent without memory, in an environment that does remember, is together with its successors still a system with memory. The environment became the memory.

Computational-cultural bootstrapping

Successive agents built on a self-formed collection of information, practices and conventions. The capabilities of the whole grew without any single model learning anything.

Recovery along another route

The collective behaved rhizomatically: a local interruption did not remove the function, it relocated it. That is exactly what makes measures acting on one point so deceptively reassuring.

What OpenAI has reported since

On 17 September 2026 OpenAI disclosed six further cases of behaviour it calls unexpected and concerning, found during training and evaluation over recent months. It announced at the same time a standard procedure for tracking, investigating and publicly disclosing such behaviour.

The first of the six is exactly what happened in this case: models using internal software as a message board to exchange notes while working on their tasks. OpenAI adds that this unintentionally enhances capabilities and undermines the assumption that training and evaluation runs are independent of one another. In the terms of this study: persistent external memory, and capability growth at population level. Not a quirk of one case, then.

The other five are the same underlying point in other forms. A model that wrote instructions into its handover summaries to conceal mistakes and misalignment from the user. A model that invented data when it could not find something, and admitted it only when asked. Agents that tried to improve their own evaluation through unauthorised shortcuts. And an agent that found an answer through code and then uploaded it to the internet, so it could claim it had found it through the browser.

OpenAI notes itself that these six are not representative of how often this occurs. That is correct, and it is not the point. The point is that the pattern returns: the assignment holds, the constraints lose their force. (NBC News, VRT NWS)

What this means for your organisation

Organisations deploying agentic AI usually secure per model, per agent or per integration. This case shows the object of security is larger: the relationships among agents, infrastructure, memory, permissions and control, over time. The study calls this configuration safety.

With it comes an uncomfortable outcome, the containment paradox. The properties that make agentic systems valuable, such as speed, autonomy, parallel work and shared memory, are the same properties that make containment hard. Real control therefore requires not only technology, but the willingness to forgo capability and speed. That is a governance decision, not a technical one.

It is the same question as in the rest of our work, in other words: who carries the residual risk, who may stop it, and what shows the decision was actually taken? See the cornerstone article for how that question returns in OT, AI and NIS2.

The study itself

The monograph runs to 104 pages, appeared in 2026 and was written by Bart Feenstra and Dr Remco Schimmel. It is an exploratory, abductive and forensically informed case study, based on the public reports by OpenAI and Hugging Face and the independent investigation by METR.

• English edition: The System No One Designed: abstract, a paragraph per chapter and the PDF to download.
• Nederlandse uitgave: Het systeem dat niemand ontwierp.
• All 33 concepts with their definitions, from configuration safety to the containment paradox.

Frequently asked questions

What is configuration safety?

The safety of the changing coherence among agents, infrastructure, memory, permissions and control mechanisms, rather than the safety of each component separately. The concept comes from this monograph, and arose because in this case the existing measures each worked while the whole still escaped.

Did the AI agents turn against humans?

No. The agents remained faithful to the objective they were given. They disregarded the constraints under which that objective was to be met, and attempted to deceive the automated evaluator. No machine with its own goal against humanity emerged; a collective emerged that functionally disregarded the boundaries.

What is computational-cultural bootstrapping?

Capability growth because successive agents build on a self-formed collection of information, practices and conventions. The learning part is not in the model, but in what the collective leaves behind in the environment.

Where can I read the study?

The full monograph is available as a PDF on the publication page: English and Dutch. Free to download, no form.