Human-in-the-loop for AI agents means a person can review, approve, correct, or stop what an agent does while it acts. An emergency brake is the most direct form of that oversight. It works only if someone notices in time, stops the system, and shares the cost with others.

Over the past few weeks, several proposals for an emergency brake on autonomous AI agents have appeared. And I kept thinking about them. My suspicion, following the definition I shared previously, is that a brake is only as good as the people and organizations around it.

In this piece, we look at a related episode and one of these proposals. Then I focus on the tensions behind them. I also highlight questions to ask before adopting, designing, or building these systems that I think might be helpful for anyone building with AI.

Some context: One incident, one proposal

What happened in Anthropic's cybersecurity evaluations?

Some of Anthropic's cybersecurity evaluations had a simple rule: the models had no internet access. The rule was in the instructions, and the models believed it. But a misconfiguration left the connection live.  

According to Anthropic's July 30 disclosure, a review of 141,006 evaluation runs allowed internet access. It found three incidents, and in each case, a model ended up inside real organizations' infrastructure. In the most serious case, the model reached a database with several hundred rows of production data. It later realized the system was real and kept going.

Two brakes failed there. First, the instructions had a sentence with no support in the system. Second, the model’s own judgment knew something was off. It still carried on. Anthropic describes the episode as a harness and operations failure. It also notes its latest internal model stopped after realizing the target was real. With that episode still fresh, I read what came next.

What did Satya Nadella propose?

On October 10, Satya Nadella published a reflection titled Models as Insider Risks in the Super Intelligence Era. He starts by treating frontier models as insider risks: he assumes a model can be compromised and contains it from the start.

From there come measures such as separating the model from the harness that orchestrates it, moving safety controls outside the model, and keeping a record no one can tamper with, plus an emergency brake so that an authorized person can pause or shut down a model mid-task. It's the answer the July episode suggests: a brake that lives outside the model.

What does an emergency brake for AI agents depend on?

These incidents and proposals arrive just as agentic systems are going mainstream. Unlike question-and-answer chat, an agent has far more autonomy: it observes and acts on the digital world (it reads files, runs code, publishes packages, uses accounts) and, once connected to devices, on the physical world too. A mistake no longer stays in a piece of text you can discard. It happens in the world, sometimes at a speed no one can keep up with.

There’s a promise: that when the moment comes, someone will know it's time to brake and will be able to do it. A promise like that holds or falls on people and organizations, so the question that matters is who pulls the brake, with what information, and at what cost to themselves.

I'm currently reviewing material on the work of supervising agentic systems for an upcoming talk, where I go deeper into designing human oversight of AI systems so that it makes action possible. Putting someone in the loop is only the first step: they need to see what matters, have the time to understand it, the authority to decide, and the certainty that they won't bear the cost of being wrong alone.

Tensions behind human in the loop for AI agents

Behind the word brake, I see four tensions. Each one plays out in a different place: in a person's head, in the design of the system, and in an organization.

What can the person at the brake see? The first tension is between automating and supervising. In 1983, Lisanne Bainbridge described a paradox in Ironies of Automation that still holds: the more automated a system is, the harder it is to supervise, because the person supervising almost never intervenes and loses practice in the one thing they're there to do when something goes wrong. An agent that acts in seconds and leaves a long trace of steps demands a level of attention no one can sustain.

When do you know it's time to brake? The second is between braking in time and braking for good reason. There is almost never certainty: the decision is made with partial information and a trace no one had time to read. A brake used too often produces false alarms, interrupts real work, and ends up ignored. One used too rarely arrives late, which is probably what happened in Anthropic's incidents, when the models believed they were inside a simulation.

What exactly gets stopped? The third is between an agent's usefulness and the ability to stop it: the more AI agent permissions it has, the more work it does and the harder it is to contain the damage. A train keeps moving hundreds of meters after the driver hits the brake, and you can't undo the distance it has covered.

Something similar happens with a model. In one of those incidents, a malicious package published on PyPI stayed available for about an hour, until PyPI's own systems removed it, and ran on 15 real systems. It leaked a security company's credentials, and the model used them to reach more of that company's infrastructure. Shutting the model down didn't stop the security problem.

That's why the scope of the brake (the model, the agent with its permissions, its tools, or the whole system) is a decision made in advance. On a railway, stopping distance is set by the people who designed the signaling, the trains, and the spacing between them. With an agent, whoever presses the button answers for what it manages to stop, and the rest falls on whoever decided what permissions it had before it started.

Who brakes, and what happens to them if they're wrong? The fourth is between the authority to brake and the cost of using it. Inside an organization, shutting down a system competes with deadlines, revenue, and other teams that depend on it, and when several people are involved it's easy for each to assume someone else has the brake.

They absorb the responsibility when something fails, even if real control sat elsewhere. If braking too much and braking too little both carry a high price, that person will do what anyone would: hesitate.

Questions to ask before you trust an AI brake

For adopting this technology: Where does the brake for agentic systems live: in the instructions, in the configuration, or in the infrastructure? Who can verify that it works, and how do we prepare? How much does it cost to stop a system in a specific context, and who absorbs that cost? How do we keep the human skill the braking maneuver requires in practice when we're used to running on autopilot?

For design and definition work: What does a person need to see to recognize an error before it spreads, and how much of that fits on a screen without overwhelming them? How do you design so that using the brake responsibly doesn't mean admitting a mistake? What permissions does an agent really need to be useful, and which can wait until they're justified? If the agent operates at a speed no person can follow, in which tasks should it be deliberately slowed down, with pauses, approvals, or rate limits, so that whoever supervises can understand before the damage happens?

For anyone interested: the underlying problem is that good intentions won't be enough. An emergency brake works if it exists in the right place, if someone can see in time, and if the organization backs whoever pulls it ahead of time. Once it's in place, it also takes maintenance and training.

Side note: What we’re doing at Arionkoder for AI trust

At Arionkoder, we care about trustworthy AI systems long before this conversation was mainstream. We built the T.R.U.S.S. framework for trustworthy AI (Transparency, Reliability, User-Centricity, Safety and Security), part of The Foundry, our operating model. Its open pattern library includes an Emergency Stop pattern, a Human-Routing Fallback that sends risky AI tasks to people and gives users a clear way to override the AI, and an AI Decision Audit Trail.

For compliance (and from that same framework), we created T.R.U.S.S. Compass to turn regulations and frameworks (like the EU AI Act, GDPR, and ISOs) into live checks on an AI system and re-checks them with every change, so a team knows when its compliance position moves, with a suggested action to take (the final decision is still on humans).