The Hardest Thing to Teach an Autonomous Agent Is When to Do Nothing

Why the Last 10% of Agent Development Is All About What Not to Do

There’s a documented story about a multi-agent system that slipped into a recursive loop and ran for 11 days before anyone noticed. Two agents had been talking to each other non-stop, burning compute while the team assumed everything was working fine. When the invoice arrived, it was $47,000. I bet there's a ‘On Day 12’ postmortem doc somewhere that reads like a true crime podcast.

Now that we’re deep into the era of autonomous agents that schedule meetings, move money, and talk to customers on our behalf, the ability to refuse, defer, or halt is turning out to be the most underrated dimension of autonomy. In the real world, 'I'm not sure, let me stop' beats 'I got this' more often than you'd think.

Most failures I’ve heard about in early agent deployments aren’t caused by bad reasoning. They’re caused by reasonable actions taken in the wrong moment, or actions that never should have happened at all. A Cruise robotaxi stopped for an emergency vehicle (exactly the right thing to do) but then refused to clear the lane afterward, blocking an ambulance’s response. A human driver would have reversed or moved aside, right? The autonomous logic lacked that judgment, and standing still seemed safest even though it was the wrong decision entirely.

So clearly, this is more a systems design problem than a prompt problem.

A common misconception is that refusal or deferral is primarily about ethics or alignment, something for the safety team to worry about while the rest of us build features. But in practice, at least in my experience, it’s really about side effects, irreversibility, partial failure, uncertainty and operational blast radius. These patterns should feel familiar to anyone who's worked with circuit breakers, rate limiters, two-phase commits, or human approvals for destructive actions. Autonomous agents are no different, except the decision logic now reasons its way through problems.

The major architectural insight that took me a while to accept is that if the model both decides AND executes an action, you’ve already lost control of the situation. In production-grade agent systems, the knowledge of when not to act lives at a boundary, not in the prompt. A typical flow separates three distinct phases where first the agent proposes an intent, then separate logic evaluates risk and confidence and scope and permissions, and only after that do authorized actions actually cause side effects.

The model proposes and the system disposes.

I assume that many teams start with a simple idea that if confidence is below some threshold, the agent shouldn’t act. This probably works for about two weeks before it starts failing quietly. The problem is that LLM confidence collapses multiple kinds of uncertainty into a single scalar, and these different kinds of uncertainty really matter. Epistemic uncertainty means the model genuinely doesn’t know something, while aleatoric uncertainty means the world itself is noisy or changing, and tool uncertainty means the model is assuming side effects it cannot actually observe. Different flavors!

A highly confident agent is not necessarily competent just like a cautious agent is not necessarily safe. More strong signals than raw confidence include things like disagreement between multiple planners, or tool verification failures, or world-state drift since planning began, or the inability to validate assumptions via read-only tools....or sometimes all of the above. So the better question isn’t “How confident are you?” but “What assumptions would make this action unsafe if they turned out to be wrong?”

One of the subtlest failure modes involves correct decisions being retried after partial failure. For eg, a tool call times out but actually succeeded on the backend. Another flavor of that would be that the agent retries a non-idempotent action because it doesn’t realize the first attempt went through. From the agent’s perspective, the plan is still valid and the last step appeared to fail so retrying seems reasonable. But from the system’s perspective, a charge has been duplicated or the same message has been sent twice.

A well architected system handles this by classifying tools according to side-effect severity, enforcing idempotency, requiring confirmation for irreversible actions, and preferring to pause and escalate rather than blindly retry. This together creates a more confident strategy.

As a “fun” thought experiment, look at the last inaction instance and see if you can explain why your agent didn’t act in a particular situation? Most teams log tool calls, errors, and outputs, but few log rejected actions, alternative plans considered, or reasons for abstention. Without this observability, you can’t debug hesitation, tune thresholds, or build trust. Probably my favorite realization so far is that decision traces, not just execution traces, make agents operable in production.

Here are some hilarious-but-ouch examples of moments when agents fail to “shut up” -> infinite defer loops where it keeps asking for clarification but never progresses("just to clarify" for eternity), over-escalation until humans stop trusting the system(the boy who cried “needs approval”), under-escalation until real damage is done, gradual confidence drift over long runs where certainty erodes but actions continue(boiling frog but make it AI). These are systems failures rather than model failures, and they never show up early enough.

Most conversations about agent safety focus on preventing bad actions through alignment and guardrails. That work is necessary but frankly, it’s insufficient. The harder problem is getting the agent to recognize when it should do nothing, not because the action would be harmful in some obvious way, but because the context is wrong, the timing is off, the assumptions are stale, or the side effects are unknown. Some builders are starting to push for uncertainty models that are traceable across reasoning steps rather than being collapsed into single confidence numbers, and that feels like the next research frontier to me because we need uncertainty that’s understandable and actionable rather than just a number between zero and one.

I think agents need meta loops like self-monitoring capabilities that measure error rates, confidence decay, and risk exposure, then use those signals to interrupt their own behavior. Something like internal halt conditions baked into the architecture, not external monitors bolted on after.

If any of this resonates, some questions worth asking about your own systems:

- Does your agent propose actions separately from executing them, or does it do both in one step?

- Can you explain why your agent didn’t act in the last 10 cases where it abstained?

- What happens when a tool call times out but actually succeeded on the backend?

- How does your agent know when its assumptions have gone stale mid-execution?

- Do you have test cases specifically for “should refuse” scenarios, or only happy paths?

As a formula

agent_reliability = (restraint × observability) / false_confidence

Unsurprisingly, the most dangerous agent is not the one that fails in obvious ways, but rather the one that confidently does the wrong thing and never had any way of knowing when to stop.