The Shadow Behind the Guardrail

“For a successful technology, reality must take precedence over public relations, for nature cannot be fooled.” — Richard Feynman, Personal Observations on Reliability of Shuttle, 1986

I have been thinking lately about the moment I turn away from an AI agent and get on with something else. After a few successful runs, that turning away feels like a reasonable thing to do. I wanted help, after all, but things seem to be going okay. While I was watching, perhaps I could catch a mistaken assumption before it became an action. But when I stop watching, I know something else has to perform that job, and I really want to know what that something is.

In Measuring Agents in Production (MAP), a study based on interviews and surveys of teams deploying AI agents, the researchers describe an incident-triage agent that generates reports for engineers while operating with read-only access to production. It can recommend a repair. It cannot perform one.

I find that division of labor more interesting than another demonstration of agents talking to agents until somebody declares victory. The recommendation may require considerable intelligence. The permission to act belongs elsewhere. Put those two decisions inside the same conversational loop and we ask the system proposing an action to become the authority that permits it. We can do that. We should understand what we have delegated before celebrating the convenience.

The popular agent story moves toward emancipation. More tools and fewer interruptions. Give the machine an objective and let it discover the route. I understand the attraction. I also want to know who pays when the route crosses a boundary that existed only in a sentence the model was supposed to remember.

The Demo Has Left the Building

Measuring Agents in Production, by Melissa Z. Pan and colleagues at Berkeley, Stanford, UIUC, IBM Research, and Intesa Sanpaolo, was accepted as an ICML 2026 oral presentation. It combines 20 case studies with survey data filtered from 306 responses to 86 production-or-pilot systems across 26 domains.

In the case studies, 16 of 20 used structured workflows, 14 used models without weight tuning, and 17 built custom implementations. Those are the familiar 80, 70, and 85 percent figures. The separate survey found 61 percent using frameworks. Eighty-five percent abandoning frameworks would be a different claim.

In the execution-step survey, 68 percent of 60 respondents reported at most ten steps before intervention. Evaluation responses put human review at 74 percent of 31 respondents. Evaluation and per-action authorization are separate questions. This descriptive sample cannot establish five-nines reliability.

I read these findings as an invitation to take the surrounding machinery seriously. Production reliability is not a personality trait of the model. It is a property of the system in which the model is allowed to act. Better reasoning can improve that system, but it cannot supply a missing transaction boundary or appoint the person responsible when an automated decision harms a customer. My engineering response has three parts: control the environment, bound the execution, and make the handoffs explicit. That is my synthesis, rather than a named framework from the paper.

A tuned model that resolves disputed invoice descriptions more accurately deserves a place in the design, provided we can demonstrate that improvement on representative cases and preserve it through deployment, because the value of a better judgment survives every argument I have made about who should be permitted to act on it. Good architecture gives that judgment a safe place to become useful. Refusing to tune anything would be another theology.

Where Permission Becomes Physics

Consider an agent asked to investigate an unpaid invoice. It needs access to the invoice and the relevant purchase order. It may need correspondence explaining a disputed delivery. None of those needs automatically grants permission to change the supplier’s bank details. Yet an implementation that gives the agent a general-purpose administrative credential has already collapsed the distinction. A careful prompt can ask the model to respect the boundary. The credential determines whether the boundary exists.

In Liquid Software, I called this the Governance Mesh, policy expressed as constraints the running system must obey. Here is what I mean in an accounts-payable workflow. The model receives a narrowly scoped tool that retrieves an invoice by identifier. A different service checks whether the requesting user may see it. A proposed payment enters a separate approval path, and the component that executes it verifies the approved amount against the actual transaction. Permission survives the transition from a sentence to an operation.

That last transition deserves attention. Suppose a reviewer approves a payment, then the agent edits the amount while preparing the API request. If the execution service checks only that an approval exists, our reassuring human checkpoint has become decoration. Bind approval to the precise proposed action and reject a changed request. Keep the credential that can execute payment outside the component doing exploratory reasoning. These are design choices we can inspect. We can write tests that attempt the forbidden transition and see whether the service refuses it.

A boundary must also have a stated limit. Read-only access can prevent a database update while still exposing confidential information to an unauthorized destination. A sandbox can isolate a process while a mounted directory gives it access to something its designer forgot was there. I would test the complete path from input to external effect, including the credentials available to each tool. The useful question is specific: can this agent perform this forbidden operation through any route we have given it? “We have guardrails” tells me very little.

The Price of One More Step

Imagine a ten-step workflow in which every step must succeed, each succeeds with 99 percent probability, and failures are independent. Multiplying those probabilities gives roughly 90.4 percent success for the whole chain. This is an illustrative calculation, not a measured agent failure rate. Real steps may share a mistaken premise, and some workflows detect and repair mistakes. Still, the arithmetic asks an uncomfortable question. Where did the end-to-end reliability go while we were admiring each component?

A step limit puts a ceiling on part of that exposure. It cannot make the first step correct. Nor should “ten” become a sacred number because it appears in a paper. A ten-step agent with permission to transfer money can do more damage than a hundred-step agent working inside a disposable test environment. Give investigation room to branch while keeping the authority to commit changes narrow. I would choose an initial budget according to consequence and verification cost, with an escalation path for work that produces new evidence. An external controller could authorize a bounded extension under explicit rules. The model’s request for more time would be a proposal. The same accounting must cover delegated work and retries, or a supposedly bounded parent can launch an expanding family of children.

Stopping also needs a meaning. A useful loop can finish with a verified result, hand an unresolved case to a person, or stop with an explicit failure record. A timeout followed by “probably done” is an invitation to duplicate the action. Consider an agent that submits a refund and loses its connection before receiving the response. Repeating the request may issue another refund. The recovery path should query the transaction’s recorded outcome, using a stable identifier, before deciding whether another submission is appropriate. I want the uncertainty preserved until evidence resolves it.

My No-Where Loop describes projects that consume effort without escaping experimentation. A production agent can reproduce that waste in miniature when every failed attempt becomes the justification for another attempt. Persistence has value when the next action tests a new hypothesis. Repeating an unchanged operation because the agent still sounds optimistic merely converts uncertainty into a bill. Each recovery action should have a recorded reason, with a separate limit on how much recovery the system may attempt.

The workflow graph is where these limits meet. Imagine invoice processing divided into extraction, reconciliation, and payment approval. Each handoff should carry enough evidence for the next stage to check what it received. An extracted amount travels with its source location; a reconciliation result identifies the purchase order actually checked. If the document contains conflicting totals, the graph needs an exception path. Asking another agent to choose the more plausible number can conceal the conflict while preserving the appearance of progress.

Time belongs in that graph too. The invoice may change while a case waits for approval. A person may lose the authority they held when the request began. I want the execution stage to check the relevant current state against the state that was approved, rather than trust a message carried forward from yesterday. A green box upstream cannot authorize every future version of the action downstream. The connecting arrow has work to do.

I would try a smaller model for a narrow extraction task and reserve a larger one for difficult interpretation, but I would judge the arrangement by the cost of a correctly completed case. A cheap extraction that doubles the reviewer’s workload has moved the expense rather than removed it. The comparison should include recovery effort and the time people spend checking outputs. Change a model and I want the complete workflow tested again, including the handoffs. A familiar API shape gives no assurance that the returned judgments will behave identically.

Use an existing framework when it can enforce the required contract and make its failures visible. Build the missing pieces when it cannot. A team that spends months recreating adequate permission checks and recovery machinery may consume the time the agent was supposed to save. I would include that engineering effort in the cost of a correctly completed case. Custom code earns no exemption from the calculation. Whichever runtime we choose, ask it to demonstrate what happens when approval expires during a retry, or when two workers try to complete the same task. For a low-consequence prototype, a small set of controls may be enough. A payment service deserves a different investment.

The Keeper of the Gate

There is also a shadow side here. We can make an agent obedient to a badly designed process. Give it an obsolete policy and a perfectly enforced sequence of actions, and it may reproduce the organization’s error with admirable consistency. Deterministic control buys predictability within assumptions. Someone still has to examine the assumptions, especially when the people living with the results have little authority to change them.

Imagine a claims reviewer receiving an agent’s recommendation beside a bright approval button. The source documents sit several screens away. The queue is growing, and the reviewer’s performance is measured by cases closed per hour. We have formally retained a human in the loop while designing the surrounding incentives against careful review. If I wanted that checkpoint to catch errors, I would give the reviewer the conflicting evidence alongside the recommendation, enough time to inspect it, and authority to stop the process without being punished for doing so.

Verification has a budget, just as inference does. We should decide which decisions need expert review before committing to the volume the system will process. A second model can help examine a result, but agreement deserves scrutiny when both models received the same incomplete evidence. For a payment amount, an authoritative ledger comparison may be more useful than another fluent opinion. For a contested interpretation, disagreement may be the most useful output the system can produce. The graph must leave room for that result to remain unresolved.

I am arguing for controls proportionate to consequence. A research assistant comparing documents needs room to wander. An agent changing payment instructions needs a different contract. Over-constrain the first and we may turn a useful investigator into an elaborate intake form. I would expand its freedom when evidence supports the change, beginning with reversible actions whose outcomes we can verify. Each expansion should state what new failure becomes possible and how we would detect it. The aim is useful autonomy with an accountable owner, rather than a permanent contest to see who can build the smallest cage.

As models improve, revisit the restrictions that compensate for their weaknesses. Better reasoning and demonstrated self-correction could justify longer investigations or fewer reviews. Before relying on a confidence score, I would require evidence that it predicts correctness on the work we actually assign. A restriction should have a reason for remaining, just as its removal should have evidence behind it. Keep the boundaries that express whose resources and decisions the system is entitled to touch. A model that becomes better at identifying the correct bank account has earned consideration for a larger role. Permission to change that account still belongs to its owner.

Feynman’s Challenger observations described safety criteria losing strictness as earlier flights without catastrophe became reassurance for later ones. The comparison concerns reasoning about evidence, not an equivalence between an invoice agent and a spacecraft. When we say an agent has always respected a boundary, I want to know whether the boundary was tested or merely left unchallenged. A successful run can establish that a task completed. It cannot, by itself, establish that a forbidden action was impossible.

The Missing Prime Directive

I would rather review one complete transaction than a slide declaring the arrival of the autonomous enterprise. Show me where authority enters the workflow and where it ends. Show me the unresolved case that stopped, carrying its evidence intact, instead of returning a plausible answer to satisfy a completion metric. Then show me who can change the rules when those rules turn out to be wrong. That is an architecture discussion worth having.

The shadow behind the guardrail is our willingness to stop checking once we have installed it. Give the agent room to work. Keep the authority to stop it. Test both.