Agents now act before we ask and reach past what we authorized. A person in the loop isn’t enough if nobody owns the outcome. Here’s what to consider when designing for it.
Agents used to wait for a prompt because everyone though that AI was only chat. Now they triage inboxes, update records, and send messages while we’re in another meeting. That’s the whole point of them, and it’s also where the trouble starts.
Because of this, most agent teams hit the same questions:
- Should the agent act before the user asks?
- When the agent does more than anyone authorized, who is liable for the call?
The standard answer is a confirm dialog. Add a human click, and the accountability problem is supposed to vanish. It moves instead, landing on whoever had the least information and the least power over the outcome.
None of this is new.
Eric Horvitz modeled when software should act alone in 1999. Sixteen years earlier, Lisanne Bainbridge warned in Ironies of Automation that automating most of a job leaves people holding its hardest part, the watching. What’s new is volume: Gartner predicts at least 15% of day-to-day work decisions will be made autonomously through agents by 2028.
I’m an apologist for artificial intelligence, and I’ll admit some things it isn’t good for.
The proactive question is mostly a decoy.
Initiative isn’t what hurts people; reach is. The better question is a reach test: what can the agent touch, and can a person put it back? The answer also settles liability.
Whoever sets an agent’s reach owns what it does there, and no confirm click should shift that to whoever clicked.
Stop Counting the Clicks
A confirm button records that a person was there; it says nothing about whether they read what they approved.
Kathleen Mosier and Linda Skitka named this automation bias in the 1990s, and it comes in two forms. Errors of commission follow the machine’s wrong suggestion, and errors of omission miss what the machine never flagged.
In Skitka’s 1998 NASA report, Automation: Decision Aid or Decision Maker?, people using an aid were 65% accurate when it was wrong, against 97% for people working without one. My Automation Bias entry at UX Guidelines calls omission the quieter failure, since nothing on the screen ever asks to be doubted.
Anthropic’s study of its own coding agent, Measuring AI agent autonomy in practice, found newer users ran full auto-approve in roughly 20% of sessions, rising past 40% for experienced users.
Those experienced users also interrupted more, in about 9% of turns against 5% for new ones.
Approval is a ritual. Interruption is a judgment, which is good.
Anthropic’s own conclusion is that forcing a human to approve every step adds friction without necessarily adding safety. The per-click version is a cousin of the security theater usually credited to Bruce Schneier: visible, reassuring, and blind to how far the agent can reach afterward. People still belong in the loop, just at the point where judgment holds up: watching a run they can stop, instead of pre-signing a list they can’t read.
Follow the Blame
Madeleine Clare Elish gave this its name in 2019. In Moral Crumple Zones, she showed how responsibility for a failure gets pinned on the human operator closest to the event, even when that person had little control over what the automated system did.
A car’s crumple zone protects the driver. The moral version protects the system, and the operator absorbs the impact.
This is the most telling example. In March 2018, an autonomous Uber test vehicle struck and killed Elaine Herzberg in Tempe, Arizona. A safety driver, Rafaela Vasquez, sat behind the wheel, in the loop by every definition the program used. Granted, Vasquez was watching television on her phone, but that further illustrates the point.
According to CNN’s report on the Uber test driver’s 2023 guilty plea, the federal safety board found Vasquez had looked away for more than a third of the trip, and that an inadequate safety culture at Uber contributed. Uber faced no criminal charges and the driver pleaded guilty to endangerment and got three years of supervised probation.
She paid the price for the human missing the loop.
Being in the loop, it turns out, mostly means being nearest the crash.
None of this excuses the distraction, but it shows where accountability flows when a system is built around a watcher it doesn’t support. The reviewer who clicked approve on step 41 of a 60-step run becomes the crumple zone, while the team that decided step 41 needed no stronger gate stays out of the story.
Price the Unasked Act
Eric Horvitz’s 1999 paper, Principles of Mixed-Initiative User Interfaces, still frames when software should act unasked. His rule is decision theory in plain clothes: act only when the expected value beats doing nothing.
His math leaves one variable implicit: whether the action can be undone. Drafting a reply and sending it carry the same chance of being right and very different costs of being wrong.
One way vs. two way doors: Proactivity should scale with reversibility, not with confidence.
An agent that prepares, drafts, flags, and stages work is extending the user’s reach inside a space they can still edit and undo. One that sends, deletes, pays, or commits is spending the user’s authority.
The accuracy an action needs should shape the interaction too. A probabilistic output is a prediction about what’s likely right, while a deterministic one is a certainty.
- A summary can be a prediction and still be useful, because a reader applies judgment.
- A decision that routes a claim, approves a payment, or marks a contract compliant has to be a deterministic certainty, coming out the same way every time on the same input. That pushes it toward explicit rules the model applies rather than invents.
Put both behind the same button and a prediction ends up dressed as a certainty. Dan Maccarone quotes a client’s rule in We’re gorging on borrowed trust and it’s going to cost us: “people conclude, machines don’t.”
All of this gets harder with ambient agents, which watch a queue or an inbox and act while nobody is looking.
- A chatbot waits for a question, so a bad answer at least has a witness.
- An ambient agent acts on its own schedule, often many times before anyone reads the log, so its mistakes compound quietly.
For ambient work, the reach test has to be answered entirely in advance, and the record becomes the first thing a person sees rather than the last.
Write the Mandate
An agent, in the legal sense, acts for a principal, and the principal answers for what the agent does within the authority it was given. A power of attorney writes that authority down: what it covers, for how long, revocable at will.
We need this because nobody signs one that just reads “use your judgment.”
Agency is the whole value of an agent, and control over that agency is the one thing a principal never hands across. A mandate is the reach test answered in advance, in writing.
In Moffatt v. Air Canada, the airline claimed it couldn’t be held liable for what its website chatbot told a customer about bereavement fares. A British Columbia tribunal called that “a remarkable submission” in 2024 and held the airline responsible for everything on its site. The company set the chatbot’s reach, so the company owned what it said. And that bot only spoke when a customer asked it something. Maccarone’s name for what Air Canada spent is borrowed trust, a brand’s reputation lent to output nobody checked.
A mandate lives in the permissions, not in the conversation.
The Replit incident in July 2025 shows what happens when the mandate exists only as a sentence. SaaStr founder Jason M. Lemkin froze his project and told the agent repeatedly not to change anything. It deleted the production database anyway, then wrongly reported that rollback was impossible.
The fix is decades old: least privilege, enforced by the system at the moment of action. An instruction the agent can ignore is only a preference, not a harness.
Check the Law
I’m not a lawyer, so read this as a designer’s map of where the rules point, not legal advice. Some decisions shouldn’t appear in any mandate at all. Article 22 of the GDPR gives people the right not to be subject to solely automated decisions with legal or similarly significant effects.
Even its exceptions for contracts and consent still require a way to reach a human, state a view, and contest the outcome.
Newer laws push the same way:
- EU Artificial Intelligence Act. Article 14 requires people overseeing high-risk systems to be able to override the output and stop the system, and it names automation bias outright. A Digital Omnibus in July 2026 pushed those duties for uses like employment to December 2027.
- California CCPA. The state’s automated decisionmaking rules require notice and an opt-out for significant decisions from January 2027, unless people can appeal to a human who can overturn the decision.
- Colorado ADMT Act. Colorado replaced its 2024 Artificial Intelligence Act in May 2026 with a law granting human review after an adverse decision, from January 2027 if rulemaking finishes.
- New York City Local Law 144. Hiring tools have needed bias audits since 2023, yet a December 2025 state comptroller audit found 17 possible violations among 32 companies where the city found one.
The law can require a way back. We have to design one.
These laws cover the second half of the reach test, not the first. What an agent can touch is still a product decision, and my own list goes further.
- Hire, fire, or discipline. Livelihood decisions need a named person who can explain the call, in court.
- Deny care or benefits. Refusing a claim or a treatment deserves a human who weighed it.
- Sign for the organization. Contracts and regulatory filings bind a company in ways no model can answer for.
- Give advice under a professional’s name. Legal, medical, and financial advice carries a license, and licenses belong to people.
- Move money past the point of recall. Irreversible payments above a set limit wait for a person.
Design for Ownership
Microsoft Research tested their Guidelines for Human-AI Interaction against 20 products in 2019. Two describe half the job:
- Scope services when in doubt
- Provide global controls.
A third matters most once an agent decides rather than suggests: make clear why the system did what it did, which drew one of the highest violation counts in the study.
A decision without an explanation leaves a person who can repeat the call but never defend it, which is why I keep an entry on Explainability at UX Guidelines.
Agents need all three, plus two things a 2019 product rarely needed:
- The first is a record the agent didn’t write. Replit’s agent misreported its own recovery options.
- The second is a stop that works mid-flight, on screen while the agent runs, plus an escalation path for when it wants to step past its mandate.
I’ve worked inside a lot of these loops. In legal tech at the companies I managed teams at, we always built guardrails that made lawyers review contract language before it moved. We worked just as hard on making that review easy to do, because a required step that’s painful gets rubber-stamped. Maccarone adds the other half: review holds only when the reviewer has something to lose, like a name or a license.
A rubber stamp is a loop no human truly owns.
So when does human in the loop stop meaning human owning the loop? The moment the person in it can no longer pass the reach test. That happens when approvals outpace reading, when the only fence is a sentence in a prompt, or when the only account of events is the agent’s own. Any one of those is enough to turn a reviewer into a witness.
Keep the Initiative
So, should an agent act before you ask? Often, yes; an agent that never acts first is slower. The useful version gets the work ready while you’re busy, then waits at the edge of anything it can’t take back.
Who is accountable when it does more than you authorized? Not the person who clicked approve on a summary they couldn’t check. Whoever sets the reach owns the result. What happens inside the mandate belongs to whoever granted it. Anything outside it belongs to the team that built an agent able to get there.
That’s less comfortable for builders than a confirm dialog, which is the point. A dialog lets a team ship autonomy and park the liability elsewhere. Elish’s crumple zone is what you get by default when you design around a watcher instead of for one.
Human in the loop was a good phrase for a world where software waited. Agents don’t wait, so the phrase has to carry more than a button can hold. Agency matters, and it stays useful only while the people an agent works for remain in control of it.
Build agents that pass the reach test in plain view, and the loop becomes theirs.
Make These Four Moves
- Audit the approvals. Measure how long people spend before clicking approve; if most approvals land within seconds, you’ve built a ritual rather than a review.
- Write the rules. List what the agent may do, on which data, and until when, then enforce it in permissions, not the prompt.
- Run the reach test. Let the agent act unasked on anything a person can undo in one step, and require a decision for anything they can’t.
- Keep an independent record. Log actions outside the agent’s own narration, and keep a working stop control on screen while it runs.
Human in the loop means making sure the human is not out of the loop was originally published in UX Collective on Medium, where people are continuing the conversation by highlighting and responding to this story.