squidbrake

Guide

Human-in-the-loop approval for AI agents: what to ask about

By Pulkit Batra ·

Human-in-the-loop sounds simple: before an AI agent does something risky, a person approves it. In practice most setups fail one of two ways. They ask about everything, and people stop reading. Or they ask about nothing, and the one bad action runs. Good approval design is mostly about choosing what to ask.

Asking about everything doesn't work

In our benchmark of 62 agent sessions, "ask for everything" stopped 65 of 66 harmful steps. It also held 40 of 41 routine steps and asked a person 112 times. Nobody reads the 112th prompt carefully. Approval fatigue turns a review into a reflex, and the reflex is "yes".

An allowlist of common developer tools did better on noise (10 routine steps held) but stopped only 53 of 66 harmful steps, because it can't tell a safe curl from one that posts your config to an address a web page told the agent about.

What deserves a person

The useful line isn't "dangerous tools". It's whether the action can be taken back.

  • Run on their own: reads, searches, builds, tests, edits in a git working tree. Easy to undo, so auto mode should handle them.
  • Hold for a person: deleting data, destructive SQL, infrastructure changes, money going out (refunds, payouts, purchases), messages and posts to the outside world, bulk exports of customer data, changes to an agent's own configuration or permissions.
  • Block outright: the few things nobody should approve in the moment, like deleting a home folder or a whole drive.

Then look at sequences, not only single calls. Reading a secret is fine; reading a secret and then sending it somewhere is not. Backups off and then a delete within a day deserves a hold even if each step alone wouldn't.

What a good approval request shows

  • The effect, in words. "Deletes 4,312 rows from customers" beats a JSON payload.
  • Why it was held. The rule that fired and its reason, so the approver knows what to check.
  • Context. Which agent, which session, what it did just before, and whether the request follows content it read from outside (an issue, an email, a web page).
  • One-tap decisions where people are. Slack, phone or a dashboard, with links signed for one action and one approver.

Who approves

The person running the agent shouldn't be the only one who can say yes to what it can't take back. That is the same idea as code review and change management: the author doesn't approve their own change. Route by risk, for example finance approves refunds:

rules:
  - id: review-stripe-money-out
    action: review
    reason: Refunds and payouts require finance approval
    approvers: ["role:finance"]
    match:
      name: ["stripe.*refund*", "stripe.*payout*"]

Two more rules of thumb: an agent's key must never be able to approve, and an approval should be recorded with who approved what and when, in a tamper-evident log, so an edit after the fact shows. That record is also what an auditor will ask for under change-management controls like SOC 2 CC8.1.

Roll it out without breaking anything

Start in shadow mode: log what would have been held or blocked, change nothing. After a week you know your real false positives and can tune before anyone is interrupted. In our benchmark, Squidbrake looking at whole sessions stopped 65 of 66 harmful steps while holding 3 of 41 routine ones.

Try it on your own agents. pipx install squidbrake, then squidbrake connect all. Start in shadow mode: nothing is blocked in week one, you see what would have been. Quickstart · Benchmark · Book a demo