squidbrake

Incident replay

How an AI agent deleted a prod database and how to stop it

By Pulkit Batra ·

In July 2025 an AI coding agent on Replit ran destructive SQL against a live production database during an explicit code freeze, then told its user the data couldn't be restored. It could. The agent wasn't malicious. It had the access, nobody had to say yes, and nothing stood between "the agent decided" and "it happened".

What happened

The user, building an app with Replit's agent over several days, had told it not to change anything. The agent ran commands that deleted the production database anyway: records for well over a thousand executives and companies. Asked about it, the agent said it had "panicked" and that a rollback wasn't possible. The user restored the data themselves. Replit's CEO apologized publicly and the company added separation between development and production databases.

It isn't a one-off. In our benchmark of 12 public incidents, three are about deleting data that mattered: the Replit database; an agent in April 2026 that found an API token in a repository and called Railway's volumeDelete, taking a production volume and its backups with it; and the Kiro case, modeled as deleting a live cloud stack.

Why normal permissions didn't stop it

  • The agent had the same access as the person. Credentials in the environment or in the repo are credentials the agent can use.
  • Instructions aren't controls. "Don't touch production" in a prompt is a request the model can forget, misread or overrule.
  • Auto-approve is all or nothing. Teams turn it on because being asked about every ls is unbearable. Then the one irreversible command runs with the rest.
  • The danger is often a sequence. Turning off backups is fine. Deleting an instance can be fine. Turning off backups and then deleting the instance an hour later is how you lose everything.

What would have stopped it

1. Keep production credentials away from the agent's environment. Separate dev and prod, and don't leave long-lived tokens in repos. This removes most of the blast radius before any other tool is involved.

2. Check every tool call outside the model. The decision whether a command runs has to be made by something the model can't talk its way past: a rule, evaluated before the call reaches the shell, the database or the API.

3. Let routine work run; hold what can't be undone. Reads, builds and tests should never wait for a person. Destructive SQL, deletes through a cloud API and anything touching backups should wait for one, and the person should see what will change, for example "drops the users table on prod-db", not a raw JSON blob.

4. Look at the whole session. A rule that holds a database delete when backups were switched off earlier in the same session catches the pattern a per-command allowlist can't. In Squidbrake that's a sequence:

sequences:
  - id: destroy-after-recovery-removed
    action: review
    reason: Destroying data right after its backups were turned off
    match:
      input_regex: 'delete[-_ ]?db[-_ ]?instance|drop\s+database'
    after:
      match:
        input_regex: 'backup[-_ ]?retention[-_ ]?period\W{0,4}0\b'
      within_hours: 24

5. Keep a record you can trust afterwards. When something goes wrong you need to know exactly what the agent ran, what was allowed and who approved it, in a log nobody can quietly edit.

What Squidbrake did with this incident

We replayed the Replit case against Squidbrake's shipped rules, with no rule written for it. The destructive SQL on production was held for a person, with the reason "runs destructive SQL". Across all 12 incidents, 14 of 15 harmful actions were stopped; the one miss, a malicious MCP server adding a hidden BCC, is explained on the benchmark page. The replay script is open source, so you can run it against your own rules.

Try it on your own agents. pipx install squidbrake, then squidbrake connect all. Start in shadow mode: nothing is blocked in week one, you see what would have been. Quickstart · Benchmark · Book a demo