Running agents
What your agents get wrong is a dataset
A script logs every command your agents ran that failed, groups them into kinds and ranks them, so the same trap is fixed once instead of met every day.
One of 15 patterns I've found working with AI coding agents. An agent drafted the words from my notes, and I haven't edited them yet. See all 15, or read how they are made.
An agent drafted the words from my notes, and I haven't edited them yet. The code was run before it went in.
The problem
When a coding agent runs a command that fails, it reads the error, tries another way and carries on. The task finishes, so from where you sit nothing happened. The failed attempts cost time and money, and they are written in a transcript, the saved log of the session, that you will probably never open.
Then the same thing happens tomorrow, in another session, for the same reason. Each agent starts with no memory of the last one and meets the same trap for the first time.
In my logs, most of these failed commands happen where you look least. A main agent often hands work to sub-agents, helpers it starts for one task, and sees only their final answer. In one codebase, 385 of 615 logged failed commands, 63%, were inside sub-agents.
The idea
Treat the failed commands as data. A hook, a small script the agent tool runs when a session ends, reads the transcript and logs each failed command. No model is involved, so it costs no model calls. The errors are then reduced to kinds, by replacing paths, numbers and quoted values with placeholders, and the kinds are ranked by count.
When you fix a kind, you enter the fix in a register with the time it landed. Older records of that kind stop counting. A new one after the fix, in the same codebase, is reported as a regression: the fix did not take.
A version small enough to read
Nine failed commands from two codebases, written out by hand, and one fix in the register.
// Groups failed agent commands into kinds, ranks them, and reports a failure// after its recorded fix as a regression. Run: bun rank.tstype Failure = { at: string; repo: string; command: string; error: string };
const log: Failure[] = [ { at: "2026-11-01T09:12", repo: "shop", command: "cat /tmp/run-8812/out.log", error: "No such file: /tmp/run-8812/out.log" }, { at: "2026-11-01T14:40", repo: "shop", command: "cat /tmp/run-1093/out.log", error: "No such file: /tmp/run-1093/out.log" }, { at: "2026-11-02T10:03", repo: "shop", command: "cat /tmp/run-5521/out.log", error: "No such file: /tmp/run-5521/out.log" }, { at: "2026-11-02T16:20", repo: "blog", command: "cat /tmp/run-7730/out.log", error: "No such file: /tmp/run-7730/out.log" }, { at: "2026-11-03T08:55", repo: "shop", command: "npm test", error: "npm: command not found" }, { at: "2026-11-03T11:30", repo: "shop", command: "npm test -- cart", error: "npm: command not found" }, { at: "2026-11-04T09:00", repo: "shop", command: "git push origin feat-41", error: "rejected: fetch first" }, { at: "2026-11-05T15:15", repo: "shop", command: "cat /tmp/run-3301/out.log", error: "No such file: /tmp/run-3301/out.log" }, { at: "2026-11-05T17:45", repo: "blog", command: "cat /tmp/run-6610/out.log", error: "No such file: /tmp/run-6610/out.log" },];
// A fix, with when it landed and where.const register = [{ kind: "No such file: <path>", repo: "shop", fixedAt: "2026-11-03T00:00", fix: "write logs inside the repo" }];
// Paths, numbers and quoted values become placeholders, so one cause gives one kind.const kind = (f: Failure) => f.error.replace(/\/[\w.\/-]+/g, "<path>").replace(/\b\d+\b/g, "<n>").replace(/"[^"]*"/g, "<value>");
const ranked = new Map<string, { counted: number; suppressed: number }>();const regressions: string[] = [];for (const f of log) { const k = kind(f); const row = ranked.get(k) ?? { counted: 0, suppressed: 0 }; const fix = register.find((r) => r.kind === k && r.repo === f.repo); if (fix && f.at < fix.fixedAt) row.suppressed++; // before the fix: shown, not ranked else row.counted++; if (fix && f.at >= fix.fixedAt) regressions.push(`${f.at} ${f.repo}: "${k}" came back after "${fix.fix}"`); ranked.set(k, row);}
for (const [k, r] of [...ranked].sort((a, b) => b[1].counted - a[1].counted)) console.log(`${String(r.counted).padStart(2)} ${k}${r.suppressed ? ` (${r.suppressed} before its fix, not counted)` : ""}`);console.log(regressions.length ? `\nRegressions:\n ${regressions.join("\n ")}` : "\nNo regressions.");Run with Bun, a JavaScript runtime, it prints:
3 No such file: <path> (3 before its fix, not counted) 2 npm: command not found 1 rejected: fetch first
Regressions: 2026-11-05T15:15 shop: "No such file: <path>" came back after "write logs inside the repo"The missing-file kind still ranks first, but on three records, not six: the three from before its fix are shown and not counted. Two of the three that count come from the second codebase, which never got the fix, so they are not regressions. The one that is a regression is named.
Where the plain form falls short
The plain form is a list of failed commands, counted. I've seen it go wrong in these ways.
- The top item is already fixed. A ranking once led with a problem whose fix had merged less than a day before. Old records kept it at the top. Recording when each fix landed takes them out of the count without deleting them.
- Things that are not mistakes are counted. A search that finds nothing exits with an error code. So can the last step of a batch that already did its work. Leave those in and they top the list.
- A fix in one codebase is judged in another. If the log is shared across a machine, a fix has to say where it applies, or records from codebases that never got it look like regressions.
- The model is asked to keep the log. It forgets, and each entry costs tokens, the units of text a model is billed for. A script reading the transcript does neither.
- Sub-agents are left out. Their transcripts need reading too, since that is where most of the failed commands were.
Trade-offs
- It needs volume. The figures above come from months of sessions in one codebase. With one agent and a few sessions a week there may be nothing worth ranking.
- The grouping can be wrong. Every rule for reducing errors to kinds will put some record under the wrong kind, and then the ranking points at the wrong thing. The log needs as much care as the code it measures.
- A count is not a cause. It tells you what failed and how often. Why an agent reached for that command still takes a person reading a few examples.
When to use it, and when not to
It suits anyone running many agent sessions over weeks, especially with sub-agents. The cheapest fixes it leads to are often tiny, such as a hook that rewrites one known bad command into a working one. It is not worth building for occasional use, where reading the odd transcript will tell you more.
Example. The ranking and the register, on GitHub
A draft skill for this pattern
A skill is a short set of instructions you give a coding agent so it applies a pattern the way you would. Copy the text below into a file called SKILL.md, in the folder your coding agent reads skills from. In Claude Code, Anthropic's coding agent, that is .claude/skills/failure-register/. It is a draft, and I haven't tried it on anyone else's project yet.
---name: failure-registerdescription: Use when coding agents keep hitting the same failed commands across sessions, or when deciding which agent mistake to fix next. Logs failed commands with a script, groups them into kinds, and reports a failure that returns after its fix as a regression.---
# Failure register
An agent that runs a failing command reads the error, tries another way andcarries on. The task finishes, so the failure is invisible unless somethingrecords it. Record it with a script, not the model.
## Capture
1. When a session ends, a hook (a script the agent tool runs at set moments) reads the transcript and appends each failed command to a log, one JSON line each: time, repository, command, error. Include sub-agents, the helpers a main agent starts. That is often where most failures are, and the main agent never sees them.2. The hook must cost no model calls, never block the session, and stay silent if it breaks.
## Group and rank
1. Reduce each error to a kind: replace paths, numbers, ids and quoted values with placeholders, so one cause gives one kind.2. Rank kinds by count. Read a few examples of the top kind before acting. The count says what failed, not why.3. Check the grouping. A search that finds nothing and exits with an error is not a failure. Remove kinds like that before you rank.
## The register
1. When you fix a kind, add a row: the kind, the repository, the time the fix landed, and what the fix was.2. Failures of that kind from before the fix stop counting in the ranking. Show them as suppressed; do not delete them.3. A failure of that kind after the fix, in the same repository, is a regression. Report it as "the fix did not take".4. A failure in a repository the fix never reached is not a regression.
## Turning kinds into fixes
The cheapest fix is often a hook before each command that rewrites a knownbad form into a working one. Log each rewrite, so you can see what it absorbs.If a hook that rewrites or blocks commands cannot tell whether a commandis a known bad form, have it let the command through.
## What it does not do
It needs volume. With one agent and a few sessions a week there may benothing worth ranking. The grouping can put a failure under the wrong kind,and when it does the ranking points at the wrong thing.