Running agents
Software that hunts its own bugs overnight
An AI coding agent, a model that reads and writes code from instructions, reads each day's changes to a codebase overnight and looks for bugs a tester would notice. Most of the work is in what you refuse to let through.
One of 15 patterns I've found working with AI coding agents. An agent drafted the words from my notes, and I haven't edited them yet. See all 15, or read how they are made.
An agent drafted the words from my notes, and I haven't edited them yet. The code was run before it went in.
The problem
Tests only catch what someone thought to write a test for. With coding agents writing much of the code, more changes land in a day than a person can try by hand, and the bugs a tester would find in ten minutes go unseen for weeks.
An agent can read the changes and go looking. Left to itself, though, I've found it files findings that sound right and are not, and I soon stopped reading them. There is a quieter problem too. A night with no findings and a night when the job never ran look the same in the morning.
The idea
Split the night into two agents and some plain code.
A script lists what changed since the last run, with each diff, the record of exactly what a change added and removed. That list is the brief. A finder agent reads it, asks the same few questions of every change, and writes its findings to one file in a fixed shape, each with a file and line. Then a refuter, a second agent in a session of its own so it starts without the finder's reasoning, tries to disprove each finding. Only what it fails to disprove reaches the morning report.
The code around the agents decides what a night means. If the finder wrote no file, the night is reported as broken and the run is marked failed, so the next night reads the same changes again. An empty list in a file that exists is a quiet night, and a good one.
A version small enough to read
This is the shape I use, cut down to one night over a sample codebase with one change in it. The finder's instructions are a plain file. They ask three questions of every change, give the exact shape of the file to write, and rule out sub-agents, which in my setup once cost a whole night: the finder handed the work to helpers and finished before they came back.
You are the FINDER in a nightly bug hunt. You look for bugs a user would hitin the changes that reached the main branch since the last run, from thediffs and the code alone. Nothing you write is filed anywhere; a person readsthe report in the morning.
1. Read `brief.md` first. It lists every change since the last run, with its diff under `diffs/`.2. For every change, read its diff and then the changed files themselves.3. For each changed piece, answer three questions: - States: which states (empty, error, partial data, long lists) give nothing, or the wrong thing? - Values: does anything compute a wrong value for an input a user would give? - Edges: what happens on a second click, a stale id, a missing value?4. Skip anything the brief lists under "Last run".
Write `findings.json` with the Write tool, exactly this shape:
{ "screened": [<the name of every change whose diff you read>], "findings": [ { "title": "<one line, as a tester would title a bug>", "severity": "crash" | "broken" | "confusing" | "cosmetic", "steps": ["<step a user takes>", "..."], "expected": "<what should happen>", "seen": "<what the code would do instead>", "evidence": { "file": "<path>", "line": <line number> }, "confidence": "high" | "medium" | "low" } ]}
Every finding has evidence.file and evidence.line. An empty "findings" listis a valid, good result.
Constraints:- Do the whole pass yourself. Never use a sub-agent: its work is lost the moment your turn ends, and the run then has no file.- Write `findings.json` after the first change you screen, and rewrite it after every change, so a run cut short still has a file.- Always write `findings.json`, even when you find nothing.- When you near your turn limit, stop reading and write the file.- Never edit any other file.The refuter's instructions start from the view that the claim is wrong, and ask for the guard, the code elsewhere that would stop the bug.
You are the REFUTER in a nightly bug hunt. Another session, which you cannotsee, claims to have found the bug below. Your job is to disprove it.
Start from the view that the claim is wrong. Read the file and line it cites,then look for what would stop the bug from reaching a user: a check earlierin the same function, a caller that never passes the bad input, a loaderthat filters it out, a type that rules it out. Follow the code; do not trustthe claim's description of it.
- If you find the guard, the verdict is "refuted", and "why" names the file and line of the guard.- If you follow every path a user could take and the bug is still there, the verdict is "confirmed", and "why" gives the path in one or two sentences.- If you cannot tell from the code alone, the verdict is "could not try", and "why" says what you would need.
Reply with the verdict and the reason only.The night's change adds a currency to an export. It has one real bug, and one thing that looks like a bug but is guarded elsewhere. The second is that the new code assumes every total is present. The code that loads the rows already leaves out the ones that are not.
import type { Row } from "./load";
// Rows whose date is between from and to, inclusive, one line each, with the currency.export function exportRows(rows: Row[], from: string, to: string, currency = "GBP"): string[] { const kept = rows.filter((r) => r.date >= from && r.date <= from); return kept.map((r) => `${r.date},${r.total!.toFixed(2)} ${currency}`);}A script builds the brief and calls both agents through the command line of Claude Code, Anthropic's coding agent, each in a new process, without the settings or add-ons of the machine it runs on.
// One night of a bug hunt over a sample repository: build the brief from what changed,// let a finder read it, let a refuter try to disprove each finding, write the morning report.// Run: bun hunt.ts. Needs git and the claude CLI, logged in. Costs well under a dollar.import { cpSync, existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";import { tmpdir } from "node:os";import { join } from "node:path";
const MODEL = process.env.HUNT_MODEL ?? "claude-sonnet-5-5";const here = import.meta.dir;const repo = mkdtempSync(join(tmpdir(), "hunt-"));const git = (...a: string[]) => Bun.spawnSync(["git", "-c", "user.name=hunt", "-c", "user.email=hunt@example.invalid", ...a], { cwd: repo }).stdout.toString().trim();
// The sample repository: yesterday's code, then the one change that reached main since.git("init", "-q");cpSync(join(here, "sample/before"), repo, { recursive: true });git("add", "."); git("commit", "-qm", "Start");const since = git("rev-parse", "HEAD");cpSync(join(here, "sample/after"), repo, { recursive: true });git("commit", "-qam", "Show the currency in the export");
// The brief: every change since the last run, each with its diff. Code, not a model.mkdirSync(join(repo, "diffs"));const changes = git("log", "--format=%h %s", `${since}..HEAD`).split("\n");for (const c of changes) writeFileSync(join(repo, "diffs", `${c.split(" ")[0]}.diff`), git("show", c.split(" ")[0]!));writeFileSync(join(repo, "brief.md"), `# What changed\n\n${changes.map((c) => `- ${c} (diffs/${c.split(" ")[0]}.diff)`).join("\n")}\n\n# Last run\n\nNothing.\n`);
// Each call is a new process with no settings, skills or hooks of the machine it runs on.function claude(system: string, prompt: string, tools: string, extra: string[] = []) { const r = Bun.spawnSync(["claude", "-p", "--model", MODEL, "--safe-mode", "--setting-sources", "", "--no-session-persistence", "--allowed-tools", tools, "--disallowed-tools", "Agent,Task", "--system-prompt", system, "--output-format", "json", ...extra], { cwd: repo, stdin: new TextEncoder().encode(prompt), }); return JSON.parse(r.stdout.toString() || "{}");}
claude(readFileSync(join(here, "finder.md"), "utf8"), "Run tonight's pass.", "Read,Glob,Grep,Write", ["--max-turns", "30"]);
// A night with no findings file is broken, not quiet. Say so, and fail.const file = join(repo, "findings.json");if (!existsSync(file)) { console.log("BROKEN: the finder wrote no findings.json, so nothing is known about this night."); process.exit(1);}type Finding = { title: string; severity: string; evidence: { file: string; line: number } };const { screened, findings } = JSON.parse(readFileSync(file, "utf8")) as { screened: string[]; findings: Finding[] };console.log(`Read ${screened.length} of ${changes.length} change(s). The finder raised ${findings.length} finding(s).`);if (findings.length === 0) console.log("A quiet night: everything that changed was read, and nothing was found.");
// The refuter: a fresh session per finding, which never sees the finder's reasoning.const VERDICT = { type: "object", properties: { verdict: { enum: ["confirmed", "refuted", "could not try"] }, why: { type: "string" } }, required: ["verdict", "why"] };for (const f of findings) { const claim = `${f.title}\nSeverity: ${f.severity}\nEvidence: ${f.evidence.file}:${f.evidence.line}`; const { verdict, why } = claude(readFileSync(join(here, "refuter.md"), "utf8"), claim, "Read,Glob,Grep", ["--json-schema", JSON.stringify(VERDICT)]).structured_output ?? { verdict: "could not try", why: "no reply" }; const label = verdict === "confirmed" ? "REPORT" : verdict === "refuted" ? "dropped" : "held"; console.log(`\n${label}: ${f.title} (${f.severity}, ${f.evidence.file}:${f.evidence.line})\n refuter, ${verdict}: ${why}`);}rmSync(repo, { recursive: true, force: true });Run with Bun, a JavaScript runtime, one real night printed:
Read 1 of 1 change(s). The finder raised 1 finding(s).
REPORT: Export only includes rows from the start date, ignoring the end of the range (broken, export.ts:5) refuter, confirmed: export.ts:5 filters with `r.date >= from && r.date <= from`. The upper bound uses `from` instead of `to`, and `to` is never used. Any call to exportRows keeps only rows dated exactly `from`, so the rest of the range is dropped. The comment above the function says the range should be inclusive of `to`.The finder found the real bug and left the guarded one alone, and the refuter confirmed it by reading the code for itself. On a real codebase the refuter matters more. In one project, a similar hunt that looks at screenshots filed twelve findings at first, and five of them contradicted a test that was passing.
Where the plain form falls short
The plain form is an agent told to find bugs and file them. These are the gaps I've seen.
- A plausible finding with nothing behind it. "The date picker feels slow" costs a person ten minutes to dismiss. A finding has to name a file and a line.
- The finder marks its own work. Asking the same agent whether it is sure tends to get a yes. The refuter gets the finding and the code, not the reasoning that produced it.
- A silent night reads as a clean one. A night that wrote no file is reported as a failed night, and the run is marked failed.
- A rejected finding comes back. When a person closes a finding, the reason is recorded where the next night's hunt can read it.
- The report becomes a ticket that sits unfixed. Filing is the easy half. I run versions of this in a few projects, and only one of them goes on to open a fix by itself.
Trade-offs
- It costs model time every night, twice over, because of the second agent. Start with one area of the product.
- The refuter can be wrong both ways. It will sometimes throw out a real bug. Count how many filed findings turn out to be real, and stop widening the hunt if that number falls.
- It reads code, not the running product. Some bugs only show when the app runs. Mark those as needing a person, or a second kind of hunt that drives the app.
When to use it, and when not to
It suits a product that changes daily, where more lands than a person can try by hand. It is not worth running when someone can try every change themselves.
Example. The morning report, on GitHub
A draft skill for this pattern
A skill is a short set of instructions you give a coding agent so it applies a pattern the way you would. Copy the text below into a file called SKILL.md, in the folder your coding agent reads skills from. In Claude Code, Anthropic's coding agent, that is .claude/skills/overnight-hunt/. It is a draft, and I haven't tried it on anyone else's project yet.
---name: overnight-huntdescription: Use when more changes land each day than anyone can try by hand, and you want an agent to look for bugs a tester would notice, overnight, without filing noise. A finder reads the day's changes, a separate refuter tries to disprove each finding, and a night with no findings file is reported as broken, not quiet.---
# Overnight hunt
Tests catch what someone thought to test. An agent can read the day'schanges and go looking for the rest, but an unchecked agent files plausiblenoise. The work is in what you let through.
## Before you start
Write two prompts and keep them in files: one for the finder, one for therefuter. The finder's prompt asks fixed questions of every change (whichstates give nothing or the wrong thing, which inputs give a wrong value,what happens on a second click or a missing value) and gives the exact shapeof the file it writes. The refuter's prompt starts from the view that theclaim is wrong.
## Build it
1. Each night, a script (not a model) builds the brief: every change merged since the last run, with its diff, and the findings already raised.2. The finder runs headless, with read tools and permission to write one file, a turn limit, and no sub-agents. A sub-agent's work is lost when the finder's turn ends, so the night ends with no file.3. Every finding names a file and a line. Tell the finder to write its file after the first change it reads and after every one since, so a run cut short still leaves something, and to write it even when it finds nothing.4. If there is no findings file, the night is broken: say so and mark the run failed, so the next night reads the same changes again. An empty list in a file that exists is a quiet night, and is a good result.5. For each finding, start the refuter in a new session that has never seen the finder's reasoning. Give it the claim and the code. It looks for the guard that would stop the bug (an earlier check, a caller, a loader, a type) and returns confirmed, refuted with the guard's file and line, or could not try.6. Report only what was confirmed. To begin with, file nothing: a person reads the report each morning. File automatically only once most reported findings turn out real, with a cap on how many a night can file.7. Check the severity of what survives. A real bug can be reported as worse than it is, and the refuter's reading is often the one that narrows it.
## When a finding is wrong
Say why when you close it, in a form the hunt can read next time. A rejectedfinding that is raised again is a cost.
## What it does not do
It finds what an agent can see from the code and the rules. A disproofagent can be wrong in either direction. Count how many filed findings werereal, and stop widening the hunt if the number falls.