Checks you can trust
A check you have never seen fail is not a check
Test-driven development's first habit, watching a test fail before you trust it, applied to every other automated check a codebase relies on: lint rules, jobs in the build pipeline, scheduled jobs, and the checks agents write.
One of 15 patterns I've found working with AI coding agents. An agent drafted the words from my notes, and I haven't edited them yet. See all 15, or read how they are made.
An agent drafted the words from my notes, and I haven't edited them yet. The code was run before it went in.
The problem
If you write tests first, you know the rule already. In test-driven development, TDD for short, you write a test, watch it fail, then write the code that makes it pass. Seeing it fail is what tells you the test can catch anything at all.
Most codebases have far more checks than unit tests, though, and the habit rarely reaches them. A lint rule, which reads the source for a known mistake. A job in the build pipeline that runs on every change, in GitHub Actions or similar. A script that checks layout, colour contrast or a list against the code. A job that runs each night. Each one guards the codebase, and in my experience few of them have ever been seen to fail.
That matters because a green tick says one of two things. The check looked and found nothing, or the check did not look. From the outside they are the same tick, and most ways a check can break make it pass. A scan that finds no files reports no problems. A skipped step is not a failed step.
With coding agents this matters more. In my experience an agent reads green as done and moves on. In my projects it also writes most of the new checks, and it applies TDD to them only when told to. A check that passes on its first run looks finished.
The idea
Treat every check the way TDD treats a test. Run it on the code as it is, and it should pass. Add one mistake it exists to catch, and it should fail. Take the mistake out, and it should pass again. A check that manages all three is proven. Anything else is unproven, with the reason. Mutation testing does something similar for unit tests at scale; this is the same instinct aimed at the checks around the code.
The first step matters more than it looks. A check that is already failing will also fail after you add the mistake, so it seems to prove itself. Checking for green first closes that.
A version small enough to read
The check being proved is deliberately dull: no file in a folder may contain the word TODO.
// The check being proved: no file under the folder may contain "TODO".// Usage: bun check.ts <folder> [--rule-off]import { existsSync, readdirSync, readFileSync } from "node:fs";import { join } from "node:path";
const [dir = "src", flag] = process.argv.slice(2);if (flag === "--rule-off") process.exit(0); // the rule switched off in config
const files = existsSync(dir) ? readdirSync(dir).filter((f) => f.endsWith(".ts")) : [];const bad = files.filter((f) => readFileSync(join(dir, f), "utf8").includes("TODO"));if (bad.length > 0) { console.error(`TODO left in: ${bad.join(", ")}`); process.exit(1);}console.log(`checked ${files.length} file(s), none with TODO`);A second script, which I call the prover, runs five versions of it in a throwaway folder. One is the check as written. Three are the same check with a fault I've seen in real codebases: pointed at a folder that does not exist, piped through another command, and with its rule switched off. The last one is already failing before the mistake goes in.
// Proves each check can fail: green first, red with a known mistake, green again.import { mkdtempSync, mkdirSync, rmSync, writeFileSync } from "node:fs";import { tmpdir } from "node:os";import { join } from "node:path";
const check = join(import.meta.dir, "check.ts");const gates: Record<string, string> = { "the check as written": `bun ${check} src`, "pointed at a folder that does not exist": `bun ${check} source`, "piped through tail": `bun ${check} src | tail -1`, "rule switched off": `bun ${check} src --rule-off`, "already failing": `bun ${check} src && exit 1`,};
const passes = (cmd: string, cwd: string) => Bun.spawnSync(["sh", "-c", cmd], { cwd, stdout: "ignore", stderr: "ignore" }).exitCode === 0;
for (const [name, cmd] of Object.entries(gates)) { const dir = mkdtempSync(join(tmpdir(), "prove-")); mkdirSync(join(dir, "src")); writeFileSync(join(dir, "src/app.ts"), "export const answer = 42;\n"); const mistake = join(dir, "src/new.ts");
let verdict = "proven"; if (!passes(cmd, dir)) verdict = "unproven: it was already failing"; else { writeFileSync(mistake, "// TODO: finish this\n"); if (passes(cmd, dir)) verdict = "unproven: it passed with the mistake in"; else { rmSync(mistake); if (!passes(cmd, dir)) verdict = "unproven: it stayed red after the mistake was removed"; } } console.log(`${name.padEnd(40)} ${verdict}`); rmSync(dir, { recursive: true });}Run with Bun, a JavaScript runtime, it prints:
the check as written provenpointed at a folder that does not exist unproven: it passed with the mistake inpiped through tail unproven: it passed with the mistake inrule switched off unproven: it passed with the mistake inalready failing unproven: it was already failingThe piped one is the one I'd watch for. A shell reports the exit code, the number a command returns to say whether it succeeded, of the last command in a pipe, so a failing check passed through tail, which only trims the output, reports tail's success.
Ways a check passes without looking
Each of these is from a real codebase. Each one showed green until someone looked.
- It measures the wrong surface. A colour contrast check compared text with a flat card background. The real card had a tinted wash over it. The check reported a pass, with a figure, for a pair that still failed.
- It runs where the mistake cannot show. A check for regions that scroll sideways ran at a screen width where the table never scrolls.
- Rules are skipped without a word. A filter by tag left some rules out. The check's own output said 70 matched and 63 ran, for weeks.
- A cached pass is replayed. A change that touched only documents showed green without the check running at all.
- It is not on the path that can refuse. The tests existed, but not among the checks a change has to pass before it can be merged. Changes merged on a green tick that had not looked.
- A scheduled job succeeds without doing its work. Every step was skipped, and the run showed green. The answer is to name the step that should have run, and show a run without it as empty, not green.
- The proof was wrong. A tool that proved checks by breaking them counted red after the break as proof. A check that was red before it started, or whose command did not exist, proved itself.
Trade-offs
- Someone has to invent the mistake. For most checks that takes a minute. For a backup or a deploy, the break is awkward to stage.
- A proof goes stale. The check was seen failing on one date, and the code around it has moved since. Re-prove a check when you change it.
- One mistake is not every mistake. The injected break shows the check can fire. It does not show it fires for every form of the problem.
- The prover is one more check. It can be wrong too, which is why it checks for green first and why it has tests of its own.
When to use it, and when not to
Use it on every check that can stop a change being merged, and on any check that has only ever passed. It is not worth the effort for a throwaway script you will run once and read yourself.
A fuller example
The small version on this page is linked at the foot, in a GitHub repository of runnable examples. The same repository holds the example for the ratchet, another pattern here, for switching on a rule over code that already breaks it. It has a fuller prover, with tests, that proves its check and marks four faulty checks unproven.
Example. The small prover, on GitHub
A draft skill for this pattern
A skill is a short set of instructions you give a coding agent so it applies a pattern the way you would. Copy the text below into a file called SKILL.md, in the folder your coding agent reads skills from. In Claude Code, Anthropic's coding agent, that is .claude/skills/prove-a-check/. It is a draft, and I haven't tried it on anyone else's project yet.
---name: prove-a-checkdescription: Use when adding or changing an automated check other than a unit test written test-first (a lint rule, a build pipeline job, a scheduled job, a custom script that can block a change), or when a check has only ever passed. The red step of test-driven development, applied to every check.---
# Prove a check
A passing check has either looked and found nothing, or not looked at all, andthe two look the same. Test-driven development answers this for unit tests bywatching each test fail first. Do the same for every other check: lint rules,build pipeline jobs, scheduled jobs and scripts. Before you trust a check,watch it fail.
## For every new or changed check
1. Run it on the code as it is. It must pass. If it already fails, stop: the result is "unproven", not "proven", because a check that is already red cannot show you that your mistake turned it red.2. Add one mistake the check exists to catch. Run it. It must fail, with a message that names the mistake.3. Take the mistake out. Run it again. It must pass.4. Put the failure message, word for word, in the description of the change.
Do this in a throwaway copy of the code, never in the copy you are working in.
## Things that make a check pass without looking
Look for each of these when a check seems too quiet:
- It reads a folder or a list of files that does not exist, or is empty. Have it print how many files it read, and fail when the number is zero.- Its output goes through a pipe (`check | tail`). The pipe reports the last command's success. Run checks bare and read their own exit code.- A rule it relies on is switched off in a config file.- It runs where the mistake cannot show: the wrong screen width, the wrong sample data, the wrong surface for a measurement.- A cache replays an old pass.- It is not in the job that can actually block a merge.- A scheduled job's steps were all skipped and the run still shows success. Name the step that must have run.
## Three answers, not two
A check reports pass, fail, or could not check. "Could not check" fails thebuild until someone looks.
## What it does not do
One injected mistake shows the check can fire. It does not show it fires forevery form of the mistake. A proof also ages: the code around the check movesafter the day you saw it fail. Re-prove a check when you change it.