← Patterns

Checks you can trust
AI helps us move at lightspeed. There's no time to write everything up. So I've used my personal voice and based this on my actual work, to be able to share in real time what I'm doing with the teams I work with.

Checks you can trust

The ratchet

A way to switch on a rule in a codebase that already breaks it, without fixing everything first and without an exemption list that only grows.

One of 15 patterns I've found working with AI coding agents. An agent drafted the words from my notes, and I haven't edited them yet. See all 15, or read how they are made.

An agent drafted the words from my notes, and I haven't edited them yet. The code was run before it went in.

The problem

You want a new rule. No colour values written outside the theme (the one place an app names its colours), no function above a set complexity, no loading spinners, every written requirement named by at least one test. The codebase already breaks it in, say, four hundred places.

There are two usual ways in. You can fix all four hundred first, which in my experience rarely happens, because it never outranks the next feature. Or you can switch the rule on with an exemption list, a file naming the places allowed to skip it. That works on the first day and tends to be forgotten within a month.

I've found the exemption list wears out faster when coding agents write most of the code. An agent that meets a failing check and an exemption list will often add a line to the list, because that is the cheapest way to make the check pass, and it looks like a fix.

The idea

A ratchet turns one way. Count the violations in each folder today and commit those numbers. I call each one a pin. The check compares the live count with the pin, so new code is held to the rule, and a folder with no pin is held to zero. When someone fixes a violation, the count falls and the pin falls with it, so the lower number becomes the new limit.

Because the pins are small numbers, one per folder, the debt, the existing code that breaks the rule, gets paid down by whoever happens to be editing it. No migration has to be scheduled.

A version small enough to read

The rule here is that a colour such as #2456d6 may only be written in src/theme. Everywhere else a file refers to a colour by name. First the pins, one number per folder:

ratchet/pins.json
{
"src/buttons": 2,
"src/cards": 1
}

Then the check, about forty lines of TypeScript with no dependencies beyond the file system. It runs in Bun, a JavaScript runtime, and any runtime that can run TypeScript would do:

ratchet/check.ts
import { existsSync, readdirSync, readFileSync } from "node:fs";
import { join } from "node:path";
import pins from "./pins.json";
const ROOT = "src";
const ALLOWED = "src/theme"; // the one place colours may be written
const COLOUR = /#[0-9a-fA-F]{3,8}\b/;
// Every folder under src, except the allowed one.
function folders(dir: string): string[] {
const subs = readdirSync(dir, { withFileTypes: true }).filter((e) => e.isDirectory());
return subs.flatMap((e) => {
const path = join(dir, e.name);
return path === ALLOWED ? [] : [path, ...folders(path)];
});
}
// Violations in the files directly inside one folder, not its subfolders.
function count(dir: string): number {
return readdirSync(dir, { withFileTypes: true })
.filter((e) => e.isFile() && e.name.endsWith(".ts"))
.flatMap((e) => readFileSync(join(dir, e.name), "utf8").split("\n"))
.filter((line) => COLOUR.test(line)).length;
}
const found = folders(ROOT);
const problems: string[] = [];
if (found.length === 0) problems.push(`No folders found under ${ROOT}, so nothing was checked.`);
for (const dir of found) {
const live = count(dir);
const pin = (pins as Record<string, number>)[dir] ?? 0; // no pin means zero
if (live > pin) problems.push(`${dir} has ${live} raw colour(s), over its pin of ${pin}. Use a theme colour instead.`);
if (live < pin) problems.push(`${dir} has ${live} raw colour(s), under its pin of ${pin}. Lower the pin to ${live} in pins.json.`);
}
for (const dir of Object.keys(pins)) {
if (!existsSync(dir)) problems.push(`${dir} is pinned but no longer exists. Delete its pin.`);
}
if (problems.length > 0) {
console.error(problems.join("\n"));
process.exit(1);
}
console.log("ratchet ok");

On a small example app with three raw colours, two in src/buttons and one in src/cards, it prints ratchet ok. These four changes each make it stop with a message that says what to do next:

What the check says
A colour added in src/cards:
src/cards has 2 raw colour(s), over its pin of 1. Use a theme colour instead.
A colour removed from src/buttons, pin left at 2:
src/buttons has 1 raw colour(s), under its pin of 2. Lower the pin to 1 in pins.json.
A new folder, src/nav, with one colour in it:
src/nav has 1 raw colour(s), over its pin of 0. Use a theme colour instead.
src/cards deleted, its pin left behind:
src/cards is pinned but no longer exists. Delete its pin.

Seven gaps in a plain ratchet

The plain form is to fail when the count goes up. I think it falls short in these ways, and I've met each of them. The version above closes five of them and leaves two for you.

  • A pin left too high hides new violations. Fix one and leave the pin, and the next new violation passes, because the count is back at the pin. Failing on any difference, as above, means the change that fixes something also lowers the number.
  • A folder excused by name can get worse. An exemption list names the folder without limiting how many violations it may hold, so the number can climb. A pin stores the count, so it cannot.
  • A pin outlives its reason. A pin for a folder that has been deleted or cleaned up is an error, so the file only describes debt that exists.
  • New code inherits an old excuse. If a pin covered a folder and everything under it, a new subfolder would start with the old allowance. Counting only the files directly in a folder gives a new folder no pin, so it starts at zero.
  • A check that looks at nothing passes. If the scan finds no folders, the check says so and stops.
  • The rule gets narrowed. Someone trims the list of folders the scan reads, so the count falls for the wrong reason and no pin is raised. The fix is a second check that the files you expect the scan to reach are still reached. The small version above leaves this one out.
  • Excusing single lines becomes the easy way out. A comment saying this line is fine is itself a loophole. Count those comments too, hold them to a pin, and require a reason on each. The small version above leaves this one out.

Trade-offs

  • One extra edited line per improvement. Failing on any difference means every fix also edits the pins file. A gentler version lets an improvement pass and tightens the pin with a command. I use both in different projects and am not sure yet which I'd recommend by default.
  • Merge conflicts on the numbers. Two people who lower the same pin on separate branches will hit a conflict that git, the version control tool, can't resolve alone. Pinning per folder keeps that rare. One number for the whole codebase would make it common.
  • A crude counter. Matching lines of text is not parsing code, so a violation written in an unusual way can be missed. That is acceptable, because the number only has to move when the code does. When the counter misses something visible, fix the counter and re-measure the pins in the same change.
  • It holds a line and says nothing more. A ratchet can't tell you whether the rule was a good one. It can also be gamed by moving violations to somewhere the counter does not read.

When to use it, and when not to

It suits a rule a script can count, debt too large to clear in one go, and fixes that arrive a few at a time. It is the wrong tool for a rule with a handful of violations, which you should fix and be done with, and for a rule that needs a person to judge each case.

A fuller example

A fuller runnable version, linked at the foot of this page, closes all seven gaps, with a demo that causes each one and a prover that checks a check can fail at all. It needs Bun 1.4.4 or later. Install with bun install, then bun run check:ratchet runs it, bun run demo shows each leak, and bun run prove shows the prover marking four faulty checks unproven.

Example. A ratchet you can run, on GitHub

A draft skill for this pattern

A skill is a short set of instructions you give a coding agent so it applies a pattern the way you would. Copy the text below into a file called SKILL.md, in the folder your coding agent reads skills from. In Claude Code, Anthropic's coding agent, that is .claude/skills/ratchet/. It is a draft, and I haven't tried it on anyone else's project yet.

ratchet/SKILL.md
---
name: ratchet
description: Use when switching on a lint, test or style rule over a codebase that already breaks it. Adds a count per folder that may only go down, in place of an exemption list.
---
# Ratchet
A ratchet holds a rule over existing debt. Record how many times each folder
breaks the rule today (a pin), fail the check if a folder differs from its pin,
and hold anything without a pin to zero. Fixes lower the numbers and the new
numbers stay. Do not use an exemption list for this: an exemption names a place,
not how bad the place is allowed to get.
## Before you start
1. The rule must be countable by a script. If a person has to judge each case,
this is the wrong tool.
2. Pick the smallest scope that makes sense, usually one folder. Pins that
cover a folder and everything under it hand a new subfolder the old excuse.
3. Check how many violations there are. If it is a handful, fix them and
switch the rule on with no pins.
## Build it
1. Write the counter: given a folder, return the number of violations in the
files directly inside it, not in its subfolders. A crude counter, such as
matching lines, is fine. It only has to move when the code does.
2. Write the pins file: one number per folder. Add a written reason beside each if you can.
3. Write the check. It compares the live count with the pin and fails on
any difference, not only when the count is higher. A change that removes a
violation then has to lower the pin in the same diff. A folder with no pin
is held to zero.
4. Make the check fail on a pin whose folder is gone and on a scan that
finds no folders at all. If you record reasons, fail on a pin with none.
5. Write every message as the next step, with the number: "Lower the pin to 1
in pins.json". Whoever reads it may be an agent with nothing else to go on.
6. Run the check as part of the build, so a failing check stops the change.
## Prove it can fail
Add one known violation to a folder, run the check, and confirm it fails.
Remove it and confirm the check passes. A check that has only ever passed
has not been shown to look at anything.
## When the check goes red
Fix the code. Do not raise a pin to make the check pass. A count that goes up
is a question about the code or the counter, not a new allowance. If the
counter is the problem, fix the counter and re-measure every pin in the same
change.
If you have to raise a pin, say why in the pins file, in a dated line.
## What it does not do
It does not say whether the rule was a good one. It does not stop the code
getting worse in a way the counter cannot see, such as moving violations to a
folder the scan does not read. Keep the scan's reach written down, and check
that the files you expect it to cover are still covered.