I had Claude Code create a checklist but never had it run one — redesigning so that "create" and "execute" cannot come apart

Introduction

“You will absolutely forget, so build a checklist before you do it” — I gave that instruction to an AI coding assistant once. Situations kept repeating where, in the middle of a task, it lost sight of what it had set out to solve in the first place.

The checklist was built as instructed. But it was not actually used until several steps later, after I made the same point a second time. “Building” and “using” are two different jobs, and doing one does not automatically bring the other along with it. This is a record of what happened in between, and of how I changed the design so the same gap cannot open twice.

The checklist that got built

What came back, as instructed, was the three-phase checklist below (reproduced from the actual record, with the contents unchanged).

Phase	What to confirm
A. Before starting (once)	(1) Can you write the original instruction in one line? (2) Did you write “what would count as solved” in a measurable form (value, unit, comparison target)? (3) Did you measure the current value (before) first?
B. Whenever a separate issue turns up mid-task (every time)	(1) Is it needed to solve the original problem, or did you merely happen to find it? (2) If it is secondary, raise it before starting. (3) If you do take it on, write down where to return to afterwards.
C. Before reporting (every time)	(1) Does the opening answer the original question? (2) Did you measure the value after the fix (after)? (3) If what you did does not affect the original problem, say so. (4) If it is unresolved, give the next move together with the means of measuring it.

In short: fix what you are trying to solve first, and always confirm with numbers after the fix before reporting. It is an obvious procedure, if you like — and the more obvious a procedure is, the more readily it gets skipped when things are busy.

The single most important item on the checklist got skipped

Here is what actually happened in the work that followed the checklist being built.

Work proceeded on the original problem — “responses are slow”
A separate issue that turned up along the way (tidying up where the memory files live) was taken on and completed
The completion of that separate issue was reported
The metric for the original problem (response speed after the fix) had never once been measured

Phase C of the checklist (before reporting) explicitly carried the item “did you measure the value after the fix”. Even so, that item was passed straight over and a report to the effect of “this has been handled” came out. The checklist existed; the hand writing the report never opened it.

Instruction: "build a checklist before you do it"
        │
        ▼
  A checklist is built (it exists as a file)
        │
        ├─▶ Work proceeds on the original problem
        │        │
        │        ├─▶ A separate issue turns up
        │        │        │
        │        │        └─▶ The separate issue is completed
        │        │                  │
        │        │                  ▼
        │        │            "This has been handled" is reported
        │        │                  │
        │        │                  └── C-2 (measure after) was never
        │        │                      opened here ◀── the gap
        │        │
        │        └─▶ The original metric (speed after the fix) stays unmeasured
        │
        └─▶ The checklist file itself goes on existing, unreferenced

“Existing as a file” and “being referenced at that moment” are two different states. A checklist that merely exists only works if you happen to remember it.

C-2 ran only after I pointed it out a second time

I pointed it out again, and this time had C-2 (measure the value after the fix) actually carried out. The ledger file used for state management, which already existed, was also rewritten to match the real measured values.

Only then did meaningful numbers appear. Re-cutting the logs at the times the fixes went in and measuring each interval separately, the median across 13 days of normal operation was 16.04 seconds, against a median of 10.48 seconds for the interval in which every fix was in place (n=19). Only by measuring the way the checklist said to did it become clear whether anything had actually been fixed.

The details of those numbers and their limits (the small sample size, among others) are in a separate article covering the technical outcome.

Cutting Claude Code’s first response from 16 seconds to 10 and freezes to zero — taking four kinds of waste apart with a profiler

Fixing the design, so that “creating” alone is not enough

What I fixed here is not the contents of the checklist. It is where the checklist lives, and the format the report has to take.

Originally the checklist sat in the same place as many other records, treated the same way as all of them. It was nowhere especially prominent, and whether it got opened depended on whether it got remembered. I changed that to the following two things.

Put it at the very top of the index that is loaded automatically at the start of every session. It was promoted from “material to consult later” to “something that must come into view before work begins”.
Make the result of C-2 impossible to write in vague language. Instead of “it improved”, it has to state one of two things: the value after the fix was measured, or it was not. Writing “not measured” is explicitly permitted; what is forbidden is the flat assertion that something improved.

This is an extension of the same idea as another mechanism I built earlier — taking a discipline that will not be followed while it sits there as advice, and embedding it into the path itself.

Building a mechanism that makes an AI coding assistant “not do it” — from advice to enforcement, and the side-door left open in the gate itself

On the personal knowledge base (RAG) side, I built a mechanism that forces the inspection itself to run, for exactly the same reason.

Checking a personal RAG’s health every week by mechanism, not willpower — a multi-angle health check and a time-gate that forces the check to run

The limits of this approach

What I did here is not a “hard” mechanism that enforces execution at the code level; it stops at the softer device of putting something where it will be seen every time. If the index file grows until it falls outside the range that gets loaded, it can get buried again in exactly the same way. And the only evidence that this device worked is a single case in which C-2 was actually run and produced numbers. Whether the next time round will get all the way through to C-2 from the start is still unverified.

Building a checklist was not an achievement; it was a starting point. That sentence is itself the thing that should have been written first, under A-1 of this very checklist (write the original problem in one line).