ferlogs

~/writing/close-the-loop

·Agents·9 min read·Fernando Valdez

Close the loop: how agents should learn from their mistakes

The model does not learn from your week. A hook captures the miss, a cheap model proposes a few lessons, a person approves once a week, and an eval is the gate.

The model does not learn from your week. The next session starts empty. It only knows what you put in the files it loads. So "the agent should get better" is not a training job. It is a change to those files, and a check that the change did not break the last thing that worked.

That sounds obvious until you watch the usual fix. Someone pastes another paragraph into a giant instruction file. The paragraph is a guess. Next Tuesday the same mistake happens, and the file is longer. A bad lesson that lands in a shared standard gets repeated by every session on the team, which is worse than the original mistake, and it is free.

What counts as a lesson

A lesson is worth keeping when someone else, in another session, would use it. A tool that fails the same way every time. A correction you have had to type twice. A detour that works. Something you had to go and find out, that the next person should not have to find out again.

A personal preference is not a lesson. Neither is something the model already does on its own. If you cannot point at the session where it would have helped, it does not go in.

Five steps, and only one of them is a person

I run it as a loop. The machine captures, proposes and checks. A person approves, on a short clock, because that is the step you cannot automate without teaching the team the wrong thing.

  1. Capture. On the machine where the work happens, a hook records two things and then stops: a tool that failed, and the end of a session. No network and no model in that hook. It has to finish in a couple of seconds or people will turn it off the first afternoon it gets in the way.
  2. Extract. Once a day, a cheap model reads those transcripts and proposes a few lessons. Three is plenty. It does not edit the standards. It writes a proposal, and it strips secrets, emails, customer names and anything that identifies a person before the proposal exists.
  3. Propose. Once a week the inbox is deduped and turned into a small change. One lesson at a time. Each change carries the eval case that should fail if you revert it.
  4. Approve. A person, on a rota, spends one short sitting on it. I budget fifteen minutes. Accept, edit, or throw it out. Fifteen minutes is only possible if the pile is small, which is why the cap of three per session exists.
  5. Protect. CI runs the evals. No case, no merge. A lesson that cannot be checked is an opinion, and opinions do not belong in a file every future session will trust.
The learning loop for an agent A hook captures failures locally. A cheap model extracts a few lessons. A weekly pass proposes one change plus an eval. A person approves. CI runs the evals. Only then do the files the next session loads change. The loop returns to the next session. 1 · capture · hook, on the laptop 2 · extract · cheap model, 3 max 3 · propose · one change + one eval 4 · a person, 15 minutes discard 5 · evals in CI · no case, no merge stop files the next session loads standards, skills, known cases, or a hook next session
Capture, extract and check are automatic. The person is the gate. The files are the memory.

The transcripts stay on the laptop. What leaves is the lesson, after the redaction, and only if a person let it through. The raw session is not the artifact. The sentence you are willing to put in a shared file is.

Where a lesson is allowed to go

The destination matters more than the wording. I use four, and I do not invent a fifth when I am tired.

  • If an environment will fail the same way again, it goes in a known-cases note. The next session should recognise it, not rediscover it.
  • If it is how the code should be written, it goes in the standards. One concern, one short file, the wrong version next to the right one.
  • If a flow is missing a step, it goes in that skill. Not in a general "be careful" document nobody loads.
  • If it must always be true, it does not stay as text. Text is advice. A hook, a permission, or a linter is a rule, and it holds on the afternoon someone is in a hurry.

Anthropic's notes on Claude Code in large codebases make the same split. A line in CLAUDE.md is a suggestion. The things you cannot afford to have ignored belong in the harness, and someone has to own that config. They also recommend a hook at the end of a session that reflects and proposes edits, rather than editing the file itself. The proposal is the point. The unattended edit is how a bad afternoon becomes policy.

Repetition is the signal

Before proposing anything, look for the same lesson already in the inbox. If it is there, add the evidence and increment a counter. One case seen over and over is heavier than a pile of one-off tips.

The weekly review should be mostly "this again". If it is mostly new ideas, the extractor is being clever and the inbox is noise. Clever is how the file gets longer without the work getting better.

The gate is an eval, not a feeling

A change to a skill or a standard does not merge unless there is a case that would fail without it. Start from real failures. Incidents, corrections, sessions where a person had to step in. Anthropic's writing on evals is blunt about the starting size: a few dozen cases, drawn from what went wrong, including cases where the skill should stay quiet.

Two traps eat the number if you let them.

The judge is too small, so the score flips between runs, and you "fix" a skill that was fine. If the skill clearly fired and the diff still comes out negative, suspect the judge before you suspect the skill. Give the judge a stronger model and a rubric a person could apply without squinting.

The workspace is dirty. In one of Anthropic's own cases the model was reading the answer off the git history of earlier attempts. Each attempt starts clean, or the percentage is fiction. For operations I care that it passes every run, not that it passed once.

What other people have published

Warp runs a scheduled skill that compares what the agent did with how people reacted, and opens small pull requests that cite that evidence. A person still merges. OpenAI has described a recurring doc-gardening pass that opens pull requests for stale documentation, which is the same shape pointed at docs. Anthropic's managed agents include a scheduled look over sessions for repeated errors, with a person in the loop.

None of them retrain the model on your tickets. They change the files the next session will read, and they put a person between the proposal and the merge. That is the whole trick. It is also the part teams skip, because a paragraph in a markdown file feels like learning and a fifteen-minute review feels like process.

The rule

Capture automatically. Propose automatically. Check automatically. A person approves.

If you only do the first three, you will get a confident, growing file full of other people's bad afternoons. If you only do the fourth, you will have a meeting and no memory. The loop is the combination, and the files are the only place the next session can learn.

Where this comes from