Runs start from nothing: no way to reuse what earlier runs of the same intent learned (needs a success signal Chronicle does not have) #203

Status: design question, not a request to implement. The idea may be a dead end for agents without a success signal (see Caveats). This issue exists to decide whether it is worth an ADR, and for which agents.

Context

TokenOps governs an agent run from inside its process. A policy watches the steps of a run and can act: halt, or inject a message into the agent's next call (ActionKind.INJECT, src/tokenops/framework/core.py:215). Today the one policy that injects, progress_guard, sends a fixed generic nudge when it sees a stall (src/tokenops/framework/policies/progress_guard.py:166: "Change approach or finalize with what you have").

Chronicle records each run as boundary crossings (an LLM call or a tool call) with the raw input and output, and saves them to the control plane, which stores and queries them. Chronicle keeps nothing locally. It records what happened; it has no notion of whether the run succeeded.

Problem

Agents deployed in production tend to see a few intents far more often than the rest. Each new run starts from nothing, so it can repeat the same expensive mistakes that earlier runs already made and recovered from, or never recovered from.

Evidence from a benchmark (Terminal Bench 2.0, a terminal-only agent, three runs of the same 25 tasks, about 150 trials per arm pooled):

• A trial sat in the less pager for 24 agent steps after a git log, trying a different command each time. The same screen came back 19 times. The lesson ("set GIT_PAGER=cat first") is short and specific. The generic nudge cannot say it.
• Several trials quit after 2 steps on "no module named pandas" and finished the task as failed. Trials that went on to solve the same task did not stop there.
• About 40% of the spend was in trials that ended themselves and failed, so a run's own claim of success is not reliable evidence.

A run that went wrong once in a recognisable way is a candidate for a hint on the next similar run. Nothing in TokenOps or Chronicle turns past runs into such a hint.

Proposed approach (a sketch to react to, not a design)

1. Index finished runs in the control plane by intent (an embedding of the first request), with the run's cost and a verified outcome.
2. At run start, or when a detector fires, retrieve the closest matches above a similarity floor and inject a short hint: a few "do" items from verified successes (cheapest first, a few kept diverse) and one to three "avoid" items. Target 100 to 200 tokens in a fixed shape, so it stays cheap and the prompt cache stays stable.
3. Distill once per stored run, offline, not at query time. An option is to have the agent emit a one-line hint when the run ends (about 100 output tokens against about 20,000 per run in the benchmark). Do not emit per step.
4. Keep a holdout: a fraction of runs per intent get no hint, so exploration continues and each hint can be measured against its own control. Retire hints that stop earning their place.
5. Simplest slice first: make the existing progress_guard nudge specific using an "avoid" rule that matches the stall signature, instead of putting anything in the prompt up front.

Caveats

These are the reasons this may not be worth building, listed first so nobody reads the sketch as a plan.

• There is no success signal. Without a trustworthy outcome per run, "pick the cheapest successful run" has nothing to rank on, and an agent's own claim of success is wrong often enough to be dangerous. The idea only works for agents with a real signal (tests, a resolution flag, a downstream check). For the others it is a deal breaker. This was the conclusion we reached when discussing it.
• Sparse data. In the benchmark, most tasks had no successful run at all and the successes were concentrated in a handful of easy tasks. Retrieval would return nothing for the tasks where the cost is. The assumption that a few frequent intents dominate production traffic is what would make it pay off, and it is untested here.
• Wrong bucket. Similarity by wording is not similarity of intent: fix-git and git-leak-recovery read alike but need different approaches. A hint applied to the wrong intent does harm. Needs a similarity floor and measurement.
• Correlated failure. A bad hint is applied to every run in its bucket, so it fails the same way everywhere. This is the opposite of an independent mistake.
• Poisoned memory. Stored runs contain tool output and web content the agent does not control. Anything retrieved and injected is a prompt-injection path. Only distilled text should be stored, never raw tool output, and hints need a review or a restricted format.
• Pigeonholing. Guidance that is too prescriptive narrows exploration. Prefer constraints ("avoid X", "check Y first") over step lists, and keep the holdout.
• Self-reports. A hint emitted by the agent must be stored as a "do" hint only after a verified success, and as an "avoid" hint after a verified failure.
• Evaluation leakage. Learning from a benchmark and testing on the same tasks proves nothing. It has to be tested on held-out tasks, with repeats, because run-to-run noise in the same benchmark was larger than the effects measured.
• Cost and privacy. Distillation is another model call per stored run, and production trajectories may contain data that must not be stored or reused across tenants.

Tradeoffs / non-scope

• A smaller variant needs no success label: build "avoid" hints only from objectively bad signals (loops, halts, timeouts), which are observable without ground truth. It is a much smaller idea, and the one I would try first if any.
• Layering: Chronicle records, the control plane would store and index, TokenOps would inject. That crosses three repositories, so it needs an ADR (docs/adr/) before any code, and probably an issue in each of the other two.
• Not proposing: changing the guidance on every run, any hint stored without a verified outcome, or any change to progress_guard's detection. Issue 199 and its PR already cover detection.
• Out of scope: how a production agent gets its success signal.

Open questions, with a recommendation each:

• Which agents have a usable success signal? Recommend naming one concrete agent before designing anything.
• Should the first experiment be the smaller "avoid"-only variant on stall signatures? Recommend yes.