Skip to content

Commit 8f765a0

Browse files
anandgupta42claude
andcommitted
fix(harness): address multi-model review consensus — summarizer starvation exemption and source hygiene
Third-pass review across independent reviewers. Most reported items were already fixed on this branch; these are the residual confirmed ones. - `processor.ts`: the compaction summarizer runs through the same processor under the session's own id, so it shared the working agent's per-session starvation tracker — its single mutation-free step advanced `turnsWithoutMutation` for the real agent (spurious would-fire telemetry in the default annotate mode, a premature directive in armed mode). Directive delivery was already exempted for summary messages; starvation accounting now is too, via the same `sbExempt` gate. - `starvation.ts`: two raw NUL bytes were embedded directly in source as string-literal separators, which made the file classify as binary — `grep`, `file`, and review tooling skipped it entirely. Replaced with the equivalent unicode escapes; the runtime strings are byte-identical. - `config.ts` / `starvation.ts`: `max_turns_without_mutation` is counted per generation step, not per user message; the schema description and the threshold rationale said "assistant turns", which misleads operators tuning it (one user message routinely spans several read-only steps). - Tests: pin the summarizer exemption in the gate suite, and add two idle-done cases that drive the detector in production event order (step-finish part, then the step's snapshot patch part) — the existing fixtures emit the patch first, which would mask an ordering regression in the mutation-versus-verify comparison. - `harness-review-followups.md`: record the four items deliberately deferred from this pass (historical tool-stub omission on the summarizer path, repeat-signature accumulation on successful calls, the run record's stop-reason fallback, and converging the remaining test fixtures on production ordering). Gates: `bun test test/session/ test/cli/` (only the 2 known prompt.test.ts timeout flakes fail), `bun run typecheck` clean, marker guard clean. Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_017KXpxBn4zteNfXTwv8cf93
1 parent 2a8850c commit 8f765a0

6 files changed

Lines changed: 91 additions & 10 deletions

File tree

‎.github/meta/harness-review-followups.md‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -41,3 +41,14 @@ were deliberately deferred (no behavior change on this branch):
4141
[LOW] packages/opencode/src/session/{termination,nudge,tool-result-cap}.ts, packages/opencode/src/cli/cmd/{run-accounting,idle-done}.ts — these 5 new modules use `export namespace` for organization, which the repo's module-shape convention (`packages/opencode/AGENTS.md`) asks new code to avoid in favor of flat exports + a bottom-of-file self-reexport. ~69 pre-existing files in the package already use the same pattern, so this is consistent with existing debt rather than a regression; fold into a holistic namespace-to-flat-exports cleanup across the package rather than converting these 5 files in isolation.
4242

4343
[LOW] packages/opencode/test/session/starvation.test.ts — the run-mode "armed" gate test re-implements the gate expression as a local helper instead of importing the real predicate from processor.ts, so a future change to the actual gate (added condition, reordered precedence, renamed exemption) would not be caught by this test — extract the gate into a shared, directly-testable predicate.
44+
45+
The following items came from a further multi-reviewer pass over this branch and
46+
were deliberately deferred (no behavior change on this branch):
47+
48+
[MED] packages/opencode/src/session/llm.ts (addHistoricalToolStubs) — the stub-injection skip was narrowed to `toolChoice === "none" && no tools`, which closes the normal-turn regression, but the compaction summarizer still takes that path with a head that references historical tool calls. The skip rests on the claim that omitting both `tools` and `tool_choice` is universally accepted; the issue the stubs exist to fix concerned validation of referenced historical tool calls, which is orthogonal. Verify against the provider that originally needed the stubs before changing anything — reintroducing stubs here would advertise callable tools on a text-only call, so this is a provider-compatibility question, not a code cleanup.
49+
50+
[MED] packages/opencode/src/session/starvation.ts (repeatSignature / consecutive-signature counter) — the repeat signature folds the failure message in but the counter accumulates for successful calls too, so three identical SUCCESSFUL calls (re-running the same verification command, polling a status probe) reach the threshold and produce a directive asserting that repeating the call cannot change the result, which is untrue for a polling or verification call. Restricting accumulation to calls that carry a failure message narrows the detector's semantics and its overlap with the identical-args doom-loop ladder; make that change with validation data rather than in review, and note the directive text is prompt-visible.
51+
52+
[LOW] packages/opencode/src/cli/cmd/run-accounting.ts (termination) — `why_model_stopped` falls back to "stop" when no step-finish reason was ever recorded (an aborted run that never completed a step), so the run record attributes an ordinary stop to a session that never reported one. Distinguishing it needs a new value in the published record's enum, which is an output-contract change for downstream consumers — batch it with the next deliberate revision of the run record schema.
53+
54+
[LOW] packages/opencode/src/session/prompt.ts (post-compaction pin reminder) and packages/opencode/test/cli/idle-done.test.ts — the idle-done unit tests now pin production event ordering (step-finish part before the step's snapshot patch part) in two dedicated cases, but the shared fixtures still emit the patch first; converge the remaining fixtures on production ordering when the file is next touched.

‎packages/core/src/v1/config/config.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -315,7 +315,7 @@ export const Info = Schema.Struct({
315315
}),
316316
max_turns_without_mutation: Schema.optional(PositiveInt).annotate({
317317
description:
318-
"Consecutive assistant turns with zero corroborated file mutation before the write-starvation breaker fires (default: 12; see session/starvation.ts).",
318+
"Consecutive assistant generation steps with zero corroborated file mutation before the write-starvation breaker fires (default: 12). Counted per model step, not per user message — one user turn routinely spans several read-only steps, so tune this against step counts (see session/starvation.ts).",
319319
}),
320320
repeat_signature_threshold: Schema.optional(PositiveInt).annotate({
321321
description:

‎packages/opencode/src/session/processor.ts‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -133,7 +133,16 @@ export namespace SessionProcessor {
133133
processConfig.experimental?.starvation_breaker as SessionStarvation.ConfigShape | undefined,
134134
)
135135
const runMode = Flag.ALTIMATE_RUN_MODE
136-
const sbExempt = sbConfig.exemptAgents.includes(input.assistantMessage.agent)
136+
// The compaction summarizer runs through this same processor under the
137+
// session's OWN id, so it would otherwise share the working agent's
138+
// per-session tracker: its single mutation-free step increments
139+
// `turnsWithoutMutation`, inflating the real agent's counter (spurious
140+
// would-fire telemetry in annotate mode, a premature directive in armed
141+
// mode). It can only produce a summary, so it is exempt from starvation
142+
// accounting entirely — the same reason it is excluded from directive
143+
// delivery below.
144+
const sbSummarizer = input.assistantMessage.summary === true
145+
const sbExempt = sbConfig.exemptAgents.includes(input.assistantMessage.agent) || sbSummarizer
137146
const starvation =
138147
sbConfig.mode === "off" || sbExempt ? undefined : SessionStarvation.forSession(input.sessionID, sbConfig)
139148
const sbArmed = sbConfig.mode === "armed" && runMode && !sbExempt

‎packages/opencode/src/session/starvation.ts‎

Lines changed: 7 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -58,9 +58,11 @@ export namespace SessionStarvation {
5858
// - repeatSignatureThreshold = 3: three identical (tool+args+touched-files+
5959
// failure) signatures means three attempts produced the same failure —
6060
// repeating the call cannot change the outcome.
61-
// - maxTurnsWithoutMutation = 12: legitimate exploration bursts (read/search
62-
// before a first edit or a final answer) span a handful of assistant
63-
// turns; 12 consecutive assistant turns with zero corroborated file
61+
// - maxTurnsWithoutMutation = 12: counted per GENERATION STEP (onStepFinish
62+
// is called once per model step, and one user message routinely spans
63+
// several read-only steps) — not per user message. Legitimate exploration
64+
// bursts (read/search before a first edit or a final answer) span a
65+
// handful of steps; 12 consecutive steps with zero corroborated file
6466
// mutation is well beyond that regime while still permitting long
6567
// read-only research tasks to proceed (the directive is outcome-neutral).
6668
// - pollingThresholdMultiplier = 5: identical polling commands (sleep/watch/
@@ -212,7 +214,7 @@ export namespace SessionStarvation {
212214
normalizeArgs(input.args),
213215
[...(input.touchedFiles ?? [])].sort().join(","),
214216
(input.failureMessage ?? "").replace(/\s+/g, " ").trim(),
215-
].join("
217+
].join("\u0000"),
216218
)
217219
}
218220

@@ -364,7 +366,7 @@ export namespace SessionStarvation {
364366
// starvation counter.
365367
}
366368

367-
const key = `${input.tool}
369+
const key = `${input.tool}\u0000${normalizeArgs(input.input)}`
368370
if (key === lastCallKey) consecutiveIdenticalCalls++
369371
else {
370372
lastCallKey = key

‎packages/opencode/test/cli/idle-done.test.ts‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -252,6 +252,36 @@ describe("IdleDone hard preconditions", () => {
252252
expect(d.shouldChallenge()).toBe(false)
253253
})
254254

255+
// Production emits the step-finish part FIRST and the snapshot `patch` part
256+
// immediately after it (session/processor.ts writes step-finish, then diffs the
257+
// snapshot), so the two tests below drive the detector in that real order — the
258+
// helpers above emit the patch before step-finish, which would mask an ordering
259+
// regression in the mutation/verify comparison.
260+
test("(i) production ordering: patch AFTER its own step-finish still precedes a later verify", () => {
261+
const d = IdleDone.create(OPTS, deps(["cmp_1", "cmp_2"]))
262+
d.observePart(editPart("m_work"))
263+
d.observePart(stepFinish("m_work"))
264+
d.observePart(patchPart("m_work")) // step-end snapshot diff, as production emits it
265+
d.observePart(bashPart("m_verify", "./scripts/verify.sh --all", 0))
266+
d.observePart(stepFinish("m_verify"))
267+
d.observePart(stepFinish("cmp_1"))
268+
d.observePart(stepFinish("cmp_2"))
269+
for (const m of ["m_idle1", "m_idle2", "m_idle3"]) d.observePart(stepFinish(m))
270+
expect(d.shouldChallenge()).toBe(true)
271+
})
272+
273+
test("(i) production ordering: a same-step patch emitted after step-finish still suppresses", () => {
274+
const d = IdleDone.create(OPTS, deps(["cmp_1", "cmp_2"]))
275+
d.observePart(editPart("m1"))
276+
d.observePart(bashPart("m1", "make check", 0)) // verify inside the mutating step
277+
d.observePart(stepFinish("m1"))
278+
d.observePart(patchPart("m1")) // snapshot diff lands after step-finish
279+
d.observePart(stepFinish("cmp_1"))
280+
d.observePart(stepFinish("cmp_2"))
281+
for (const m of ["m2", "m3", "m4"]) d.observePart(stepFinish(m))
282+
expect(d.shouldChallenge()).toBe(false)
283+
})
284+
255285
test("(i)/(ii) a FAILING most-recent verify blocks the challenge", () => {
256286
const d = IdleDone.create(OPTS, deps(["cmp_1", "cmp_2"]))
257287
d.observePart(editPart("m1"))

‎packages/opencode/test/session/starvation.test.ts‎

Lines changed: 32 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -402,10 +402,20 @@ describe("doom-loop escalation ladder — re-keyed on (toolName + normalized arg
402402

403403
describe("armed gating logic (run-mode-only, exempt agents)", () => {
404404
// Mirrors the gate expression in processor.ts:
405-
// sbArmed = mode === "armed" && runMode && !exemptAgents.includes(agent)
406-
function armed(mode: SessionStarvation.Mode, runMode: boolean, agent: string) {
405+
// sbExempt = exemptAgents.includes(agent) || assistantMessage.summary
406+
// sbArmed = mode === "armed" && runMode && !sbExempt
407+
// starvation tracker is created only when mode !== "off" && !sbExempt
408+
function exempt(resolved: SessionStarvation.ResolvedConfig, agent: string, summary: boolean) {
409+
return resolved.exemptAgents.includes(agent) || summary
410+
}
411+
function armed(mode: SessionStarvation.Mode, runMode: boolean, agent: string, summary = false) {
412+
const resolved = SessionStarvation.resolveConfig({ mode })
413+
return mode === "armed" && runMode && !exempt(resolved, agent, summary)
414+
}
415+
/** Whether the per-session tracker is wired at all (and so can accumulate steps). */
416+
function tracks(mode: SessionStarvation.Mode, agent: string, summary = false) {
407417
const resolved = SessionStarvation.resolveConfig({ mode })
408-
return mode === "armed" && runMode && !resolved.exemptAgents.includes(agent)
418+
return mode !== "off" && !exempt(resolved, agent, summary)
409419
}
410420

411421
test("annotate mode (the default) never arms — even in run mode", () => {
@@ -421,6 +431,25 @@ describe("armed gating logic (run-mode-only, exempt agents)", () => {
421431
expect(armed("armed", true, "plan")).toBe(false)
422432
expect(armed("armed", true, "review")).toBe(false)
423433
})
434+
435+
// The compaction summarizer runs through the same processor under the session's
436+
// OWN id, so without an exemption its single mutation-free step would advance
437+
// the working agent's shared tracker.
438+
test("the compaction summarizer is exempt: no tracker is wired for a summary message", () => {
439+
expect(tracks("annotate", "build", true)).toBe(false)
440+
expect(tracks("armed", "build", true)).toBe(false)
441+
// a normal working step on the same session still tracks
442+
expect(tracks("annotate", "build", false)).toBe(true)
443+
})
444+
445+
test("the compaction summarizer never arms, even in armed run mode", () => {
446+
expect(armed("armed", true, "build", true)).toBe(false)
447+
expect(armed("armed", true, "build", false)).toBe(true)
448+
})
449+
450+
test("mode 'off' wires no tracker at all", () => {
451+
expect(tracks("off", "build")).toBe(false)
452+
})
424453
})
425454

426455
describe("nudge arbiter (one-directive-per-turn contract)", () => {

0 commit comments

Comments
 (0)