Skip to content

Commit 873cff0

Browse files
anandgupta42claude
andcommitted
fix(learn): scope recurrence to the named store and harden snapshot reads
- `learn status --name` and reflection attribute only that store. - Compaction replays (text metadata `learnOriginalMessage`) have unknown turn starts; attachment-only and agent-mention turns stay request boundaries even though their text companions are synthetic. - Snapshot reads fail closed on unsafe paths and skip a file whose name does not match its session. - The `r` prompt line asks for an EDIT only when the wording is unclear and to keep a correct rule the agent did not follow. - Tests: assert the analyzer really fails in the reflection fallback test, check status text with learning disabled, assert a real unmatched-topic line. Co-Authored-By: Claude Opus 5.5 <[email protected]>
1 parent cd539a8 commit 873cff0

12 files changed

Lines changed: 47 additions & 15 deletions

File tree

‎docs/docs/configure/learn.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -236,7 +236,7 @@ Shows whether learning is on, whether automatic promotion is on with its last pr
236236

237237
A correction matches a lesson when it names one of the lesson's backtick identifiers plus another word from the lesson, or contains most of the lesson's words. The text output lists only lessons with a matching correction and summarizes the rest as "no observed correction". That is not proof the agent followed them. Corrections are recorded only while capture is on. Status also lists up to 5 identifiers that were corrected in 2 or more sessions with no matching current lesson. They are taken from your local corrections and never leave the project. If the analysis fails, status shows `Recurrence unavailable` (`recurrence_error` in JSON), and the rest of the status still prints.
238238

239-
When the current text of a lesson has `recurred` in 2 or more sessions, reflection marks it `r:N` in the playbook it sends the model. The model is asked to make the lesson clearer or more specific, never weaker. The result is a candidate like any other and goes through the same review or automatic-promotion gates.
239+
When the current text of a lesson has `recurred` in 2 or more sessions, reflection marks it `r:N` in the playbook it sends the model. The model is asked to make the lesson clearer or more specific when the feedback shows the wording is unclear, and to keep it when the agent simply did not follow a correct rule. The result is a candidate like any other and goes through the same review or automatic-promotion gates.
240240

241241
```bash
242242
altimate-code learn status

‎packages/opencode/src/altimate/learn/capture.ts‎

Lines changed: 11 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -92,16 +92,22 @@ export interface MessageLike {
9292
time?: { created?: number; completed?: number }
9393
}
9494

95-
export type TurnMessage = MessageLike & { synthetic?: boolean; compaction?: boolean }
95+
export type TurnMessage = MessageLike & { synthetic?: boolean; compaction?: boolean; replay?: boolean }
9696

97-
/** Session history as turn boundaries: all-synthetic text continues a turn; a compaction part marks compaction. */
98-
export function turnMessages(messages: readonly { info: MessageLike; parts: readonly { type: string; synthetic?: boolean }[] }[]): TurnMessage[] {
97+
type TurnPart = { type: string; synthetic?: boolean; metadata?: Record<string, unknown> }
98+
99+
/**
100+
* Session history as turn boundaries. All-synthetic text continues a turn unless the message carries user input
101+
* (an attachment or agent mention, whose text companions are synthetic). Compaction and its replay are marked.
102+
*/
103+
export function turnMessages(messages: readonly { info: MessageLike; parts: readonly TurnPart[] }[]): TurnMessage[] {
99104
return messages.map(({ info, parts }) => {
100105
const text = parts.filter((part) => part.type === "text")
101106
return {
102107
...info,
103-
synthetic: text.length > 0 && text.every((part) => part.synthetic),
108+
synthetic: text.length > 0 && text.every((part) => part.synthetic) && !parts.some((part) => part.type === "file" || part.type === "agent"),
104109
compaction: parts.some((part) => part.type === "compaction"),
110+
replay: text.some((part) => typeof part.metadata?.learnOriginalMessage === "string"),
105111
}
106112
})
107113
}
@@ -114,7 +120,7 @@ export function correctedTurnStart(messages: readonly TurnMessage[], messageID:
114120
const end = messages.findIndex((message) => message.id === messageID && message.role === "user")
115121
if (end < 0) return
116122
const previous = messages.slice(0, end).findLastIndex((message) => message.role === "user" && !message.synthetic && !message.compaction)
117-
if (previous < 0) return
123+
if (previous < 0 || messages[previous].replay) return
118124
if (messages.slice(0, previous).findLast((message) => message.role === "user")?.compaction) return
119125
return messages.slice(previous + 1, end).find((message) => message.role === "assistant")?.time?.created
120126
}

‎packages/opencode/src/altimate/learn/delivery.ts‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -103,10 +103,12 @@ export async function readShownSnapshots(root: string, options: { limit?: number
103103
for (const { file } of files.sort((a, b) => b.mtime - a.mtime || a.file.localeCompare(b.file)).slice(0, limit)) {
104104
try {
105105
const state = State.parse(JSON.parse((await read(root, file)) ?? ""))
106+
if (path.basename(file) !== Store.sha256(state.session) + ".json") throw new Error("Learn session state belongs to another session")
106107
snapshots.push({ session: state.session, shown: state.shown.map(({ name, lesson, tier, at }) => ({
107108
name, id: lesson.id, textHash: lessonTextHash(lesson.text), tier, at,
108109
})) })
109110
} catch (error) {
111+
if (error instanceof SafeFS.UnsafeLearnPathError) throw error
110112
coverage.sessions_skipped_malformed++
111113
log.warn("learn session snapshot skipped", { file, error })
112114
}

‎packages/opencode/src/altimate/learn/prompt.txt‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,4 +36,4 @@ Bullet text rules (deterministic checks reject unsafe content and flag verificat
3636
Output: a JSON object {"deltas": [...]}. Each delta is {"op": "ADD"|"EDIT"|"REMOVE"|"HELPFUL"|"HARMFUL", "id": "<bullet id, for every op except ADD>", "text": "<bullet text, for ADD and EDIT>", "reason": "<one short sentence citing the feedback>"}.
3737
Optional fields: "supersedes": "<existing bullet id>" for ADD, and "coexists": ["<existing bullet id>", ...] for ADD or EDIT. Omit these fields when they do not apply.
3838

39-
r = sessions where this rule was shown and the user still corrected the same topic; consider making it more specific or clearer (EDIT), never weaker
39+
r = sessions where this rule was shown and the user still corrected the same topic. If the feedback shows the wording is unclear or incomplete, EDIT it to be clearer or more specific. If the agent simply did not follow a correct rule, keep it. Narrow a rule only when the feedback shows it is too broad; never drop the behavior it requires.

‎packages/opencode/src/altimate/learn/recurrence.ts‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -107,7 +107,8 @@ export function attribute(signals: readonly Signal[], snapshots: readonly ShownS
107107
}
108108
}
109109

110-
export async function analyzeRecurrence(root: string, options: { limit?: number } = {}) {
110+
/** With `name`, only that store's lessons and signals are attributed. */
111+
export async function analyzeRecurrence(root: string, options: { limit?: number; name?: string } = {}) {
111112
const { snapshots, coverage } = await readShownSnapshots(root, options)
112113
const dir = path.join(root, ".altimate-code", "learn")
113114
await SafeFS.assertSafePath(root, dir)
@@ -121,6 +122,7 @@ export async function analyzeRecurrence(root: string, options: { limit?: number
121122
for (const entry of entries.sort((a, b) => a.name.localeCompare(b.name))) {
122123
if (!entry.isDirectory()) continue
123124
try { validateName(entry.name) } catch { continue }
125+
if (options.name !== undefined && entry.name !== options.name) continue
124126
if (sessions.size) signals.push(...(await readSignalsSnapshot(root, entry.name)).filter((signal) => sessions.has(signal.sessionID)))
125127
const file = Store.paths(root, entry.name).approved
126128
try {

‎packages/opencode/src/altimate/learn/session-reflect.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -108,7 +108,7 @@ export async function reflectCore(input: ReflectCoreInput): Promise<ReflectCoreR
108108
const digest = buildDigest(input.source)
109109
const snapshot = await prepareReflection(root, name, input.applyPaths)
110110
const originalText = new Map(Playbook.bullets(snapshot).map((b) => [b.id, b.text]))
111-
const recurrence = await analyzeRecurrence(root).catch((error) => {
111+
const recurrence = await analyzeRecurrence(root, { name }).catch((error) => {
112112
log.warn("learn recurrence unavailable", { error })
113113
return undefined
114114
})

‎packages/opencode/src/cli/cmd/learn.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -381,7 +381,7 @@ const StatusCommand = effectCmd({
381381
retired: (await Store.loadRetired(root, name)).length,
382382
open_signals: signals.length,
383383
// Attribution is advisory; its failure must not hide the rest of the status.
384-
...await analyzeRecurrence(root).then((recurrence) => ({ recurrence }), (error) => ({ recurrence_error: error instanceof Error ? error.message : String(error) })),
384+
...await analyzeRecurrence(root, { name }).then((recurrence) => ({ recurrence }), (error) => ({ recurrence_error: error instanceof Error ? error.message : String(error) })),
385385
pending_recoveries: sessions.length,
386386
pending_replacements: (await Store.readPendingReplacements(root, name)).length,
387387
backoff_sessions: sessions.filter((session) => (state.recoveries[session]?.retryAt ?? 0) > Date.now()).length,

‎packages/opencode/test/altimate/learn/capture.test.ts‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -370,6 +370,11 @@ describe("correction turn start", () => {
370370
expect(mapped.map(({ synthetic, compaction }) => [synthetic, compaction])).toEqual([
371371
[false, false], [false, false], [true, false], [false, true], [false, false],
372372
])
373+
const [attachment, replay] = turnMessages([
374+
{ info: info("attach", "user", 4000), parts: [{ type: "text", synthetic: true }, { type: "file" }] },
375+
{ info: info("replay", "user", 4100), parts: [{ type: "text", metadata: { learnOriginalMessage: "u1" } }] },
376+
])
377+
expect([attachment.synthetic, replay.synthetic, replay.replay]).toEqual([false, false, true])
373378
})
374379
test("a replayed request after compaction has no known turn start", () => {
375380
const at = (id: string, role: string, created: number, extra = {}) => ({ id, sessionID: "s", role, time: { created, completed: created + 1 }, ...extra })
@@ -379,6 +384,9 @@ describe("correction turn start", () => {
379384
at("replay", "user", 3200), at("final", "assistant", 4000), at("correction", "user", 5000),
380385
]
381386
expect(correctedTurnStart(compacted, "correction")).toBeUndefined()
387+
// A replay marked by its text metadata is unknown even when no marker precedes it.
388+
const marked = compacted.filter((m) => m.id !== "marker").map((m) => m.id === "replay" ? { ...m, replay: true } : m)
389+
expect(correctedTurnStart(marked, "correction")).toBeUndefined()
382390
// A manual compaction without a replay leaves the original request as the boundary.
383391
expect(correctedTurnStart(compacted.filter((m) => m.id !== "replay"), "correction")).toBe(2000)
384392
})

‎packages/opencode/test/altimate/learn/cli.test.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -176,7 +176,7 @@ describe("learn opt-in and status", () => {
176176
lessons: [], unmatched_topics: [],
177177
coverage: { sessions_analyzed: 0, sessions_skipped_malformed: 0, truncated: false },
178178
})
179-
const shown = await learn(dir.path, "status")
179+
const shown = await runLearn(dir.path, ["status"], undefined, { ALTIMATE_LEARN: "0" })
180180
expect(shown.stdout).toContain("Recurrence (observed counts need capture on)")
181181
expect(shown.stdout).toContain("no matching current lesson")
182182
})

‎packages/opencode/test/altimate/learn/recurrence.test.ts‎

Lines changed: 15 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -150,7 +150,9 @@ describe("recurrence attribution", () => {
150150
})
151151

152152
test("human status reports coverage, uncertainty and no matching current lesson", () => {
153-
const text = formatRecurrence({ ...run([]), coverage: { sessions_analyzed: 1, sessions_skipped_malformed: 0, truncated: false } })
153+
const unmatched = (session: string) => signal(session, { id: `sig_u_${session}`, text: "No, filter out _is_deleted rows" })
154+
const text = formatRecurrence({ ...run([unmatched("s1"), unmatched("s2")], [snapshot("s1"), snapshot("s2")]), coverage: { sessions_analyzed: 2, sessions_skipped_malformed: 0, truncated: false } })
155+
expect(text).toContain(" _is_deleted: 2 sessions (sig_u_s1, sig_u_s2)")
154156
expect(text).toContain("Recurrence")
155157
expect(text).toContain("no observed correction")
156158
expect(text).toContain("counts need capture on")
@@ -168,8 +170,8 @@ test("snapshot analysis is bounded, skips malformed files and never repairs any
168170
name: lesson.name, lesson: { ...lesson.lesson, tags: [], scope: "project", helpful: 0, harmful: 0, applied: 0, created: time(0), updated: time(0) },
169171
tier: "file", at: time(1), queryHash: "hash",
170172
}], requests: [], compactions: [], counted: [] })
171-
const old = path.join(dir, "old.json")
172-
const good = path.join(dir, "good.json")
173+
const old = path.join(dir, Store.sha256("old") + ".json")
174+
const good = path.join(dir, Store.sha256("s1") + ".json")
173175
const bad = path.join(dir, "bad.json")
174176
await fs.writeFile(old, JSON.stringify(state("old")))
175177
await fs.writeFile(good, JSON.stringify(state("s1")))
@@ -190,5 +192,15 @@ test("snapshot analysis is bounded, skips malformed files and never repairs any
190192
expect(await fs.readFile(signals, "utf8")).toBe(raw)
191193
expect((await fs.readdir(p.learnDir)).sort()).toEqual([path.basename(p.approved), "signals.jsonl"].sort())
192194
expect((await readShownSnapshots(root, { limit: 0 })).coverage).toMatchObject({ sessions_analyzed: 0, truncated: true })
195+
// A copied snapshot under another session's file name is not that session's evidence.
196+
await fs.copyFile(good, path.join(dir, Store.sha256("copy") + ".json"))
197+
expect((await readShownSnapshots(root)).coverage).toMatchObject({ sessions_analyzed: 2, sessions_skipped_malformed: 2 })
198+
// Another store's lessons and signals stay out of a named analysis.
199+
expect((await analyzeRecurrence(root, { name: "other-store" })).lessons.every((row) => row.recurred === 0)).toBe(true)
200+
// Unbounded, both valid sessions ("old" and "s1") carry a matching correction after exposure.
201+
expect((await analyzeRecurrence(root, { name: lesson.name })).lessons[0]).toMatchObject({ recurred: 2 })
202+
// An unsafe snapshot path fails closed instead of counting as malformed.
203+
await fs.symlink(good, path.join(dir, Store.sha256("link") + ".json"))
204+
await expect(readShownSnapshots(root)).rejects.toThrow()
193205
} finally { await fs.rm(root, { recursive: true, force: true }) }
194206
})

0 commit comments

Comments
 (0)