Skip to content

Commit 4e6e378

Browse files
saravmajesticclaude
andcommitted
fix(learn): address sync review findings
- Outbox: a malformed `outbox.json` keeps its valid entries and is set aside as `outbox.json.corrupt-<ms>` before it is replaced; oversized legacy usage batches are split when read - Backfill runs again whenever a pull finds no earlier cache, and proposes a lesson changed since the server last saw it (skips only unchanged ones) - Batch results match only one exact `(lesson_key, submission_hash)`; proposals sharing both never share a request; unmatched, duplicate or incomplete results stay queued and report an error - A whole-request refusal (422) is never resent as is: items are sent one by one and the refused item is held - Usage over 500 items is split into entries with derived `batch_id`s; parse `skipped_items` - Wire limits (tags, path triggers, coexists, replaces) are checked after coalescing and again before sending; provenance is cut to 4000 - `lessons_not_available` backs off like an older server and keeps the cache - Promotions derive proposals in `beforePublish` and queue them in a new `afterPublish` hook, so a failed promote never touches the queue - Session views track the account and a per-project binding epoch; LRU memo - Rotate `local_keys` past 1000 across pulls (`local_keys_offset`) - Keyset-page `GET /submissions` with `since` (echoed as received) and `after_id` - `learn sync --json` exits non-zero on failure; `disable --sync` checks the override before dropping the cache; `learn show` marks held proposals and workspace-wide team lessons - Build lesson delivery on a fresh clone when sync is on - `lesson-api.ts` uses flat exports with `export * as LessonApi` - Docs and a note on regenerating `lint_cases.json` Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
1 parent be70521 commit 4e6e378

17 files changed

Lines changed: 874 additions & 163 deletions

File tree

‎docs/docs/configure/learn.md‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -212,15 +212,15 @@ With sync off, learn makes no network requests for lessons and delivers no team
212212
- each lesson's text, tags, path triggers, pin, provenance, and coexistence declarations;
213213
- how often each team lesson was delivered, and helpful or harmful marks the curator gave it.
214214

215-
Everything you send lands as a proposal. It reaches the team only after the workspace owner approves it, and that includes lessons promoted automatically on your machine. The first sync also uploads the lessons the project already has: the candidate if there is one, otherwise the approved set, at most 200 per run, starting with the most helpful. A lesson the candidate removed counts as an intentional removal and is not uploaded.
215+
Everything you send lands as a proposal. It reaches the team only after the workspace owner approves it, and that includes lessons promoted automatically on your machine. The first sync also uploads the lessons the project already has: the candidate if there is one, otherwise the approved set, at most 200 per run, starting with the most helpful. A lesson the candidate removed counts as an intentional removal and is not uploaded. The same upload runs again after sync was off or the pulled lessons were deleted. It skips a lesson the workspace already holds as it is now (your open proposal, an earlier submission, or the team version) and proposes one you changed since: a revision of your open proposal, or a new proposal. A lesson the team has a version of, or retired, is not proposed by this upload; edit it to propose a change. A lesson the workspace would refuse is not sent, and `learn status` lists it as not syncable: text over 140 characters, more than 32 tags or a tag over 64 characters, more than 32 path triggers, more than 100 coexistence declarations, or more than 50 replaced lessons. Provenance over 4,000 characters is shortened.
216216

217217
**Editing team lessons.** Reflection sees team lessons but never writes them into `candidate.json` or `approved.json`. Editing, removing, or superseding a team lesson becomes a proposal that names the approved version it replaces. The current lesson stays live until the owner approves the change. Proposals are queued in `outbox.json` and sent after reflection and promotion, at session start, and by `learn sync` and `learn push`.
218218

219219
**Conflicts.** If someone else changed a lesson on the server first, for example the owner edited your proposal or approved another edit, your proposal is held instead of being sent again over their change. If the server already holds what you proposed, the held proposal is dropped. Otherwise `learn status` lists it, and you resend it after review with `learn push --resubmit <lesson key>`. A proposal the owner rejected is never sent again.
220220

221221
**Retirement.** When the owner retires a lesson, the next pull hides it in every project, including a copy restored by a Git checkout or `learn rollback`. A running session drops it from its "Team rules" section and from later requests, but text the agent has already received is not rewritten. Rolling back locally changes only your local lessons. If the workspace still serves a playbook skill published earlier with `learn promote --publish`, its lessons keep being delivered until the owner detaches that skill in the workspace's Skills page; `learn sync` warns about it. With sync on, `promote --publish` is refused.
222222

223-
**Accounts and links.** The pulled lessons and the queue belong to one account, workspace, and repository. After you switch accounts, relink, or unlink, the old cache is no longer used. Queued proposals from the old scope are held and are never sent to the new one. `learn disable --sync` turns sync off and deletes the pulled lessons, and leaves capture and local lessons as they are. If the server does not support lesson sync yet, learn retries once a day.
223+
**Accounts and links.** The pulled lessons and the queue belong to one account, workspace, and repository. After you switch accounts, relink, or unlink, the old cache is no longer used. Queued proposals from the old scope are held and are never sent to the new one. `learn disable --sync` turns sync off and deletes the pulled lessons, and leaves capture and local lessons as they are. When `ALTIMATE_LEARN_SYNC` or a higher-precedence config keeps sync on, it reports that and keeps the pulled lessons. If the server does not support lesson sync yet, or not for your tenant, learn retries once a day.
224224

225225
## Learn vs memory vs a knowledge base
226226

@@ -273,7 +273,7 @@ altimate-code learn status --json
273273

274274
### show
275275

276-
Prints approved lessons, the candidate, a diff between them, and pending replacements. Each lesson shows its id, text, and counters (`helpful`, `harmful`, `applied`). Flagged lessons print a `WARNING` line. Lessons over 140 characters from older stores are marked `long`. With lesson sync on, lessons are marked `(team)` (approved in the workspace), `(pending review)` (your proposal is waiting for the owner) or `(local)`, and a "Team" section lists the team lessons.
276+
Prints approved lessons, the candidate, a diff between them, and pending replacements. Each lesson shows its id, text, and counters (`helpful`, `harmful`, `applied`). Flagged lessons print a `WARNING` line. Lessons over 140 characters from older stores are marked `long`. With lesson sync on, lessons are marked `(team)` (approved in the workspace), `(pending review)` (your proposal is waiting for the owner or to be sent), `(held — run learn push --resubmit <key>)` (your proposal conflicted with a change on the server) or `(local)`, and a "Team" section lists the team lessons.
277277

278278
```bash
279279
altimate-code learn show
@@ -383,7 +383,7 @@ altimate-code learn reject
383383

384384
### sync / push
385385

386-
`sync` pulls the approved team lessons and sends queued proposals; `push` only sends. Both need [lesson sync](#sync-with-your-workspace) to be on. `sync` reports each store's pull, what was sent, and warns when the workspace still serves a learn-managed playbook skill.
386+
`sync` pulls the approved team lessons and sends queued proposals; `push` only sends. Both need [lesson sync](#sync-with-your-workspace) to be on. `sync` reports each store's pull, what was sent, and warns when the workspace still serves a learn-managed playbook skill. Both exit non-zero when sync was skipped, a pull failed, or a proposal was not sent; with `--json` the report is printed first.
387387

388388
```bash
389389
altimate-code learn sync

‎packages/opencode/src/altimate/learn/README.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -56,7 +56,8 @@ there is no network traffic and no team lesson is delivered.
5656
requests use the credential captured for the scope, and a result whose scope changed in flight is discarded.
5757
A route the server lacks (404 without a code) backs off for 24 hours.
5858
- `ledger.ts` owns the private per-store files `remote.json`, `outbox.json` and `sync.json` (ignored by the learn
59-
`.gitignore`, written atomically with mode 0600).
59+
`.gitignore`, written atomically with mode 0600). A malformed outbox is never read as empty: its valid entries are
60+
kept and the file is set aside as `outbox.json.corrupt-<ms>` before it is replaced.
6061
- `effective.ts` overlays team lessons on local ones by `(repo_identity, store, lesson_key)`: remote wins, an
6162
exact-identity tombstone hides the local copy. `Store.loadApproved()` stays local-only.
6263
- `proposals.ts` turns local changes into proposals by the ledger (add, revision of your open proposal, or edit/remove

‎packages/opencode/src/altimate/learn/auto-promote.ts‎

Lines changed: 2 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ import { assertLearnLock } from "./lock"
1616
import { errText } from "./session-reflect"
1717
import type { AutoPromoteLimits } from "./config"
1818
import { readRemote, type Scope } from "./ledger"
19-
import { recordPromotion } from "./proposals"
19+
import { promotionHooks } from "./proposals"
2020
import { Log } from "@/util/log"
2121

2222
const log = Log.create({ service: "learn.auto-promote" })
@@ -332,9 +332,7 @@ export async function autoPromote(input: AutoPromoteInput): Promise<AutoPromoteR
332332
publish: publishText,
333333
keepCandidate,
334334
onPublished: () => { published = true },
335-
...(input.sync ? { beforePublish: async (before: Lessons.Lesson[], after: Lessons.Lesson[]) => {
336-
await recordPromotion(root, name, input.sync!.scope, before, after, "auto_promote")
337-
} } : {}),
335+
...(input.sync ? promotionHooks(root, name, input.sync.scope, "auto_promote") : {}),
338336
// Every flagged lesson left in the candidate is unchanged from the approved set (checked above).
339337
allowFlagged: true,
340338
history: {

‎packages/opencode/src/altimate/learn/delivery.ts‎

Lines changed: 4 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ import { validateName } from "./playbook"
1313
import { estimateTokens, fileHookEnabled, lessonLine, renderSection, resolveLimits, retrieve, selectFile, selectStart, type Limits } from "./select"
1414
import * as Store from "./store"
1515
import { effectiveLessons, resolve, type Source } from "./effective"
16-
import { identityKey, readOutbox, readRemote, remoteFor, Scope, sameScope, scopeKey, writeOutbox, type Identity } from "./ledger"
16+
import { identityKey, readOutbox, readRemote, remoteFor, Scope, sameScope, scopeKey, usageBatches, uuidFrom, writeOutbox, type Identity } from "./ledger"
1717
import type { RemoteView } from "./sync"
1818

1919
const log = Log.create({ service: "learn.delivery" })
@@ -91,12 +91,6 @@ function dropLines(text: string, lines: ReadonlySet<string>): string {
9191
return kept.length ? [heading, ...kept].join("\n") : ""
9292
}
9393

94-
/** A stable RFC 4122-shaped id from a hex digest (the server only requires a UUID). */
95-
function uuidFrom(hex: string): string {
96-
const variant = ((parseInt(hex[16], 16) & 0x3) | 0x8).toString(16)
97-
return `${hex.slice(0, 8)}-${hex.slice(8, 12)}-4${hex.slice(13, 16)}-${variant}${hex.slice(17, 20)}-${hex.slice(20, 32)}`
98-
}
99-
10094
async function read(root: string, file: string) {
10195
try {
10296
await SafeFS.assertSafePath(root, file)
@@ -582,11 +576,9 @@ export class Delivery {
582576
for (const [key, batch] of batches) {
583577
const ids = [...batch.items.keys()].sort()
584578
const batchId = uuidFrom(Store.sha256(JSON.stringify([session, name, key, ids])))
585-
if (outbox.usage.some((usage) => usage.batch_id === batchId)) continue
586-
outbox.usage.push({
587-
batch_id: batchId, scope: batch.scope, created_at: new Date().toISOString(),
588-
items: ids.map((id) => ({ public_id: id, applied: batch.items.get(id)!, helpful: 0, harmful: 0 })),
589-
})
579+
const items = ids.map((id) => ({ public_id: id, applied: batch.items.get(id)!, helpful: 0, harmful: 0 }))
580+
for (const chunk of usageBatches(batchId, batch.scope, new Date().toISOString(), items))
581+
if (!outbox.usage.some((usage) => usage.batch_id === chunk.batch_id)) outbox.usage.push(chunk)
590582
}
591583
await writeOutbox(this.root, name, outbox)
592584
}

‎packages/opencode/src/altimate/learn/ledger.ts‎

Lines changed: 81 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -13,11 +13,14 @@ import { createHash } from "node:crypto"
1313
import { constants } from "node:fs"
1414
import path from "node:path"
1515
import z from "zod"
16+
import { Log } from "@/util/log"
1617
import { canonical } from "./lesson"
1718
import { assertLearnLock } from "./lock"
1819
import * as SafeFS from "./safe-fs"
1920
import * as Store from "./store"
20-
import { BatchItem, LessonOut, Tombstone, UsageItem } from "../workspace/lesson-api"
21+
import { BatchItem, LessonOut, MAX_USAGE_ITEMS, Tombstone, UsageItem } from "../workspace/lesson-api"
22+
23+
const log = Log.create({ service: "learn.ledger" })
2124

2225
/** Where lessons sync to: an account (credential digest) on a host and tenant, a workspace, and a repository. */
2326
export const Scope = z.object({
@@ -75,7 +78,11 @@ export const SyncState = z.object({
7578
}).optional(),
7679
/** The server has no lesson routes: retry after this time. */
7780
unsupported_until: z.string().optional(),
81+
/** Keyset cursor into the caller's submissions: the last row read, `(updated_at, public_id)`. */
7882
submissions_cursor: z.string().optional(),
83+
submissions_after_id: z.string().optional(),
84+
/** Where the next pull's `local_keys` window starts when the store has more keys than one request carries. */
85+
local_keys_offset: z.number().int().nonnegative().default(0),
7986
backfill: z.object({ complete: z.boolean(), done: z.array(z.string()) }).default({ complete: false, done: [] }),
8087
ledger: z.record(z.string(), LedgerEntry).default({}),
8188
/** Local lessons the server would refuse as they are (for example grandfathered text over 140 characters). */
@@ -129,7 +136,7 @@ async function read(root: string, file: string): Promise<string | undefined> {
129136
}
130137
}
131138

132-
/** A malformed state file reads as absent: sync state is a cache and a queue, rebuilt by the next pull. */
139+
/** A malformed cache or sync state reads as absent: both are rebuilt by the next pull. */
133140
async function readParsed<T>(root: string, file: string, schema: z.ZodType<T>): Promise<T | undefined> {
134141
const raw = await read(root, file)
135142
if (raw === undefined) return undefined
@@ -149,10 +156,52 @@ export async function removeRemote(root: string, store: string) {
149156
await SafeFS.remove(root, files(root, store).remote)
150157
}
151158

152-
export async function readOutbox(root: string, store: string): Promise<Outbox> {
153-
return (await readParsed(root, files(root, store).outbox, Outbox)) ?? { version: 1, proposals: [], usage: [] }
159+
/** The outbox in `raw`; when it is malformed (`intact: false`), every entry that is still valid. */
160+
function parseOutbox(raw: string): { outbox: Outbox; intact: boolean } {
161+
let value: unknown
162+
try { value = JSON.parse(raw) } catch { value = undefined }
163+
const whole = Outbox.safeParse(value)
164+
if (whole.success) return { outbox: whole.data, intact: true }
165+
const fields = value && typeof value === "object" ? (value as Record<string, unknown>) : {}
166+
const valid = <T>(list: unknown, schema: z.ZodType<T>) =>
167+
Array.isArray(list) ? list.flatMap((item) => { const parsed = schema.safeParse(item); return parsed.success ? [parsed.data] : [] }) : []
168+
return { outbox: { version: 1, proposals: valid(fields.proposals, Proposal), usage: valid(fields.usage, UsageBatch) }, intact: false }
169+
}
170+
171+
/** An outbox as read: `intact: false` when the file was malformed and must be set aside before it is replaced. */
172+
export type ReadOutbox = Outbox & { intact: boolean }
173+
174+
/**
175+
* The outbox is the only copy of what is waiting to be sent, so a malformed file never reads as empty: it yields
176+
* its valid entries, and `writeOutbox` sets the file aside (`outbox.json.corrupt-<ms>`) before replacing it.
177+
* Usage batches over `MAX_USAGE_ITEMS` (queued by an earlier version) are split here, never sent whole.
178+
*/
179+
export async function readOutbox(root: string, store: string): Promise<ReadOutbox> {
180+
const raw = await read(root, files(root, store).outbox)
181+
if (raw === undefined) return { version: 1, proposals: [], usage: [], intact: true }
182+
const { outbox, intact } = parseOutbox(raw)
183+
if (!intact) log.warn("learn outbox is malformed; using its valid entries", { store, proposals: outbox.proposals.length, usage: outbox.usage.length })
184+
const usage = outbox.usage.flatMap((batch) => usageBatches(batch.batch_id, batch.scope, batch.created_at, batch.items))
185+
return { ...outbox, usage, intact }
186+
}
187+
188+
/** `outbox` is what `readOutbox` returned, changed; a malformed file it was read from is kept aside first. */
189+
export async function writeOutbox(root: string, store: string, outbox: Outbox & { intact?: boolean }) {
190+
await assertLearnLock(root)
191+
const file = files(root, store).outbox
192+
const { intact, ...value } = outbox
193+
if (intact === false) {
194+
const aside = `${file}.corrupt-${Date.now()}`
195+
const moved = await SafeFS.rename(root, file, aside).then(() => true, (error: NodeJS.ErrnoException) => {
196+
if (error.code === "ENOENT") return false
197+
throw error
198+
})
199+
if (moved) log.warn("learn outbox was malformed; kept a copy and started a new one", { store, file: path.basename(aside) })
200+
}
201+
await write(root, store, file, value)
202+
// Set aside once: a second write from the same read must not move the new file.
203+
if (intact === false) outbox.intact = true
154204
}
155-
export const writeOutbox = (root: string, store: string, outbox: Outbox) => write(root, store, files(root, store).outbox, outbox)
156205

157206
export async function readSyncState(root: string, store: string): Promise<SyncState> {
158207
return (await readParsed(root, files(root, store).sync, SyncState)) ?? SyncState.parse({ version: 1 })
@@ -162,7 +211,7 @@ export const writeSyncState = (root: string, store: string, state: SyncState) =>
162211
/** The ledger as it stands for `scope`; a different scope's ledger and backfill are discarded, never reused. */
163212
export function forScope(state: SyncState, scope: Scope): SyncState {
164213
if (state.scope === scopeKey(scope)) return state
165-
return { ...state, scope: scopeKey(scope), submissions_cursor: undefined, backfill: { complete: false, done: [] }, ledger: {}, unsyncable: {} }
214+
return { ...state, scope: scopeKey(scope), submissions_cursor: undefined, submissions_after_id: undefined, backfill: { complete: false, done: [] }, ledger: {}, unsyncable: {} }
166215
}
167216

168217
/** The pulled cache, only when it was pulled for exactly this scope. */
@@ -196,7 +245,11 @@ const OPEN = "candidate"
196245
/** Record the caller's own proposals: the open one is the pending slot, every other status is a receipt. */
197246
export function applySubmissions(state: SyncState, store: string, rows: { public_id: string; lesson_key: string; repo_identity: string | null; store: string; status: string; version: number; submission_hash: string; content_hash?: string | null; updated_at: string }[]) {
198247
for (const row of rows) {
199-
if (!state.submissions_cursor || row.updated_at > state.submissions_cursor) state.submissions_cursor = row.updated_at
248+
const cursor = state.submissions_cursor
249+
if (!cursor || row.updated_at > cursor || (row.updated_at === cursor && row.public_id > (state.submissions_after_id ?? ""))) {
250+
state.submissions_cursor = row.updated_at
251+
state.submissions_after_id = row.public_id
252+
}
200253
if (row.store !== store) continue
201254
const value = entry(state, identityKey(row))
202255
const receipt = value.receipts.find((r) => r.submission_hash === row.submission_hash)
@@ -217,3 +270,24 @@ export function applyResult(state: SyncState, key: string, result: { submission_
217270
if (result.status === OPEN) value.pending = { public_id: result.public_id, version: result.version, submission_hash: result.submission_hash }
218271
else if (value.pending?.public_id === result.public_id) delete value.pending
219272
}
273+
274+
/** A stable RFC 4122-shaped id from a hex digest (the server only requires a UUID for `batch_id`). */
275+
export function uuidFrom(hex: string): string {
276+
const variant = ((parseInt(hex[16], 16) & 0x3) | 0x8).toString(16)
277+
return `${hex.slice(0, 8)}-${hex.slice(8, 12)}-4${hex.slice(13, 16)}-${variant}${hex.slice(17, 20)}-${hex.slice(20, 32)}`
278+
}
279+
280+
/**
281+
* Usage items as outbox batches of at most `MAX_USAGE_ITEMS`. The server keys its receipt on `batch_id`, so
282+
* each chunk gets its own id, derived from `id` and the chunk index; a single chunk keeps `id` itself.
283+
*/
284+
export function usageBatches(id: string, scope: Scope, created_at: string, items: readonly z.infer<typeof UsageItem>[]): UsageBatch[] {
285+
if (items.length <= MAX_USAGE_ITEMS) return [{ batch_id: id, scope, created_at, items: [...items] }]
286+
const batches: UsageBatch[] = []
287+
for (let start = 0; start < items.length; start += MAX_USAGE_ITEMS)
288+
batches.push({
289+
batch_id: uuidFrom(createHash("sha256").update(`${id}\0${start / MAX_USAGE_ITEMS}`).digest("hex")),
290+
scope, created_at, items: items.slice(start, start + MAX_USAGE_ITEMS),
291+
})
292+
return batches
293+
}

0 commit comments

Comments
 (0)