Skip to content

Commit ce874ec

Browse files
committed
fix(workspace): close the delete-during-sweep hole, keep a failed seed retryable
Addresses the cubic and Kilo reviews on this PR. **Deleted memory could still come back.** The previous round serialized cloud operations per block, but `backfill` only enters that queue when a worker dequeues an item — a block sitting in `pending` has no queue entry, so a delete issued mid-sweep runs first and the block's `push` then undoes it. Two ways it went wrong: a never-synced block got a live record created for something the user had just deleted; a previously-synced one had its tombstone updated, and because `MemoryApi.update` replaces metadata wholesale, that dropped `archived`. Either way the block was then indexed as synced, so no later sweep re-archived it and it kept injecting into sessions. The local store is now the authority: `push` skips any block that no longer exists locally. The index short-circuit also ignores archived records — it bypassed `isSameBlock`, the only place that checked — so a block recreated under an old id gets a fresh record instead of reviving the tombstone. **Ordering could invert before a block was queued.** `mirrorBlock` and `archiveBlock` resolved the binding and the `memory_enabled` flag before calling `serialize`, both async, so two operations on one block could reach the queue in the opposite order to the writes that triggered them. Both lookups now happen inside the queued operation. **A seed that never ran was recorded as done.** The cache-warm skip added last round keyed on binding identity alone, so a backfill that failed — or that was gated because memory was off — left the binding looking seeded, and the blocks this machine already held never reached the workspace until a rebind. Seeding is now tracked by a `seededAt` marker written only after a sweep completes, and `backfill` reports `gated` so "never ran" is distinguishable from "ran and stored nothing". Also extracts `originSuffix()` so `formatBlock` and `formatTrainingEntry` cannot drift on the sibling-project label. Every guard is mutation-checked: removing the local-existence check, the archived-tombstone check, the seed marker, or the `gated` flag each fails a test. Tests: 4465 pass across memory + altimate, 5 new. Full suite is green — the subprocess smoke suites time out only under load and pass 84/84 on their own.
1 parent b1992aa commit ce874ec

6 files changed

Lines changed: 219 additions & 30 deletions

File tree

‎packages/opencode/src/altimate/workspace/memory-backfill.ts‎

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -23,19 +23,24 @@ const log = Log.create({ service: "altimate-workspace-memory-backfill" })
2323
* Covers both scopes: project blocks attach to the workspace just bound, and
2424
* global blocks go up account-level. A bind is the only moment global memory is
2525
* swept; blocks written later ride the ordinary per-write mirror. */
26-
export async function backfillOnBind(directory: string, binding: CachedBinding): Promise<void> {
27-
if (!isEnabled()) return
26+
export async function backfillOnBind(directory: string, binding: CachedBinding): Promise<boolean> {
27+
if (!isEnabled()) return false
2828
try {
2929
// The directory and binding are passed in rather than rediscovered. The
3030
// `link` subcommand binds from a plain yargs handler with no instance
3131
// context, so resolving project scope from the ambient instance throws
3232
// there — silently, because this catch turns it into a log line while the
3333
// CLI still prints "Linked". Reading project memory was the entire point.
3434
const blocks = await MemoryStore.listAll({ directory })
35-
if (blocks.length === 0) return
35+
if (blocks.length === 0) return true
3636
const result = await backfill(blocks, binding)
3737
log.info("workspace memory seeded after bind", result)
38+
// Only a sweep that stored everything it meant to counts as seeded. A
39+
// failure here must leave the binding eligible for a retry, or local blocks
40+
// stay absent from the workspace until a rebind or an unrelated edit.
41+
return !result.gated && result.failed === 0
3842
} catch (err) {
3943
log.warn("workspace memory backfill after bind failed", { err: String(err) })
44+
return false
4045
}
4146
}

‎packages/opencode/src/altimate/workspace/memory-sync.ts‎

Lines changed: 76 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -106,6 +106,8 @@ export function isEnabled(): boolean {
106106
* active instance; tests set it so they need not boot one. */
107107
export const syncInternals: {
108108
resolveBinding?: () => Promise<CachedBinding | null>
109+
/** Test seam for the local-existence check. Production reads the store. */
110+
blockExists?: (block: MemoryBlock) => Promise<boolean>
109111
} = {}
110112

111113
/** Instance.directory throws synchronously with no instance context, so a
@@ -296,6 +298,26 @@ type KnownRecords = { records: CloudMemoryRecord[]; truncated: boolean }
296298
* counting it as success made a sweep report blocks it had not stored. */
297299
type PushOutcome = "stored" | "unchanged" | "declined" | "skipped"
298300

301+
/** Is this block still in the local store?
302+
*
303+
* Imported lazily: ``@/memory/store`` reaches this module on its write path, so
304+
* a static import would close an eval-order cycle (see ./memory-backfill.ts).
305+
* A read failure answers "yes" — refusing to mirror on a transient read error
306+
* would silently drop a memory the user does have. */
307+
async function existsLocally(block: MemoryBlock): Promise<boolean> {
308+
if (syncInternals.blockExists) return syncInternals.blockExists(block)
309+
try {
310+
const { MemoryStore } = await import("@/memory/store")
311+
return !!(await MemoryStore.read(block.scope, block.id))
312+
} catch (err) {
313+
log.warn("could not confirm a block still exists locally; mirroring anyway", {
314+
id: block.id,
315+
err: String(err),
316+
})
317+
return true
318+
}
319+
}
320+
299321
async function push(
300322
block: MemoryBlock,
301323
binding: CachedBinding | null,
@@ -331,7 +353,29 @@ async function push(
331353
}
332354
}
333355

334-
const match = existing?.memoryId ?? view.records.find((r) => isSameBlock(r, block, binding))?.id
356+
// The local store is the authority on whether this block still exists.
357+
// `backfill` registers a block on the serialize queue only when a worker
358+
// dequeues it, so a delete issued mid-sweep runs first and this push would
359+
// otherwise undo it -- recreating a record the user deleted, or reviving an
360+
// archived one, and then marking it synced so no later sweep re-archives it.
361+
if (!(await existsLocally(block))) {
362+
log.warn("skipping mirror for a block that no longer exists locally", {
363+
id: block.id,
364+
scope: block.scope,
365+
})
366+
return "skipped"
367+
}
368+
369+
// An archived record is a tombstone. Ignoring the index here sends us to the
370+
// identity search below, which excludes archived records -- so a block the
371+
// user recreated under an old id gets a fresh record instead of un-archiving
372+
// the old one (`MemoryApi.update` replaces metadata wholesale, which would
373+
// drop the archived marker).
374+
const indexed = existing?.memoryId
375+
? view.records.find((r) => r.id === existing.memoryId)
376+
: undefined
377+
const indexUsable = existing?.memoryId && (!indexed || !isArchived(indexed))
378+
const match = (indexUsable ? existing?.memoryId : undefined) ?? view.records.find((r) => isSameBlock(r, block, binding))?.id
335379
if (match) {
336380
// Refuse to move a record backwards. Two machines editing the same block,
337381
// or a stale clone running a sweep, would otherwise overwrite a newer cloud
@@ -415,29 +459,37 @@ function serialize<T>(scope: "global" | "project", blockId: string, op: () => Pr
415459
* disabled. */
416460
export async function mirrorBlock(block: MemoryBlock): Promise<void> {
417461
if (!isEnabled()) return
418-
// A binding is required for EVERY scope, not just project. Memories are
419-
// associated with a workspace, and the workspace is what carries the
420-
// memory_enabled setting — mirroring from an unbound directory would upload
421-
// with nothing to consult and nothing to attribute it to. Global blocks still
422-
// carry no workspace themselves, so they apply everywhere on read; the
423-
// binding governs only whether we upload at all.
424-
const binding = await currentBinding()
425-
if (!binding) return
426-
if (!(await memoryEnabled(binding))) return
427-
await serialize(block.scope, block.id, () => push(block, binding))
462+
// Queued BEFORE the binding lookup, not after. Both are async, so resolving
463+
// them first let two operations on one block reach `serialize` in the
464+
// opposite order to the writes that triggered them.
465+
await serialize(block.scope, block.id, async () => {
466+
// A binding is required for EVERY scope, not just project. Memories are
467+
// associated with a workspace, and the workspace is what carries the
468+
// memory_enabled setting — mirroring from an unbound directory would upload
469+
// with nothing to consult and nothing to attribute it to. Global blocks still
470+
// carry no workspace themselves, so they apply everywhere on read; the
471+
// binding governs only whether we upload at all.
472+
const binding = await currentBinding()
473+
if (!binding) return
474+
if (!(await memoryEnabled(binding))) return
475+
await push(block, binding)
476+
})
428477
}
429478

430479
/** Archive a block's cloud record rather than deleting it, so the workspace
431480
* keeps the history. Only this client filters the marker — other readers do
432481
* not — so an archived record stays visible elsewhere. */
433482
export async function archiveBlock(scope: "global" | "project", blockId: string): Promise<void> {
434483
if (!isEnabled()) return
435-
const binding = await currentBinding()
436-
if (!binding) return
437-
if (!(await memoryEnabled(binding))) return
438484
// Queued behind any in-flight mirror for the same block, so a delete cannot
439-
// run before the create it is meant to undo.
440-
return serialize(scope, blockId, () => archiveNow(scope, blockId, binding))
485+
// run before the create it is meant to undo. The binding lookup happens
486+
// inside the queued op for the same reason as in `mirrorBlock`.
487+
return serialize(scope, blockId, async () => {
488+
const binding = await currentBinding()
489+
if (!binding) return
490+
if (!(await memoryEnabled(binding))) return
491+
await archiveNow(scope, blockId, binding)
492+
})
441493
}
442494

443495
async function archiveNow(
@@ -526,13 +578,16 @@ async function runQueue<T>(
526578
export async function backfill(
527579
blocks: MemoryBlock[],
528580
explicitBinding?: CachedBinding,
529-
): Promise<{ ok: number; failed: number; skipped: number; declined: number }> {
530-
if (!isEnabled()) return { ok: 0, failed: 0, skipped: 0, declined: 0 }
581+
): Promise<{ ok: number; failed: number; skipped: number; declined: number; gated: boolean }> {
582+
// ``gated`` says the sweep never ran, as opposed to running and storing
583+
// nothing. A caller recording "this binding is seeded" must be able to tell
584+
// those apart: memory being off is not a completed seed.
585+
if (!isEnabled()) return { ok: 0, failed: 0, skipped: 0, declined: 0, gated: true }
531586
// The bind path passes the binding it just recorded; there is no ambient
532587
// instance to resolve one from on the `link` subcommand.
533588
const binding = explicitBinding ?? (await currentBinding())
534589
if (!binding || !(await memoryEnabled(binding)))
535-
return { ok: 0, failed: 0, skipped: blocks.length, declined: 0 }
590+
return { ok: 0, failed: 0, skipped: blocks.length, declined: 0, gated: true }
536591
const index = await readIndex()
537592

538593
const pending: { block: MemoryBlock; binding: CachedBinding | null }[] = []
@@ -556,7 +611,7 @@ export async function backfill(
556611
pending.push({ block, binding: target })
557612
}
558613

559-
if (pending.length === 0) return { ok: 0, failed: 0, skipped, declined: 0 }
614+
if (pending.length === 0) return { ok: 0, failed: 0, skipped, declined: 0, gated: false }
560615

561616
// One read for the whole sweep. Every block in a first bind is an index miss,
562617
// so resolving each through its own lookup made a bind cost one full record
@@ -579,7 +634,7 @@ export async function backfill(
579634
// `skipped` combines blocks filtered before the queue (already synced, or
580635
// project blocks with no workspace) with those the queue itself declined to
581636
// act on — a truncated read, or a remote copy that is newer.
582-
const totals = { ...result, skipped: result.skipped + skipped }
637+
const totals = { ...result, skipped: result.skipped + skipped, gated: false }
583638
log.info("workspace memory backfill finished", totals)
584639
return totals
585640
}

‎packages/opencode/src/altimate/workspace/state.ts‎

Lines changed: 35 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,10 @@ export interface CachedBinding {
3030
repoRemote: string | null
3131
projectPath: string | null
3232
linkedAt: number
33+
/** Set once a bind-time seed completed without failures. Absent means the
34+
* seed has not run, errored, or was skipped because memory was off — all of
35+
* which must stay retryable, so a later warm sweeps again. */
36+
seededAt?: number
3337
}
3438

3539
interface CacheFile {
@@ -72,6 +76,8 @@ function isValidCacheFile(raw: unknown): raw is CacheFile {
7276
(typeof b.projectPath === "string" && b.projectPath.length > 0)
7377
if (!hasIdentity) return false
7478
if (typeof b.linkedAt !== "number") return false
79+
// A corrupt marker must not read as "already seeded" and suppress the sweep.
80+
if (b.seededAt !== undefined && typeof b.seededAt !== "number") return false
7581
}
7682
return true
7783
}
@@ -201,6 +207,21 @@ export async function readLocalBinding(directory: string): Promise<CachedBinding
201207
return null
202208
}
203209

210+
/** Record that this binding's seed completed, so later warms skip the sweep. */
211+
function markSeeded(directory: string, binding: CachedBinding): void {
212+
try {
213+
const cache = readCache()
214+
if (!cache) return
215+
const key = canonicalizeKey(directory)
216+
const current = cache.bindings[key]
217+
if (!current || !sameBinding(current, binding)) return
218+
cache.bindings[key] = { ...current, seededAt: Date.now() }
219+
writeCache(cache)
220+
} catch (err) {
221+
log.warn("could not record the workspace memory seed marker", { err: String(err) })
222+
}
223+
}
224+
204225
/** Same workspace and same project identity — i.e. nothing to re-seed.
205226
* ``linkedAt`` is deliberately ignored: it moves on every warm. */
206227
function sameBinding(a: CachedBinding, b: CachedBinding): boolean {
@@ -230,6 +251,7 @@ export async function recordApprovedBinding(
230251
// costs a full read of local memory and a round trip per block -- now paid
231252
// synchronously on the `link` path, which awaits the seed.
232253
let bindingChanged = true
254+
let alreadySeeded = false
233255
try {
234256
const existing = readCache()
235257
const cache: CacheFile =
@@ -238,7 +260,10 @@ export async function recordApprovedBinding(
238260
: { version: CACHE_VERSION, tenant: key.tenant, apiUrl: key.apiUrl, bindings: {} }
239261
const prior = cache.bindings[canonicalizeKey(directory)]
240262
bindingChanged = !prior || !sameBinding(prior, binding)
241-
cache.bindings[canonicalizeKey(directory)] = binding
263+
alreadySeeded = !bindingChanged && !!prior?.seededAt
264+
// Carry the seed marker across a warm so a completed seed is not repeated.
265+
cache.bindings[canonicalizeKey(directory)] =
266+
!bindingChanged && prior?.seededAt ? { ...binding, seededAt: prior.seededAt } : binding
242267
writeCache(cache)
243268
} catch (err) {
244269
// An unreadable cache means the prior binding is unknown, so fall back to
@@ -260,11 +285,19 @@ export async function recordApprovedBinding(
260285
// as a command handler returns (src/index.ts): a detached sweep is killed
261286
// mid-flight there, so a bind that reported success could seed nothing. The
262287
// TUI stays resident and leaves it detached so the dialog closes at once.
263-
if (!bindingChanged) return
288+
// Skip only when this exact binding has already been seeded successfully. A
289+
// warm after a failed or skipped seed must try again, or the blocks this
290+
// machine already holds never reach the workspace.
291+
if (alreadySeeded) return
264292
const seeded = import("./memory-backfill")
265293
.then((m) => m.backfillOnBind(canonicalizeKey(directory), binding))
294+
.then((ok) => {
295+
if (ok) markSeeded(directory, binding)
296+
return ok
297+
})
266298
.catch((err) => {
267299
log.warn("could not start workspace memory backfill", { err: String(err) })
300+
return false
268301
})
269302
if (opts?.awaitBackfill) await seeded
270303
else void seeded

‎packages/opencode/src/memory/prompt.ts‎

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -92,7 +92,7 @@ export namespace MemoryPrompt {
9292
export function formatBlock(block: MemoryBlock & { origin?: string }): string {
9393
const tagsStr = block.tags.length > 0 ? ` [${block.tags.join(", ")}]` : ""
9494
const expiresStr = block.expires ? ` (expires: ${block.expires})` : ""
95-
const originStr = block.origin ? ` — from workspace project \`${block.origin}\`` : ""
95+
const originStr = originSuffix(block.origin)
9696
let result = `### ${block.id} (${block.scope})${tagsStr}${expiresStr}${originStr}\n${block.content}`
9797

9898
if (block.citations && block.citations.length > 0) {
@@ -108,6 +108,13 @@ export namespace MemoryPrompt {
108108
}
109109

110110
/** Format a training entry for display (with applied count). */
111+
/** Label for a block that came from a sibling project in the workspace.
112+
* Shared so ``formatBlock`` and ``formatTrainingEntry`` cannot drift — they
113+
* rendered the same literal twice. */
114+
function originSuffix(origin?: string): string {
115+
return origin ? ` — from workspace project \`${origin}\`` : ""
116+
}
117+
111118
function formatTrainingEntry(block: MemoryBlock & { origin?: string }): string {
112119
const meta = parseTrainingMeta(block.content)
113120
const appliedStr = meta && meta.applied > 0 ? ` (applied ${meta.applied}x)` : ""
@@ -118,8 +125,7 @@ export namespace MemoryPrompt {
118125
// does. mergeOverlay deliberately keeps both blocks when a sibling shares an
119126
// id with this project's, so without this the model sees two identical
120127
// headings it cannot tell apart — the mis-reading the label exists to stop.
121-
const originStr = block.origin ? ` — from workspace project \`${block.origin}\`` : ""
122-
return `#### ${name}${appliedStr}${originStr}\n${content}`
128+
return `#### ${name}${appliedStr}${originSuffix(block.origin)}\n${content}`
123129
}
124130

125131
/** Score a block for relevance to the current agent context. */

‎packages/opencode/test/altimate/plugin/workspace.test.ts‎

Lines changed: 47 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -228,7 +228,7 @@ describe("workspace binding cache", () => {
228228
const originalFetch = globalThis.fetch
229229
globalThis.fetch = (async (_input?: unknown, _init?: unknown) => {
230230
calls++
231-
return new Response(JSON.stringify({ datamates: [] }), {
231+
return new Response(JSON.stringify({ datamates: [{ id: 9, name: "Warm", memory_enabled: true }] }), {
232232
status: 200,
233233
headers: { "Content-Type": "application/json" },
234234
})
@@ -253,6 +253,52 @@ describe("workspace binding cache", () => {
253253
}
254254
})
255255

256+
test("a seed that never ran stays retryable on the next warm", async () => {
257+
// Memory disabled at bind time means the sweep is a no-op, not a completed
258+
// seed. Treating it as done left the blocks this machine already holds
259+
// absent from the workspace until a rebind or an unrelated edit.
260+
const ORIGINAL_FLAG = process.env.ALTIMATE_WORKSPACE
261+
process.env.ALTIMATE_WORKSPACE = "1"
262+
const proj = path.join(SANDBOX, "gated-proj")
263+
mkdirSync(path.join(proj, ".altimate-code", "memory"), { recursive: true })
264+
const now = new Date().toISOString()
265+
writeFileSync(
266+
path.join(proj, ".altimate-code", "memory", "gated.md"),
267+
["---", "id: gated", "scope: project", `created: ${now}`, `updated: ${now}`, "---", "", "A fact.", ""].join("\n"),
268+
)
269+
const binding = {
270+
datamateId: 11,
271+
datamateName: "Gated",
272+
repoRemote: null,
273+
projectPath: proj,
274+
linkedAt: 1,
275+
}
276+
277+
let memoryEnabled = false
278+
let calls = 0
279+
const originalFetch = globalThis.fetch
280+
globalThis.fetch = (async (_input?: unknown, _init?: unknown) => {
281+
calls++
282+
return new Response(
283+
JSON.stringify({ datamates: [{ id: 11, name: "Gated", memory_enabled: memoryEnabled }] }),
284+
{ status: 200, headers: { "Content-Type": "application/json" } },
285+
)
286+
}) as typeof fetch
287+
288+
try {
289+
await recordApprovedBinding(proj, binding, { awaitBackfill: true })
290+
const afterGated = calls
291+
// Memory switched on later: the same binding must sweep this time.
292+
memoryEnabled = true
293+
await recordApprovedBinding(proj, { ...binding, linkedAt: 2 }, { awaitBackfill: true })
294+
expect(calls).toBeGreaterThan(afterGated)
295+
} finally {
296+
globalThis.fetch = originalFetch
297+
if (ORIGINAL_FLAG === undefined) delete process.env.ALTIMATE_WORKSPACE
298+
else process.env.ALTIMATE_WORKSPACE = ORIGINAL_FLAG
299+
}
300+
})
301+
256302
test("chmods the cache file to 0o600 after write", async () => {
257303
await recordApprovedBinding("/work/proj-a", {
258304
datamateId: 1,

0 commit comments

Comments
 (0)