Skip to content

Commit 6123be3

Browse files
lidge-junntdatt812
andauthored
fix(history): resolve a rollout's session_meta by thread id (#3026) (#3103)
* fix(history): resolve a rollout's session_meta by thread id A forked or branched session appends the SOURCE thread's session_meta after its own. codex-rs `apply_session_meta_from_item` discards any record whose payload id is not the canonical thread id, so that trailing line is ordinary rollout content -- but three call sites folded the file with the id-agnostic `readLatestSessionMeta` and answered with the foreign thread's provider. The writer skipped such a file outright, while the database row still flipped. That pair is unrestorable: the manifest expects an OpenCodex post-image the file never received, so `snapshotRolloutForRestore` throws history_backup_rollout_postimage_mismatch. `preflightRestoreTargets` is all-or-nothing, so a handful of forked threads blocks every entry, `ocx stop` exits 1 on every invocation, and `ocx update` aborts at its gate. The state never clears on retry. Resolve the record by id in the writer and in both validators. The append now carries this thread's own metadata, which the app honors, instead of cloning a foreign record it would discard. Fixes #3026 * fix(history): stop restore from erasing a user's first message Carries @ntdat812's 25ce1ed for the id-aware half of #3026 and adds the second half the issue reports. has_user_event has two writers, so a final state cannot establish who wrote it. Routing derives the post-image value from first_user_message AT SNAPSHOT TIME, but restore recomputed it from the message as it is now. A row routed while its message was null, then given a first message by the user, looked exactly like OpenCodex's own write - and restore put it back to 0. The manifest entry now records whether the message was non-empty when the snapshot was taken. A boolean only, never the text: the manifest is a file on disk and the message is user content. Ownership is decided by which tuple the row wears. The recorded original and the expected post-image are unambiguous. Drift from 0 to 1 on the routed tuple is the user's, because a row wearing that tuple was written by OpenCodex and anything on top of it followed. Drift on the original tuple is decided by a per-entry relabel marker written before the routing write and resolved after it, so a crash between the two leaves an honest unknown that the observed row resolves. One cell genuinely cannot be decided: an OpenAI-origin entry whose routing would have produced a 1, wearing the original tuple, with the marker still pending. Routing-never-landed-plus-activity and routing-landed-then-legacy- recovery produce an identical row and nothing durable separates them. That refuses, which is what dev already does there. A guess would either erase real activity or fabricate it. Manifests written before these fields exist keep the current-row reading and restore exactly as they do today - refusing them would brick the population the fix exists to repair. * fix(history): finish the manifest v2 migration and reopen stale markers Review of the has_user_event work found three holes. A surviving manifest means a previous route/restore cycle did not consume it, and its relabel marker describes that attempt. Carrying a stale "committed" into a new routing attempt let a later restore treat it as proof that OpenCodex authored an event flag the user had since set. Every new attempt now reopens the marker as pending; the recorded snapshot itself stays, since it is the original provenance. Manifest v2 existed only in the type. The validator still rejected version 2 and the writer still emitted version 1, so the new fields were being written into a manifest that claimed not to have them. The validator now accepts 1 and 2 and checks both optional fields when present - an unvalidated hadFirstUserMessage: "false" is truthy and would have selected the wrong restore verdict. The classifier's D branch matched any opencodex row, so C and D both matched an exec-origin row and D won only by being tested first. D is now the tuple routing actually produces: routeOpenai keeps the source, routeExec moves exec to cli. The plan asked for a state-transition matrix and I had shipped two cases. There are now sixteen, one per cell, and writing them found the exec-origin post-image case where routeExec always writes 1 - so that row is B, not D. * fix(history): re-snapshot a manifest that outlived its restore Review found the reopen fix incomplete and its regression vacuous. The regression let an ordinary restore consume the manifest, so the second route saw no existing entry and the reopen code never ran. It now forces the consume to fail with setBeforeHistoryBackupConsumeForTests, which is the only way a manifest actually survives into a second routing attempt. That exposed the real gap. Reopening the marker was not enough: the entry also carries hadFirstUserMessage and hasUserEvent, and both described the PREVIOUS attempt. The stale message flag made the new routed row match the expected post-image, and the stale event value restored the thread to a state two events old - a restore that had already happened, plus whatever the user did afterwards. Both are re-recorded for the new attempt. The provenance tuple stays, because it is the original and a routed row must never overwrite it; the event baseline is only refreshed when the row is back at that tuple. Manifest promotion was also incomplete: only a brand-new manifest started at version 2, so an existing v1 kept writing the new fields under a schema that did not declare them. Any manifest written forward is now v2. The regression asserts the whole sequence: route, failed consume, user types, route again, restore - and the user's first message survives. Red without the refreshed baseline. * fix(history): prove a relabel was undone before trusting a fresh baseline Third review round found the baseline refresh resting on the same inference the classifier already refuses: tuple equality does not establish ownership. Route to opencodex, legacy recovery back to openai, and the row wears the original tuple with a 1 that OpenCodex wrote - refreshing there adopts that as the user's baseline and the next restore keeps an event the user never generated. So the refresh now requires proof, and the proof is a relabel marker reading "none". Nothing wrote that value before; a restore that lands and passes its readback now records it on the surviving manifest, which is exactly the state where a later attempt needs to tell "already restored" from "still mid-route". Absent the proof the recorded baseline is kept: one event stale still restores to a state the user was in, while a wrong refresh silently rewrites what OpenCodex owns. The write happens after finalization fails rather than before consuming it - rewriting the manifest ahead of the consume changes the fingerprint the consume checks, which broke twelve tests when I tried it that way. Adds the legacy-return row to the classifier matrix, which is the history the audits kept circling: committed marker, expected-event-1 route, original tuple, restores to 0. * fix(history): refuse an ambiguous reroute instead of picking a baseline Fourth review round: the "none" proof is best-effort, so a failed write leaves the entry reading committed while the row has drifted. Keeping the recorded baseline erases the user's event; refreshing it preserves one OpenCodex authored. That is another undecidable cell, and the reviewer was right that neither reading is the safe default. A reroute in that state now refuses with history_apply_ambiguous_reroute rather than committing to an interpretation. Nothing is written, the manifest stays for a human, and the refusal surfaces as an integrity failure with zero rows applied. Adds the end-to-end legacy-return regression the reviewer asked for: route writes opencodex/vscode/1, legacy recovery pulls it back to openai/vscode/1, and the row now wears its original tuple carrying an event OpenCodex wrote. The classifier row alone could not express this history - it needed the real route and the real recovery. The assertion is the contract rather than one implementation: the 1 must not survive as the user's, so either restore returns it to 0 or the layer refuses and keeps the manifest. Drops the duplicate classifier row that added no coverage over its neighbour. * fix(history): refuse only the reroute that is genuinely undecidable Fifth review round found the refusal too broad and the legacy-return test green for the wrong reason. The guard rejected every unproven event mismatch, including the case where the previous route would have written 0 - there an observed 1 can only be the user's, which is the expected-event-0 cell the classifier deliberately preserves. Refusal is now conditioned on whether the prior route could itself have authored the difference. The end-to-end legacy-return test passed because the fixture's rollout omits source, so restore refused during rollout preflight before the classifier mattered - a broken classifier would have passed it too. It now writes the matching session_meta so the real path runs, and asserts the exact outcome: the 1 returns to 0 and the manifest is consumed. Two smaller things: leftover debug logging is gone, and the specific integrity code now travels with the result. failureReason: "integrity" alone reads as "retry or run doctor", and history_apply_ambiguous_reroute is not retryable - it means two histories produced the same row and the manifest needs a human. * fix(history): decide a reroute by direction and origin, and route the reason out Sixth review round found the reroute guard collapsed to a single flag and the integrity code stopping at the worker boundary. The condition now names all five cells. A 1 to 0 drift is always foreign - nothing in this system clears the flag, so a baseline that moved down is a decision the manifest does not own. A 0 to 1 drift is the user's when the entry is exec-origin (routeExec moves source to cli and legacy recovery does not move it back, so it was never routed away and back), when a "none" marker proves the previous relabel was undone, or when the previous route would have written 0. It refuses when the previous route would have written 1, and when a legacy entry records nothing about it - undefined is not false. integrityCode existed on the sync result and went nowhere: HistoryWorkerResult and CodexHistoryJobOutcome had no field for it and classifyWorkerResult dropped it, so an operator still read "failed integrity checks; run doctor". It is typed and forwarded through both, and an ambiguous reroute now says what it is - the manifest cannot prove whether an earlier relabel was undone, nothing changed, resolve it manually rather than retrying. Retrying reaches the same refusal. * docs(devlog): record what wp3's seven review rounds changed about the plan --------- Co-authored-by: Nguyen Thanh Dat <[email protected]>
1 parent 9d122dd commit 6123be3

7 files changed

Lines changed: 623 additions & 29 deletions

File tree

‎devlog/_plan/260831_prio70_train_round2/030_wp3_forked_rollout_restore.md‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -394,3 +394,35 @@ scan on `ssh lidge`.
394394
`Closes #3026`. PR #3056 is 630 commits behind `dev`; carry its id-aware reader with
395395
credit to its author rather than rebasing that branch, the same way round 1's wp3
396396
carried Ingwannu's `aec717722`.
397+
398+
## What implementation added beyond this plan
399+
400+
Seven adversarial review rounds against the built branch (findings 3, 2, 1, 1, 2, 2, 0).
401+
The eleven planning rounds got the classifier right; every implementation finding was in
402+
the machinery around it, which is worth recording separately:
403+
404+
- **A surviving manifest has to be re-snapshotted, not just re-marked.** The plan said to
405+
reopen the relabel marker on a new routing attempt. That was half of it: the entry also
406+
carries `hadFirstUserMessage` and `hasUserEvent`, and both described the previous
407+
attempt. The stale message flag made the new routed row match the expected post-image;
408+
the stale event value restored the thread to a state two events old.
409+
- **Refreshing the baseline needs the same proof the classifier needs.** Tuple equality
410+
does not establish that the previous relabel was undone — route-then-legacy-recovery
411+
lands on the original tuple. A restore that lands and passes its readback now records
412+
`relabel: "none"`, which is the proof. Nothing wrote that value before.
413+
- **A sixth cell, and it refuses.** When the proof is absent and the row has drifted,
414+
neither reading is safe. That is the plan's undecidable shape arriving one layer up, in
415+
`rememberOriginal` rather than in the classifier, and it refuses the same way.
416+
- **The decision is direction and origin, not one flag.** Reverse drift is always foreign;
417+
`0 → 1` is the user's for an exec-origin entry, under a `none` marker, or when the
418+
previous route would have written 0; everything else refuses, and a legacy `undefined`
419+
is not `false`.
420+
- **The refusal has to reach a person.** `failureReason: "integrity"` alone reads as
421+
"retry or run doctor", and an ambiguous reroute is not retryable. The specific code now
422+
travels through the worker and job layers to an operator message that says the manifest
423+
needs manual resolution.
424+
425+
And two about verification: a regression that let an ordinary restore consume the manifest
426+
never reached the reopen path at all, and the legacy-return end-to-end test passed because
427+
the fixture's rollout omitted `source`, so restore refused at rollout preflight before the
428+
classifier was consulted. Both were green for reasons unrelated to what they claimed.

‎src/codex/history-job.ts‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -110,6 +110,8 @@ export type CodexHistoryJobOutcome =
110110
| { readonly kind: "blocked"; readonly reason: "busy" | "database" | "unsafe-path" | "desired_disabled" | "desired_enabled" }
111111
| { readonly kind: "failed"; readonly reason: "worker-error" | "worker-died" | "timeout";
112112
readonly message: string; readonly historyFailureReason?: CodexHistoryFailureReason;
113+
/** Specific integrity condition when `historyFailureReason` is `"integrity"`. */
114+
readonly historyIntegrityCode?: string;
113115
readonly rows?: number; readonly files?: number };
114116

115117
/**
@@ -258,6 +260,13 @@ export function describeHistoryJobFailure(
258260
return "permission was denied while writing Codex history; this is not a Codex app lock. Run 'ocx doctor'.";
259261
}
260262
if (outcome.historyFailureReason === "integrity") {
263+
// Not every integrity stop is a retry. An ambiguous reroute means two histories
264+
// produced the same row and no durable fact separates them, so retrying reaches the
265+
// same refusal - the manifest needs a person, and saying "run doctor" sends them the
266+
// wrong way.
267+
if (outcome.historyIntegrityCode === "history_apply_ambiguous_reroute") {
268+
return "a Codex history entry could not be re-routed because its manifest cannot prove whether an earlier relabel was undone; nothing was changed and the manifest was kept. Resolve it manually rather than retrying.";
269+
}
261270
return partiallyChanged
262271
? "the history backup or its restore target changed after a partial restore; the manifest was retained for review and safe retry. Run 'ocx doctor'."
263272
: "the history backup or its restore target failed integrity checks; no unverified provider metadata was applied. Run 'ocx doctor'.";
@@ -298,6 +307,7 @@ function classifyWorkerResult(result: HistoryWorkerResult): CodexHistoryJobOutco
298307
reason: "worker-error",
299308
message: redactWorkerMessage(result.message),
300309
...(result.reason ? { historyFailureReason: result.reason } : {}),
310+
...(result.integrityCode ? { historyIntegrityCode: result.integrityCode } : {}),
301311
...(result.rows !== undefined && result.files !== undefined
302312
? { rows: result.rows, files: result.files }
303313
: {}),

‎src/codex/history-manifest.ts‎

Lines changed: 35 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -10,10 +10,37 @@ export interface CodexHistoryBackupEntry {
1010
modelProvider: string;
1111
source: string;
1212
hasUserEvent: 0 | 1;
13+
/**
14+
* Whether the row had a non-empty `first_user_message` when the snapshot was taken.
15+
*
16+
* Routing derives the post-image `has_user_event` from the message AT SNAPSHOT TIME
17+
* (`history-provider.ts` `routeOpenai`), so a restore that recomputes it from the
18+
* message as it is NOW will mistake the user's first message for OpenCodex's own write
19+
* and erase it. Only the emptiness is recorded, never the text: this manifest is a file
20+
* on disk and the message is user content.
21+
*
22+
* Optional because manifests written before this field exists cannot be given one. An
23+
* entry without it falls back to the current-row reading, which is exactly as imprecise
24+
* as the behaviour it replaces and no worse.
25+
*/
26+
hadFirstUserMessage?: boolean;
27+
/**
28+
* Whether OpenCodex's routing relabel is known to have landed for this entry.
29+
*
30+
* `pending` is written before the routing write and resolved after it, so a crash
31+
* between the two leaves an honest "unknown" rather than a confident wrong answer. The
32+
* observed row resolves it: the recorded original means the write did not land, the
33+
* expected post-image means it did.
34+
*
35+
* Absent on entries written before the field existed. Those refuse only in the one
36+
* genuinely undecidable case — original tuple with `has_user_event` moved 0 to 1 —
37+
* which `dev` already refuses today.
38+
*/
39+
relabel?: "pending" | "committed" | "none";
1340
}
1441

1542
export interface CodexHistoryBackupManifest {
16-
version: 1;
43+
version: 1 | 2;
1744
stateDbPath: string;
1845
entries: Record<string, CodexHistoryBackupEntry>;
1946
}
@@ -76,7 +103,7 @@ export function validateCodexHistoryBackupManifest(
76103
expectedStateDbPath: string,
77104
): CodexHistoryManifestValidation {
78105
if (!isRecord(raw)
79-
|| raw.version !== 1
106+
|| (raw.version !== 1 && raw.version !== 2)
80107
|| typeof raw.stateDbPath !== "string"
81108
|| !raw.stateDbPath.trim()
82109
|| !isAbsolute(raw.stateDbPath)
@@ -103,6 +130,12 @@ export function validateCodexHistoryBackupManifest(
103130
|| typeof value.hasUserEvent !== "number"
104131
|| !Number.isSafeInteger(value.hasUserEvent)
105132
|| (value.hasUserEvent !== 0 && value.hasUserEvent !== 1)
133+
// Optional, but not unvalidated: a truthy `hadFirstUserMessage: "false"` would select
134+
// the wrong restore verdict, and an unrecognized `relabel` would be read as a state
135+
// the classifier does not have.
136+
|| (value.hadFirstUserMessage !== undefined && typeof value.hadFirstUserMessage !== "boolean")
137+
|| (value.relabel !== undefined
138+
&& value.relabel !== "pending" && value.relabel !== "committed" && value.relabel !== "none")
106139
|| !hasAllowedProvenance(value)) {
107140
return { ok: false, reason: "schema", scope: "entry-provenance" };
108141
}

0 commit comments

Comments
 (0)