Skip to content

fix(codex/app-server): stable mirror idempotency to prevent transcript loss#77046

Merged
steipete merged 2 commits into
openclaw:mainfrom
openperf:fix/issue-77012-mirror-idempotency-stable
May 4, 2026
Merged

fix(codex/app-server): stable mirror idempotency to prevent transcript loss#77046
steipete merged 2 commits into
openclaw:mainfrom
openperf:fix/issue-77012-mirror-idempotency-stable

Conversation

@openperf

@openperf openperf commented May 4, 2026

Copy link
Copy Markdown
Member

Summary

  • Problem: mirrorCodexAppServerTranscript builds its idempotency key as `${idempotencyScope}:${message.role}:${index}`, where index is the position of the message inside the per-turn messagesSnapshot. The snapshot itself is built dynamically by event-projector.ts:188 — it always starts with the user prompt, then conditionally pushes a Codex reasoning mirror, a Codex plan mirror, and the final assistant reply. If mirrorCodexAppServerTranscript runs more than once for the same logical turn (a retried turn, or any flow that re-mirrors the just-finished turn) and the second snapshot includes a reasoning record that the first didn't, every assistant-role record after the inserted slot shifts by one index. The original assistant entry's first-call key (:assistant:1) collides with the new reasoning entry's key (also :assistant:1), so the legitimately-new reasoning record is silently dropped at the dedupe gate, and the assistant content gets re-appended under :assistant:2 — the on-disk transcript ends up with a duplicate assistant entry and a missing reasoning entry. Files involved: extensions/codex/src/app-server/transcript-mirror.ts:33 and the call site in extensions/codex/src/app-server/run-attempt.ts:1782.

  • Root Cause: Positional index is not a stable identity for a logical message. The dedupe key needs to be invariant under snapshot reordering inside a fixed scope. The cross-turn dimension is already handled correctly by the existing call-site scope `codex-app-server:${threadId}:${turnId}` — distinct turns produce distinct scopes and therefore can never collide, even when role+content match (which is what preserves repeated-message turns such as a user typing "yes" twice in one thread).

  • Fix: A three-part change across three files:

    1. event-projector.ts — each entry in messagesSnapshot is now tagged with attachCodexMirrorIdentity before the snapshot is passed to mirrorCodexAppServerTranscript. Tags use a ${turnId}:${kind} shape (:prompt, :reasoning, :plan, :assistant) so each logical message carries a stable per-turn identity that is invariant under snapshot reordering and across re-invocations.
    2. transcript-mirror.ts — the dedupe key is rebuilt: when a message carries an explicit identity tag (set by attachCodexMirrorIdentity), the key is `${scope}:${identity}`; when no tag is present (fallback for callers that did not tag), the key falls back to `${scope}:${role}:${sha256({role,content}).slice(0,16)}`. The messages filter is narrowed via MirroredAgentMessage = Extract<AgentMessage, { role: "user" | "assistant" }>.
    3. run-attempt.ts — the call-site idempotencyScope is shortened from `codex-app-server:${threadId}:${turnId}` to `codex-app-server:${threadId}`. Per-turn distinction is now carried by the ${turnId}:${kind} identity tag on each individual entry, so the scope no longer needs to embed turnId. Dropping turnId from the scope is what makes a re-emitted prior-turn entry — which still carries its original ${turnId}:${kind} identity — collide with its existing on-disk key and become a true no-op.
  • What changed:

    • extensions/codex/src/app-server/event-projector.ts — import attachCodexMirrorIdentity; wrap each messagesSnapshot push (user prompt, reasoning mirror, plan mirror, assistant) with attachCodexMirrorIdentity(message, \${turnId}:${kind}`)`.
    • extensions/codex/src/app-server/transcript-mirror.ts — add attachCodexMirrorIdentity (exported, for event-projector.ts), readMirrorIdentity, fingerprintMirrorMessageContent, and buildMirrorDedupeIdentity; replace :${index} suffix with buildMirrorDedupeIdentity(message) (identity tag when present, content fingerprint fallback); narrow filter via the MirroredAgentMessage type predicate.
    • extensions/codex/src/app-server/run-attempt.ts — change idempotencyScope from `codex-app-server:${params.threadId}:${params.turnId}` to `codex-app-server:${params.threadId}`. Per-turn distinctness is now provided by the ${turnId}:${kind} identity tag on each entry; keeping turnId in the scope would be redundant and would break cross-turn re-emit idempotency.
    • extensions/codex/src/app-server/transcript-mirror.test.ts — update the three existing key-format assertions to match the identity-tag shape; add dedupes mirrored messages despite snapshot positional shifts (asserts the exact text sequence after a re-mirror that inserts a reasoning record; pre-fix code produces ["hello", "hi there", "hi there"] with reasoning dropped and assistant duplicated); add keeps repeated same-content turns distinct (verifies that two turns with identical role+content but different turnId segments produce two full user+assistant pairs, not one).
  • What did NOT change (scope boundary):

    • The on-disk JSONL schema is unchanged: entries still carry { type, id, parentId, timestamp, message } with message.idempotencyKey as a flat string. Pre-existing transcripts with the old :role:index keys remain valid; their old keys never collide with new identity-tag keys, which is the expected behaviour for a pre-existing entry under any dedupe scheme.
    • appendSessionTranscriptMessage, migrateLinearTranscriptToParentLinked, selectBoundedActiveTailRecords, the session write-lock, and the gateway-injected assistant write path are untouched.
    • Telegram and BlueBubbles channels (different mirror paths) are unaffected. No public API surface changes; mirrorCodexAppServerTranscript's parameter shape is identical. No any types are introduced.

Honest scope vs. #77012

This PR addresses one mechanism that produces transcript drift in the Codex app-server mirror: re-invocation of mirror for a logical turn whose snapshot has reordered between calls. The end-user-visible symptom in the linked issue is a sibling-branch on disk where a new user message shares its parentId with the prior assistant entry, after which the active-tail walker orphans the prior assistant. A hermetic two-turn mirror reproducer with the production per-turn snapshot pattern does not produce that sibling-branch state on current main — the parent chain stays well-formed, which means a single mirror call per turn with disjoint per-turn content is not by itself the proximate cause of the on-disk shape reported there. The within-scope index collision fixed in this PR is therefore one contributing factor, not the full root cause; this PR is intentionally Refs rather than Fixes.

Reproduction

Hermetic, against current main:

git clone https://github.com/openclaw/openclaw && cd openclaw
# Apply this PR
pnpm install
pnpm vitest run extensions/codex/src/app-server/transcript-mirror.test.ts
#   Test Files  1 passed (1)
#        Tests  9 passed (9)

Verify the new test is a real regression (it would fail without the fix):

# Temporarily revert just the production change
git stash push extensions/codex/src/app-server/transcript-mirror.ts
pnpm vitest run extensions/codex/src/app-server/transcript-mirror.test.ts
#   FAIL  > dedupes mirrored messages despite snapshot positional shifts
#     AssertionError: expected [ 'hello', 'hi there', 'hi there' ]
#       to deeply equal [ 'hello', 'hi there', '[Codex reasoning] thinking' ]
#   Tests:  4 failed | 5 passed (9)
git stash pop

Risk / Mitigation

  • Risk: a thread-only scope would collapse distinct same-content turns. Each entry's identity tag already includes turnId, so distinct turns produce distinct keys even when role+content match. Covered by keeps repeated same-content turns distinct.
  • Risk: hash collision (fallback path). SHA-256 truncated to 16 hex chars (64 bits) is well above any practical threshold for the small number of records inside a single (threadId, turnId) scope.
  • Risk: JSON.stringify field ordering (fallback path). The fingerprint hashes { role, content } in a fixed object literal — ordering is deterministic on every supported Node version (engines.node >= 22.14.0).
  • Risk: hook-driven content mutation. The fingerprint is computed before runAgentHarnessBeforeMessageWriteHook, so a hook that mutates content does not destabilize the dedupe anchor on retries.
  • Mitigation: Two regression tests cover both halves of the design — within-scope reorder (the bug being fixed) and cross-scope distinctness (the regression that mustn't be introduced). pnpm tsgo:extensions and pnpm tsgo:extensions:test are clean.

Change Type (select all)

  • Bug fix

Scope (select all touched areas)

  • Codex (app-server transcript mirror — extensions/codex/src/app-server/transcript-mirror.ts, event-projector.ts, run-attempt.ts)
  • Tests (regression coverage in transcript-mirror.test.ts)

Linked Issue/PR

Refs #77012 — addresses the within-scope index-collision contributing factor; the broader sibling-branch symptom in that issue likely involves at least one additional path and should remain open until that path is identified and fixed.

@clawsweeper

clawsweeper Bot commented May 4, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge.

Summary
Tags Codex app-server mirror messages with stable per-turn identities, switches transcript dedupe away from positional indexes, narrows the mirror scope to the thread, adds regression tests, and records the fix in the changelog.

Reproducibility: yes. source-reproducible: current main keys mirrored messages by role and snapshot index while the projector can insert optional reasoning or plan assistant records before the final assistant. I did not run tests in this read-only review.

Next step before merge
No repair lane is needed because the current head already includes the changelog entry and I found no narrow automated blocker; maintainers can proceed with normal review and validation.

Security
Cleared: The diff is limited to Codex transcript mirror logic, tests, and changelog, with no dependency, workflow, secret, package-resolution, or release-script changes.

Review details

Best possible solution:

Review and land this stable Codex mirror idempotency fix after normal validation, while keeping #77012 open for any remaining transcript-loss cause.

Do we have a high-confidence way to reproduce the issue?

Yes, source-reproducible: current main keys mirrored messages by role and snapshot index while the projector can insert optional reasoning or plan assistant records before the final assistant. I did not run tests in this read-only review.

Is this the best way to solve the issue?

Yes. Stable ${turnId}:${kind} mirror identities are the narrow maintainable dedupe anchor, and the added tests cover positional shifts plus repeated same-content turns; the broader linked report should remain open for other causes.

What I checked:

Likely related people:

  • steipete: Local blame for the central transcript-mirror and event-projector lines attributes the available history boundary for this Codex mirror path to Peter Steinberger. (role: recent maintainer / transcript persistence owner; confidence: medium; commits: 061af13bf3fc; files: extensions/codex/src/app-server/transcript-mirror.ts, extensions/codex/src/app-server/event-projector.ts)
  • Eva: Current main HEAD by Eva is the latest local touch for extensions/codex/src/app-server/run-attempt.ts, the call site whose mirror scope changes in this PR. (role: recent adjacent maintainer; confidence: medium; commits: cb3853587576; files: extensions/codex/src/app-server/run-attempt.ts)

Remaining risk / open question:

  • I did not run the PR's targeted tests because this review is read-only.
  • The broader linked transcript-loss report remains open because this PR intentionally addresses one mirror idempotency drift mode, not every possible sibling-branch cause.

Codex review notes: model gpt-5.5, reasoning high; reviewed against cb3853587576.

@openperf
openperf force-pushed the fix/issue-77012-mirror-idempotency-stable branch from 912a20d to a74117b Compare May 4, 2026 11:02
@openperf
openperf force-pushed the fix/issue-77012-mirror-idempotency-stable branch from a74117b to f411534 Compare May 4, 2026 13:36
@steipete
steipete merged commit 14aa988 into openclaw:main May 4, 2026
88 checks passed
@steipete

steipete commented May 4, 2026

Copy link
Copy Markdown
Contributor

Landed via squash merge.

  • Gate: GitHub CI/check rollup green at source SHA 46e4dd7
  • Land commit: 14aa988

Thanks @openperf!

openperf added a commit that referenced this pull request May 5, 2026
openperf added a commit that referenced this pull request May 5, 2026
…Unreleased (#77728)

Merged via squash.

Prepared head SHA: 1bd228f
Co-authored-by: openperf <[email protected]>
Co-authored-by: openperf <[email protected]>
Reviewed-by: @openperf
vincentkoc added a commit to VintageAyu/openclaw that referenced this pull request May 5, 2026
…ainer-hardening

* origin/main: (843 commits)
  docs(changelog): relocate openclaw#77046 and openclaw#77280 entries from 2026.5.3 to Unreleased (openclaw#77728)
  docs: reorder unreleased changelog
  fix: expose ollama thinking profile before activation (openclaw#77617) (thanks @yfge)
  fix: expose ollama thinking profile before activation
  test(gateway): preserve dispatch timers in waiter
  test(gateway): keep startup context timer live
  docs: document cache-friendly activity helper
  ci: install ffmpeg for Mantis media previews
  fix: avoid impossible device token rotation advice (openclaw#77688) (thanks @Conan-Scott)
  docs(changelog): note doctor device pairing advice fix
  fix(doctor): avoid impossible device token rotation advice
  ci: use Crabbox media previews for Mantis
  docs: filter maintainer-owned triage noise
  test: cover GitHub activity helper
  fix(session-file-repair): drop null-role message entries instead of preserving them (openclaw#77288)
  fix: prune orphan session artifacts
  perf: reduce GitHub activity cache misses
  fix: cache session list model resolution (openclaw#77650) (thanks @ragesaq)
  ci: embed Mantis desktop previews
  fix(replay-history): drop trailing stream-error placeholder before provider send (openclaw#77287)
  ...

# Conflicts:
#	CHANGELOG.md
lxe pushed a commit to lxe/openclaw that referenced this pull request May 6, 2026
…t loss (openclaw#77046)

* fix(codex/app-server): stable mirror idempotency to prevent transcript loss

* Changelog: note codex/app-server transcript mirror dedupe stabilization (openclaw#77046)
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request May 9, 2026
…t loss (openclaw#77046)

* fix(codex/app-server): stable mirror idempotency to prevent transcript loss

* Changelog: note codex/app-server transcript mirror dedupe stabilization (openclaw#77046)
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request May 9, 2026
…rom 2026.5.3 to Unreleased (openclaw#77728)

Merged via squash.

Prepared head SHA: 1bd228f
Co-authored-by: openperf <[email protected]>
Co-authored-by: openperf <[email protected]>
Reviewed-by: @openperf
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request May 24, 2026
…t loss (openclaw#77046)

* fix(codex/app-server): stable mirror idempotency to prevent transcript loss

* Changelog: note codex/app-server transcript mirror dedupe stabilization (openclaw#77046)
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request May 24, 2026
…rom 2026.5.3 to Unreleased (openclaw#77728)

Merged via squash.

Prepared head SHA: 1bd228f
Co-authored-by: openperf <[email protected]>
Co-authored-by: openperf <[email protected]>
Reviewed-by: @openperf
jameslcowan pushed a commit to jameslcowan/openclaw that referenced this pull request Jun 2, 2026
…t loss (openclaw#77046)

* fix(codex/app-server): stable mirror idempotency to prevent transcript loss

* Changelog: note codex/app-server transcript mirror dedupe stabilization (openclaw#77046)
jameslcowan pushed a commit to jameslcowan/openclaw that referenced this pull request Jun 2, 2026
…rom 2026.5.3 to Unreleased (openclaw#77728)

Merged via squash.

Prepared head SHA: 1bd228f
Co-authored-by: openperf <[email protected]>
Co-authored-by: openperf <[email protected]>
Reviewed-by: @openperf
sablehead pushed a commit to sablehead/openclaw that referenced this pull request Jun 10, 2026
…t loss (openclaw#77046)

* fix(codex/app-server): stable mirror idempotency to prevent transcript loss

* Changelog: note codex/app-server transcript mirror dedupe stabilization (openclaw#77046)
sablehead pushed a commit to sablehead/openclaw that referenced this pull request Jun 10, 2026
…rom 2026.5.3 to Unreleased (openclaw#77728)

Merged via squash.

Prepared head SHA: 1bd228f
Co-authored-by: openperf <[email protected]>
Co-authored-by: openperf <[email protected]>
Reviewed-by: @openperf
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants