fix(agents): dedupe transcript rewrite suffix replay by dripsmvcp · Pull Request #89387 · openclaw/openclaw

dripsmvcp · 2026-06-02T08:34:17Z

Summary

Context-overflow recovery re-runs transcript suffix replay without a replay-identity guard, so byte-identical role=user entries are re-appended on every recovery pass and the persisted session JSONL accumulates duplicate messages (issue #66443).

Collapses only the proven overflow-clone pattern during replay: byte-identical role=user messages and byte-identical compactions. Exact suffix history is preserved for assistant, tool-result, and custom entries.
A legitimately repeated user message carries a distinct timestamp, so the full-payload hash keeps it while collapsing the identical recovery clone; compaction identity folds in the remapped firstKeptEntryId so two distinct compactions are not merged.
Intended outcome: overflow recovery becomes idempotent for the clone pattern without violating the rewrite's "preserve exact suffix history" contract for legitimate entries.
Intentionally out of scope: the broader duplicate-transcript umbrella, and repeated-bootstrap-custom dedup (custom entries are deliberately left untouched to avoid dropping legitimate history).
Reviewers should focus on the user/compaction scoping and the timestamp-based clone-vs-repeat distinction.

Linked context

Closes #66443

Related #79150 — superseded prior attempt; this PR addresses its "include the kept boundary in compaction identity" review note.

Not maintainer-requested; selected from the clawsweeper:queueable-fix backlog.

Real behavior proof (required for external PRs)

Behavior or issue addressed: Overflow recovery duplicated role=user entries in the persisted session JSONL (issue Overflow recovery duplicates role=user messages in session JSONL, amplifying transcript growth #66443 captured one unique message stored as three byte-identical copies).
Real environment tested: Linux / Node 24, exercising the actual production functions rewriteTranscriptEntriesInSessionManager and rewriteTranscriptEntriesInState against a real in-memory SessionManager and persisted transcript-file-state — these functions are run for real, not mocked.
Exact steps or command run after this patch: node scripts/run-vitest.mjs src/agents/embedded-agent-runner/transcript-rewrite.test.ts
Evidence after fix (screenshot, recording, terminal capture, console output, redacted runtime log, linked artifact, or copied live output): copied terminal output. The regression reproduces the bug on unfixed main and passes after the fix, and added tests prove legitimate repeats survive:

unfixed main: × dedupes byte-identical replayed messages ...
              AssertionError: expected 2 to be 1   (the clone is re-appended)

with the fix: Test Files  1 passed (1)
              Tests       10 passed (10)           (clone collapsed; legitimate repeats kept)

Observed result after fix: after the overflow-style suffix replay, a byte-identical role=user clone is collapsed to one entry, while (a) two user messages with distinct timestamps, (b) byte-identical repeated assistant/tool-result entries, and (c) two distinct compactions are all preserved.
What was not tested: a live gateway context-overflow on a running OpenClaw deployment (see limitations).
Proof limitations or environment constraints: this is unit/integration-level proof against the real production functions, which CONTRIBUTING.md treats as supplemental rather than a live-setup behavior capture. I could not drive a live gateway into context overflow in this environment (it needs a full running OpenClaw + provider + a forced overflow, and the sandbox network blocked both a clean install and the crabbox check:changed gate). A maintainer with a live setup can confirm end-to-end behavior, or apply proof: override for this logic-only dedup fix; I am happy to add a live capture if someone can reproduce an overflow.
Before evidence (optional but encouraged): the failing assertion above (expected 2 to be 1) is the before-state captured from unfixed main.

Tests and validation

node scripts/run-vitest.mjs src/agents/embedded-agent-runner/transcript-rewrite.test.ts -> 10 passed (4 new).
Sibling callers: tool-result-truncation, context-engine-maintenance, compact.hooks, cli-runner.context-engine -> 126 passed.
tsgo -p tsconfig.core.json and core test types: no errors in the changed files (two unrelated pre-existing errors in src/config/io.ts and src/secrets/config-io.ts are present on main).
Regression coverage added in transcript-rewrite.test.ts: clone collapse (fails first with expected 2 to be 1, passes after); distinct compactions preserved; legitimate repeated user messages (distinct timestamps) preserved; byte-identical repeated assistant/tool-result entries preserved.

Risk checklist

Did user-visible behavior change? No — internal transcript maintenance that removes only byte-identical recovery clones.
Did config, environment, or migration behavior change? No.
Did security, auth, secrets, network, or tool execution behavior change? No.
Highest-risk area: collapsing genuinely distinct entries during replay (the merge-risk: session-state concern).
How is that risk mitigated? Dedup is constrained to role=user messages and compactions only; the full-payload hash (including timestamp) distinguishes a recovery clone from a legitimate repeat; compaction identity includes the remapped firstKeptEntryId; assistant/tool-result/custom entries are never deduped. Covered by the four regression tests above.

Current review state

Next action: maintainer review / ClawSweeper re-review (auto-triggered by this push).
Addressed ClawSweeper P1 "Preserve distinct replayed transcript entries": constrained the replay dedup to the proven user-message + compaction clone pattern (per the recommended "Constrain Replay Dedupe" option), restored exact-history for assistant/tool/custom entries, and added regression coverage proving legitimate repeated messages/tool entries survive. The dedup helper is shared by both the SessionManager and TranscriptFileState rewrite paths.
Still open: real behavior proof is unit-level only; a live-overflow capture needs a real setup or a maintainer proof: override.

clawsweeper · 2026-06-02T08:36:27Z

Codex review: needs real behavior proof before merge. Reviewed June 29, 2026, 7:56 PM ET / 23:56 UTC.

Summary
The PR adds replay-identity dedupe to transcript rewrite suffix replay for byte-identical user-message and compaction clones, with regression tests for preserving legitimate repeated entries.

PR surface: Source +81, Tests +173. Total +254 across 2 files.

Reproducibility: yes. at source level: current main replays every transcript rewrite suffix entry, and the linked issue provides persisted JSONL/log evidence. I did not run a live forced-overflow reproduction in this read-only review.

Review metrics: 1 noteworthy metric.

Rewrite replay loops guarded: 2 changed. Both the in-memory and persisted transcript rewrite paths now skip replay identities, so reviewers need to check the shared session-state invariant in both loops.

Stored data model
Persistent data-model change detected: serialized state: src/agents/embedded-agent-runner/transcript-rewrite.ts. Confirm migration or upgrade compatibility proof before merge.

Root-cause cluster
Relationship: fixed_by_candidate
Canonical: #66443
Summary: This PR is the active candidate fix for the canonical Track A overflow transcript-replay/idempotency issue; earlier implementation attempts are closed unmerged, and the broader duplicate transcript umbrella remains adjacent.

Members:

canonical: Overflow recovery duplicates role=user messages in session JSONL, amplifying transcript growth #66443 - The PR body uses closing syntax for the open issue, and the issue describes the same overflow recovery role=user duplication in persisted session JSONL.
partial_overlap: Umbrella: duplicate transcript, replay, and context assembly across channels #69208 - The umbrella tracks broader duplicate transcript and replay symptoms, while this PR targets one Track A transcript rewrite replay path.
superseded: fix(agents): dedupe transcript rewrite replay #79150 - The closed unmerged prior PR targeted transcript rewrite replay dedupe and is explicitly cited as superseded by this PR.
superseded: fix(transcript-rewrite): dedupe identity-equal suffix entries on replay (#66443) #72511 - The closed unmerged earlier PR targeted the same issue with broader identity rules and was superseded by later attempts.
partial_overlap: [Bug]: MSTeams DM inbound messages appear duplicated in agent session context #67323 - The closed Teams duplicate context report is part of the duplicate transcript family but concerns a channel-visible symptom rather than this exact overflow rewrite clone path.

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🦪 silver shellfish
Proof: 🦪 silver shellfish
Patch quality: 🐚 platinum hermit
Result: blocked until real behavior proof from a real setup is added.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

[P1] Add redacted persisted-session or live overflow proof showing before/after transcript counts, with private paths, tokens, phone numbers, IPs, and endpoints removed.
Rebase or otherwise resolve the conflict against current main and rerun focused transcript rewrite validation.

Proof guidance:

[P1] Needs real behavior proof before merge: The PR body supplies copied unit/integration test output against production functions, but no redacted persisted-session capture, live overflow run, screenshot, logs, or maintainer proof override. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.

Risk before merge

[P1] GitHub currently reports the PR as dirty/conflicting against main, so the final merge result is not reviewable until it is rebased or otherwise resolved.
[P1] The PR intentionally skips and remaps persisted session transcript entries; merge still needs maintainers to own the session-state upgrade risk and proof bar.

Maintainer options:

Rebase and prove persisted-session behavior (recommended)
Resolve the dirty merge state and add redacted persisted-session or live overflow output showing duplicate user rows collapse while legitimate repeats remain.
Accept source-level proof by maintainer override
A maintainer can explicitly accept the targeted tests as enough for this logic-only transcript rewrite change while owning the remaining session-state proof risk.
Replace with a fresh narrow branch
If conflict resolution would be risky, keep the canonical issue open and replace this stale branch with a fresh implementation on current main.

Next step before merge

[P1] The remaining blockers are conflict resolution, contributor real-behavior proof or maintainer override, and session-state risk ownership rather than a narrow mechanical patch defect.

Security
Cleared: The diff uses Node's built-in crypto hashing and transcript tests only; no dependency, workflow, secret, network, package, or command-execution surface changes were found.

Review details

Best possible solution:

Rebase onto current main, preserve current transcript rewrite branch/leaf behavior, keep the narrow replay dedupe, and merge only after redacted persisted-session/live overflow proof or an explicit maintainer proof override.

Do we have a high-confidence way to reproduce the issue?

Yes at source level: current main replays every transcript rewrite suffix entry, and the linked issue provides persisted JSONL/log evidence. I did not run a live forced-overflow reproduction in this read-only review.

Is this the best way to solve the issue?

Yes as a scoped repair shape: the shared transcript rewrite seam is the narrow maintainable layer because overflow tool-result truncation and context-engine maintenance both call it. The PR is not merge-ready until conflict resolution and real-behavior proof or maintainer override are handled.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 18b2ff683fd3.

Label changes

Label justifications:

P1: The PR targets a confirmed overflow recovery bug that can duplicate persisted session transcript state and worsen real agent workflows.
merge-risk: 🚨 session-state: The patch changes how transcript suffix entries are skipped and remapped during session rewrites, which can alter persisted session history if the identity is too broad.
rating: 🦪 silver shellfish: Overall readiness is 🦪 silver shellfish; proof is 🦪 silver shellfish and patch quality is 🐚 platinum hermit.
status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs real behavior proof before merge: The PR body supplies copied unit/integration test output against production functions, but no redacted persisted-session capture, live overflow run, screenshot, logs, or maintainer proof override. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.

Evidence reviewed

PR surface:

Source +81, Tests +173. Total +254 across 2 files.

View PR surface stats

Area	Files	Added	Removed	Net
Source	1	98	17	+81
Tests	1	173	0	+173
Docs	0	0	0	0
Config	0	0	0	0
Generated	0	0	0	0
Other	0	0	0	0
Total	2	271	17	+254

What I checked:

Repository policy read: Root AGENTS.md and scoped src/agents/AGENTS.md were read fully; their high-confidence PR review, session-state, and agent-test guidance shaped this review. (AGENTS.md:1, 18b2ff683fd3)
PR live state: Live GitHub reports the PR open, non-draft, maintainer-modifiable, with head 2e18181, mergeable CONFLICTING and mergeStateStatus DIRTY, and a closing reference to the canonical issue. (2e18181289c6)
Patch read: The PR adds replay identity hashing and duplicate skip/remap behavior to both rewriteTranscriptEntriesInSessionManager and rewriteTranscriptEntriesInState, plus four focused SessionManager regression tests. (src/agents/embedded-agent-runner/transcript-rewrite.ts:38, 2e18181289c6)
Current main in-memory replay: Current main branches from the first replacement and appends every in-memory suffix entry without a replay identity guard. (src/agents/embedded-agent-runner/transcript-rewrite.ts:248, 18b2ff683fd3)
Current main persisted-state replay: Current main mirrors the unguarded suffix replay in rewriteTranscriptEntriesInState and returns every appended entry for persistence. (src/agents/embedded-agent-runner/transcript-rewrite.ts:401, 18b2ff683fd3)
Overflow caller: Tool-result truncation calls rewriteTranscriptEntriesInSessionManager and rewriteTranscriptEntriesInState, matching the reported context-overflow recovery path. (src/agents/embedded-agent-runner/tool-result-truncation.ts:936, 18b2ff683fd3)

Likely related people:

jalehman: Commit 751d5b7 added context-engine transcript maintenance and the transcript rewrite path this PR changes. (role: introduced transcript rewrite seam; confidence: high; commits: 751d5b7849ca; files: src/agents/pi-embedded-runner/transcript-rewrite.ts, src/agents/pi-embedded-runner/context-engine-maintenance.ts)
Tyler Yust: Commit 0deb8b0 added recovery from context overflow caused by oversized tool results, the caller path that reaches transcript rewrite during overflow handling. (role: introduced overflow recovery path; confidence: medium; commits: 0deb8b0da187; files: src/agents/pi-embedded-runner/tool-result-truncation.ts, src/agents/embedded-agent-runner/tool-result-truncation.ts)
Harjoth Khara: Current-main blame for both embedded transcript rewrite replay loops is carried by a recent embedded-runner refactor/carry-forward commit. (role: recent area contributor; confidence: medium; commits: 355c43fe0c9e; files: src/agents/embedded-agent-runner/transcript-rewrite.ts)

What the crustacean ranks mean

🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works

ClawSweeper keeps one durable marker-backed review comment per issue or PR.
Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
Maintainers can also comment @clawsweeper review to request a fresh review only.
Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

Review history (1 earlier review cycle)

reviewed 2026-06-21T10:23:57.091Z sha 2e18181 :: needs real behavior proof before merge. :: none

dripsmvcp · 2026-06-02T10:59:36Z

@clawsweeper

Context-overflow recovery re-runs transcript suffix replay (rewriteTranscriptEntriesInSessionManager / rewriteTranscriptEntriesInState) without a replay-identity guard, so byte-identical role=user entries are re-appended on every recovery pass. The persisted session JSONL then accumulates duplicate messages, amplifying transcript growth and cascading into further overflows. Collapse only the proven overflow-clone pattern during replay: byte-identical role=user messages (whose timestamp distinguishes a recovery clone from a legitimate repeat) and byte-identical compactions (identity folds in the remapped firstKeptEntryId so distinct kept boundaries survive). Exact suffix history is preserved for assistant, tool-result, and custom entries, whose identical payloads can be legitimate history. Closes openclaw#66443

openclaw-barnacle · 2026-07-20T04:39:00Z

This pull request has been automatically marked as stale due to inactivity.
Please add updates or it will be closed.

openclaw-barnacle Bot added agents Agent runtime and tooling size: S triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels Jun 2, 2026

openclaw-barnacle Bot added proof: supplied External PR includes structured after-fix real behavior proof. and removed triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels Jun 2, 2026

dripsmvcp force-pushed the fix/66443-transcript-rewrite-dedupe branch from f0591e3 to 2e18181 Compare June 3, 2026 00:39

openclaw-barnacle Bot added size: M and removed size: S labels Jun 3, 2026

clawsweeper Bot mentioned this pull request Jun 13, 2026

Overflow recovery duplicates role=user messages in session JSONL, amplifying transcript growth #66443

Closed

clawsweeper Bot mentioned this pull request Jun 26, 2026

Umbrella: duplicate transcript, replay, and context assembly across channels #69208

Open

revision-co-ltd mentioned this pull request Jul 10, 2026

Dead branches from transcript-rewrite (overflow recovery) are never GC'd, and compaction rotation copies them whole into every successor file — rotation frequency increases monotonically #103934

Closed

ObliviateRickLin mentioned this pull request Jul 11, 2026

fix(agents): collect unreachable transcript branches during compaction rotation #104034

Closed

openclaw-barnacle Bot added the stale Marked as stale due to inactivity label Jul 20, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

fix(agents): dedupe transcript rewrite suffix replay#89387

fix(agents): dedupe transcript rewrite suffix replay#89387
dripsmvcp wants to merge 1 commit into
openclaw:mainfrom
dripsmvcp:fix/66443-transcript-rewrite-dedupe

dripsmvcp commented Jun 2, 2026 •

edited

Loading

Uh oh!

clawsweeper Bot commented Jun 2, 2026 •

edited

Loading

Uh oh!

dripsmvcp commented Jun 2, 2026

Uh oh!

openclaw-barnacle Bot commented Jul 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

dripsmvcp commented Jun 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Linked context

Real behavior proof (required for external PRs)

Tests and validation

Risk checklist

Current review state

Uh oh!

clawsweeper Bot commented Jun 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dripsmvcp commented Jun 2, 2026

Uh oh!

openclaw-barnacle Bot commented Jul 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

dripsmvcp commented Jun 2, 2026 •

edited

Loading

clawsweeper Bot commented Jun 2, 2026 •

edited

Loading