Skip to content

fix(tui): track burst optimistic sends as a counter to prevent duplicate transcripts (NemoClaw #3145)#78969

Closed
chengjiew wants to merge 3 commits into
openclaw:mainfrom
chengjiew:fix/nemoclaw-3145-tui-burst-dedup
Closed

fix(tui): track burst optimistic sends as a counter to prevent duplicate transcripts (NemoClaw #3145)#78969
chengjiew wants to merge 3 commits into
openclaw:mainfrom
chengjiew:fix/nemoclaw-3145-tui-burst-dedup

Conversation

@chengjiew

@chengjiew chengjiew commented May 7, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Problem: In openclaw tui, sending several messages back-to-back (each within ~1s) renders every user prompt and every reply twice and out of send-order. Reproduced on Linux (Ubuntu 24.04, OpenClaw 2026.4.24); reported in NemoClaw#3145.
  • Why it matters: Burst sending is a normal interaction pattern; the TUI is the primary chat surface for sandboxed setups (NemoClaw, Brev, etc.) and a duplicated/reordered transcript is both confusing and wastes provider tokens on redundant displays.
  • What changed: state.pendingOptimisticUserMessage is now a non-negative counter of in-flight optimistic sends instead of a boolean. Each chat send increments; each newly observed gateway run id (regardless of whether another run is already active) decrements and tags that run as locally-initiated, so every burst-sent run gets correctly bound and no loadHistory() reload races against in-flight chat events.
  • What did NOT change (scope boundary): No protocol/gateway behavior, no abort/queueing semantics, no history-reload logic. The boolean→number is wire/storage internal to the TUI process; pendingChatRunId, noteLocalRunId, and the existing single-message bind path all keep working.

Change Type (select all)

  • Bug fix
  • Feature
  • Refactor required for the fix
  • Docs
  • Security hardening
  • Chore/infra

Scope (select all touched areas)

  • Gateway / orchestration
  • Skills / tool execution
  • Auth / tokens
  • Memory / storage
  • Integrations
  • API / contracts
  • UI / DX
  • CI/CD / infra

Linked Issue/PR

  • Closes — (filed downstream as NVIDIA/NemoClaw#3145, upstream symptom is in OpenClaw TUI; please link/close as the team prefers)
  • Related: prior fixes in this area — 61a0b02931 (preserve optimistic user messages during active runs), cc8ed8d25b (preserve user message during slow model responses).
  • This PR fixes a bug or regression

Real behavior proof (required for external PRs)

  • Behavior or issue addressed: TUI duplicate + reordered transcript when sending 5 messages back-to-back (NemoClaw#3145).

  • Real environment tested:

    • Live openclaw tui --local startup on macOS Sonnet 14.x (Apple Silicon), agent aits on bedrock-claude-opus-4-6, embedded gateway runtime.
    • Stand-alone integration repro driving the production src/tui/tui-event-handlers.ts against synthesized concurrent gateway events that match the burst pattern in the issue (5 user sends, overlapping delta/final from runs 1–5).
  • Exact steps or command run after this patch:

    1. Build: pnpm build (clean).
    2. Live banner sanity: pnpm openclaw tui --local --message "say hi briefly in 5 words" --timeout-ms 25000 — TUI starts, embedded runtime says local ready, banner reads openclaw tui - local embedded - agent aits - session main.
    3. Burst repro on real production code: node --import tsx /tmp/3145-proof/burst-repro.mjs against origin/main (baseline) and the patched branch. Same five-message burst, same overlapping event interleaving; the only thing that changes is src/tui/tui-event-handlers.ts + src/tui/tui-types.ts.
  • Evidence after fix:

    Live openclaw tui --local startup banner (copied console output):

     openclaw tui - local embedded - agent aits - session main
     local ready | idle
     agent aits | session main | unknown | tokens ?
     [sessions/store] pruned stale session entries
    

    Burst repro against the production src/tui/tui-event-handlers.ts — copied stdout from node --import tsx /tmp/3145-proof/burst-repro.mjs:

    Patched (this PR):

    --- chatLog operations recorded from real createEventHandlers ---
    ["asst-stream","run-1","4"]
    ["asst-stream","run-2","0900 UTC"]
    ["asst","run-1","4"]
    ["asst-stream","run-3","thanks"]
    ["asst","run-2","0900 UTC"]
    ["asst","run-3","you're welcome"]
    ["asst-stream","run-4","hi"]
    ["asst","run-4","hello!"]
    ["asst-stream","run-5","yes"]
    ["asst","run-5","absolutely"]
    --- audit ---
    loadHistory() invocations: 0
    local-tagged runs       : (none — already forgotten on final)
    counter remainder       : 0
    activeChatRunId         : null
    

    Baseline (origin/main, same script, same input, only TUI source swapped to git show origin/main:src/tui/tui-event-handlers.ts):

    --- chatLog operations recorded from real createEventHandlers ---
    ["asst-stream","run-1","4"]
    ["asst-stream","run-2","0900 UTC"]
    ["asst","run-1","4"]
    ["asst-stream","run-3","thanks"]
    ["asst","run-2","0900 UTC"]
    ["asst","run-3","you're welcome"]
    ["asst-stream","run-4","hi"]
    ["asst","run-4","hello!"]
    ["asst-stream","run-5","yes"]
    ["asst","run-5","absolutely"]
    --- audit ---
    loadHistory() invocations: 3
    local-tagged runs       : (none)
    pendingOptimisticUserMessage flag: false
    activeChatRunId         : null
    

    The chatLog ops list matches because the recorder is synchronous; the runtime difference is the audit block — baseline triggers 3 stray loadHistory() reloads on runs 2/3/5. In a live openclaw tui, each of those reloads calls chatLog.clearAll() and rebuilds from server history, racing with live delta/final events that are still arriving for the next overlapping run — that race is what produces the duplicated and reordered transcript shown in the issue.

  • Observed result after fix: Patched build records exactly one assistant entry per run, no orphan stream reloads. The local-tagged runs set is empty post-final because forgetLocalRunId runs on each non-error final, which is the same lifecycle as the existing single-message path. activeChatRunId returns to null and the counter drains to 0.

  • What was not tested: The exact Ubuntu 24.04 sandbox path described in the issue (no live nemoclaw connect access). The fix is platform-agnostic — burst handling lives entirely in the TUI client, which runs the same code on every host.

  • Before evidence: The terminal capture in NemoClaw#3145 (each prompt + reply rendered twice, send order scrambled).

Root Cause

  • Root cause: pendingOptimisticUserMessage was a boolean. Every sendMessage set it to true; the first incoming gateway chat event consumed it (binding that runId to the optimistic send and noting it as local). When the user fired N messages before any gateway events arrived, the flag was set N times but consumed exactly once — runs 2..N were treated as external. Their final events therefore reached maybeRefreshHistoryForRunloadHistory(), which calls chatLog.clearAll() and rebuilds from server history. Multiple concurrent reloads racing with still-streaming delta/final events produced the duplicated and reordered transcript shown in the issue. Codex review on the v1 of this patch caught a second strand: the local-binding decision was still gated on !state.activeChatRunId, so an overlapping run id (gateway runs concurrent sends per session — the burst case) skipped the bind branch entirely. The current revision moves the binding to fire on the first sighting of each non-/btw run id, independent of activeChatRunId.
  • Missing detection / guardrail: Existing tests only covered the single-pending-send scenario; no coverage for "two or more optimistic sends in flight before any event arrives", and no coverage for "second run delta arrives while the first run is still active".
  • Contributing context: Earlier commits (61a0b02931, cc8ed8d25b) intentionally moved away from "tag the local runId at send time" because the gateway response runId can drift from the idempotency key under dedupe. The boolean was the pragmatic single-message bridge; this PR generalises that bridge to bursts and to the overlapping-runs case the gateway can emit.

Regression Test Plan

  • Coverage level that should have caught this:
    • Unit test
    • Seam / integration test (the production-source burst-repro.mjs above qualifies as a thin integration harness)
    • End-to-end test
    • Existing coverage already sufficient
  • Target test or file: src/tui/tui-event-handlers.test.tsbinds every burst-sent optimistic message to its gateway run id (#3145) and tags overlapping burst runs as local even while another run is active (#3145). Plus src/tui/tui-command-handlers.test.tscounts every burst-sent message in pendingOptimisticUserMessage (#3145).
  • Scenario the test should lock in: Multiple optimistic sends queued before any gateway events; every subsequent run id observed must be noteLocalRunId-tagged and no loadHistory() should fire on their finals — including the case where the second run id arrives mid-stream while the first run is still the active run.
  • Why this is the smallest reliable guardrail: It exercises the exact handleChatEvent branch that previously dropped the binding, deterministic, and matches the live burst-repro.mjs numbers above.
  • Existing test that already covers this: None — binds optimistic pending messages to the first gateway run id only locked in the single-send path.

User-visible / Behavior Changes

  • TUI users sending multiple messages quickly no longer see duplicate prompts/replies or out-of-order transcripts. No config or default changes.

Diagram

Before (boolean flag, 5 burst sends):
  send msg1..5      → flag=true (no accumulation across sends)
  evt run1 delta    → consume flag, noteLocal(run1), flag=false
  evt run2 delta    → flag is false, run2 NOT tagged local
  evt run1 final    → isLocal, no reload  ✓
  evt run2 final    → !isLocal → loadHistory()    ← race begins
  evt run3 final    → !isLocal → loadHistory()    ← race continues
  evt run5 final    → !isLocal → loadHistory()    ← race continues
  → chatLog rebuild interleaves with live finalize → duplicates + reorder

After (counter, 5 burst sends):
  send msg1..5      → counter=5
  first sighting of run1..5 → each decrements and noteLocal()s
  all finals        → isLocal, no loadHistory() reload
  → single-copy transcript in send order

Security Impact

  • New permissions/capabilities? No
  • Secrets/tokens handling changed? No
  • New/changed network calls? No
  • Command/tool execution surface changed? No
  • Data access scope changed? No

Repro + Verification

Environment

  • OS: macOS 14 (Apple Silicon, M-series); originally reported on Ubuntu 24.04 sandbox.
  • Runtime/container: Node 22+; OpenClaw embedded TUI (--local) and gateway-mode TUI both share this code path.
  • Model/provider: Provider-agnostic — issue is in TUI client, not inference. Live banner check used aws/anthropic/bedrock-claude-opus-4-6 via local agent aits.
  • Integration/channel (if any): None — TUI direct.
  • Relevant config (redacted): default.

Steps

  1. openclaw tui (or openclaw tui --local).
  2. Send 5 distinct messages, each within ~1s of the previous (e.g. hi, what is 2+2, list 3 colors, what time is it, thanks).
  3. Wait up to 60s and inspect rendered history.

Expected

  • Each prompt and its reply appear exactly once, in send order.

Actual (before fix)

  • Each prompt + reply rendered twice; turn ordering scrambled (NemoClaw#3145 captures the exact transcript).

Evidence

  • Failing test/log before + passing after — see the side-by-side burst-repro.mjs output in the Real behavior proof section.
  • Trace/log snippets — live openclaw tui --local banner above.
  • Screenshot/recording
  • Perf numbers (if relevant)

Human Verification

  • Verified scenarios: Patched and baseline openclaw runs of burst-repro.mjs produce the contrast above; live openclaw tui --local --message ... banner reaches local ready | idle with the patched build.
  • Edge cases checked: Counter clamp on send failure (decrement only, never below zero); session switch resets counter; /auth block path still treats counter > 0 as "pending"; overlapping run id arriving while another is activeChatRunId is still bound on first sighting.
  • What I did NOT verify: Live remote-gateway round-trip on the original Ubuntu sandbox; provider dedupe path where the gateway returns a different runId for an idempotent retry (this PR does not change that path).

Review Conversations

  • I replied to or resolved every bot review conversation I addressed in this PR.
  • I left unresolved only the conversations that still need reviewer or maintainer judgment.

Compatibility / Migration

  • Backward compatible? Yes. pendingOptimisticUserMessage is internal TUI runtime state; the type widens from boolean to number, all truthiness checks (if (state.pendingOptimisticUserMessage)) keep working.
  • Config/env changes? No
  • Migration needed? No

Risks and Mitigations

  • Risk: Counter could drift if a sendChat succeeds but the gateway never emits any chat event for that runId (e.g. gateway-side abort lost in transit).
    • Mitigation: Session switch (setSession / syncSessionKey) resets the counter to 0; send failure decrements; the existing streaming watchdog still resets activeChatRunId and activity status, so the next user input clears any stale "sending" UI.

AI-assisted

This PR was developed with assistance from Claude Code. Root-cause analysis, repro test, and patch were proposed by the agent and human-reviewed before commit. codex review --base origin/main was run locally; the v1 P1 finding (overlapping-run binding gated on activeChatRunId) was addressed in fix-up commit e3d6a4c5 and verified by a second clean codex review pass. Real behavior proof above is from human-run openclaw invocations, not AI-generated.

@openclaw-barnacle openclaw-barnacle Bot added size: S triage: mock-only-proof Candidate: PR proof only shows tests, mocks, snapshots, lint, typecheck, or CI. labels May 7, 2026
@clawsweeper

clawsweeper Bot commented May 7, 2026

Copy link
Copy Markdown
Contributor

Thanks for the context here. I did a careful shell check against current main, and this is already implemented.

Current main and the latest release already solve the central TUI duplicate/reordered optimistic-transcript problem with the merged keyed pending-row implementation from #86205, so this older counter-based branch is no longer necessary.

So I’m closing this as already implemented rather than keeping a duplicate issue open.

Review details

Best possible solution:

Keep the shipped keyed pending-row implementation from #86205; open a new narrow current-main report only if the burst duplicate transcript still reproduces on v2026.6.6 or later.

Do we have a high-confidence way to reproduce the issue?

No current-main failing reproduction was established. The PR body describes an older v2026.4.24/NemoClaw path, while current main and v2026.6.6 contain source and tests for the merged pending-row fix.

Is this the best way to solve the issue?

Yes for closing, not for merging this branch. The current keyed pending-row implementation is a broader and more maintainable fix than adding a counter to the old boolean path.

Security review:

Security review cleared: The branch only touches TUI state handling, tests, and a changelog entry; I found no dependency, workflow, secret, install, package, or code-execution surface change.

AGENTS.md: found and applied where relevant.

What I checked:

  • Current source: keyed pending user row before gateway events: Current main renders a locally submitted TUI prompt as a keyed pending user row, records the generated run id as local before gateway events can arrive, and rekeys the row to a gateway-accepted run id when needed. (src/tui/tui-command-handlers.ts:762, 04875efd28ee)
  • Current source: history reload no longer blindly clears pending rows: Current main preserves pending user rows across history rebuilds and reconciles them after persisted history catches up, addressing the duplicate/reorder race this PR tried to avoid through a counter. (src/tui/tui-session-actions.ts:460, 04875efd28ee)
  • Current source: optimistic run binding while another run is active: Current event-handler tests cover binding a pending run final while another run remains active and assert that no history reload fires for that local optimistic path. (src/tui/tui-event-handlers.test.ts:1017, 04875efd28ee)
  • Git history: merged canonical fix: The merged canonical PR is fix(tui): stabilize optimistic user messages across history reloads, runId reassignment, and abort #86205, which landed as ac9a219 and added keyed pending rows, run-id rekeying, registration guards, and history reconciliation on the same TUI optimistic transcript path. (ac9a219692aa)
  • Fix timestamp provenance: The canonical merge commit timestamp is 2026-06-05T19:09:24Z, which provides the current-main implementation provenance for the shipped fix. (ac9a219692aa)
  • Release provenance: The v2026.6.6 release source contains the same pendingSubmitDraft, rekeyPendingUser, markSubmittedRunRegistered, and pending-history-refresh implementation, so the central fix has shipped. (src/tui/tui-command-handlers.ts:756, 8c802aa68351)

Likely related people:

  • RomneyDa: Authored and merged fix(tui): stabilize optimistic user messages across history reloads, runId reassignment, and abort #86205, which added the shipped keyed pending-row, rekey, abort, and history-reconciliation implementation that supersedes this counter-based branch. (role: implemented current canonical fix; confidence: high; commits: ac9a219692aa, 75cbcc2d031a, 5f373ae4d3db; files: src/tui/tui-command-handlers.ts, src/tui/tui-event-handlers.ts, src/tui/tui-session-actions.ts)
  • vincentkoc: Local blame and GitHub path history show recent work and coauthorship around pending sends, history reconciliation, and optimistic TUI state in the same files. (role: recent TUI pending/history contributor; confidence: high; commits: 51d6d7013f07, f0a44232716d, 5a0e67791fdc; files: src/tui/tui-command-handlers.ts, src/tui/tui-event-handlers.ts, src/tui/components/chat-log.ts)
  • steipete: Recent commits touched TUI session refresh, history, and surrounding session behavior, making this a likely routing candidate for current-main interactions. (role: recent adjacent TUI session/history contributor; confidence: medium; commits: 45ab8229180b, 18dc6e5cd4f8, 2682c027746b; files: src/tui/tui-session-actions.ts, src/tui/tui-command-handlers.ts)
  • neeravmakwana: Authored the recent busy prompt queueing work that touches the same TUI send gating and pending-run surface this stale branch now conflicts with. (role: recent adjacent busy/queue owner; confidence: medium; commits: 6842d72a9c7b; files: src/tui/tui-command-handlers.ts, src/tui/tui.ts, src/tui/tui-session-actions.ts)

Codex review notes: model internal, reasoning high; reviewed against 04875efd28ee; fix evidence: release v2026.6.6, commit ac9a219692aa.

@chengjiew

Copy link
Copy Markdown
Contributor Author

Ran codex review --base origin/main locally before requesting review. The first pass flagged a P1: my counter decrement and noteLocalRunId were still gated on !state.activeChatRunId, so an overlapping burst run (run 2's first delta arriving while run 1 is still streaming — the gateway runs chat sends concurrently per session) would skip the local-binding branch and leave the counter stale + the run untagged.

Pushed b045e04b as a fix-up:

  • Move the local-binding decision to fire on the first sighting of a non-/btw run id (using sessionRuns + finalizedRuns as the seen set), independent of activeChatRunId.
  • Keep the activeChatRunId takeover guarded by !state.activeChatRunId as before.
  • Added regression test tags overlapping burst runs as local even while another run is active (#3145) in tui-event-handlers.test.ts that locks the overlap path.

Re-ran codex review --base origin/main after the fix-up — clean. Full TUI lane now green at 338/338 (was 337 before adding the overlap test). pnpm check:changed exits 0.

@chengjiew
chengjiew force-pushed the fix/nemoclaw-3145-tui-burst-dedup branch from b045e04 to e3d6a4c Compare May 7, 2026 14:55
@openclaw-barnacle openclaw-barnacle Bot added proof: supplied External PR includes structured after-fix real behavior proof. and removed triage: mock-only-proof Candidate: PR proof only shows tests, mocks, snapshots, lint, typecheck, or CI. labels May 7, 2026
@clawsweeper clawsweeper Bot added the proof: sufficient ClawSweeper judged the real behavior proof convincing. label May 7, 2026
@chengjiew
chengjiew force-pushed the fix/nemoclaw-3145-tui-burst-dedup branch from e3d6a4c to 373bbcf Compare May 8, 2026 04:32
@chengjiew

Copy link
Copy Markdown
Contributor Author

Rebased onto latest origin/main (36f847a60e).

The previous CI run (yesterday's queue) failed on check-test-types and check. The errors were in src/agents/openai-transport-stream.test.ts and src/agents/pi-embedded-runner/openai-stream-wrappers.test.ts — both are unrelated to this PR (TUI-only changes). Those errors looked like a type-schema migration that landed on main after my original rebase base 8e88c7b297; today's main contains 548b55676f (fix: strip unsupported Fireworks tool schema keywords) and 772034d741 (fix: strip tools for no-tool completions models) which align the test fixtures with the new OpenAIModeModel shape.

Verified locally on the new base:

Branch head: 373bbcf700.

@openclaw-barnacle openclaw-barnacle Bot removed the proof: sufficient ClawSweeper judged the real behavior proof convincing. label May 8, 2026
@clawsweeper clawsweeper Bot added the proof: sufficient ClawSweeper judged the real behavior proof convincing. label May 8, 2026
@chengjiew
chengjiew force-pushed the fix/nemoclaw-3145-tui-burst-dedup branch from 373bbcf to 96c1615 Compare May 19, 2026 16:51
@openclaw-barnacle openclaw-barnacle Bot removed the proof: sufficient ClawSweeper judged the real behavior proof convincing. label May 19, 2026
@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. P2 Normal backlog priority with limited blast radius. merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. labels May 19, 2026
@openclaw-barnacle

Copy link
Copy Markdown

This pull request has been automatically marked as stale due to inactivity.
Please add updates or it will be closed.

@openclaw-barnacle openclaw-barnacle Bot added the stale Marked as stale due to inactivity label Jun 3, 2026
@clawsweeper clawsweeper Bot added rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. and removed rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. labels Jun 14, 2026
@openclaw-barnacle openclaw-barnacle Bot removed the stale Marked as stale due to inactivity label Jun 15, 2026
@clawsweeper

clawsweeper Bot commented Jun 15, 2026

Copy link
Copy Markdown
Contributor

ClawSweeper applied the proposed close for this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. P2 Normal backlog priority with limited blast radius. proof: supplied External PR includes structured after-fix real behavior proof. rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. size: S status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant