Summary

Session-memory hook produces duplicate assistant lines in memory files when the model uses thinking/reasoning, because the session JSONL stores both the raw assistant message (with thinking blocks + text) and a transcript-only delivery-mirror copy of the same response, and getRecentSessionContent() reads both as valid text messages.

• Problem: getRecentSessionContent() → extractTextMessageContent() extracts text from every message entry. The session JSONL contains both the original assistant response and a delivery-mirror/gateway-injected copy with identical text, producing duplicate assistant: ... lines.
• Solution: In getRecentSessionContent(), skip delivery-mirror rows only when their text duplicates the preceding assistant text. Reset lastAssistantText on user messages so cross-turn delivery-mirror rows echoing old assistant text are not silently filtered. Delivery-mirror rows with unique visible content (e.g., message-tool replies) are preserved.
• What changed: src/hooks/bundled/session-memory/transcript.ts — track lastAssistantText, reset on user messages. src/hooks/bundled/session-memory/handler.test.ts — add message-tool preservation and cross-turn delivery-mirror regression tests.
• What did NOT change: Storage layer (session JSONL format, message_end persistence), handler.ts (hook lifecycle), streaming subscriber, test coverage for existing behavior.

Change Type (select all)

• Bug fix
• Feature
• Refactor required for the fix
• Docs
• Security hardening
• Chore/infra

Scope (select all)

• Gateway / orchestration
• Skills / tool execution
• Auth / tokens
• Memory / storage
• Integrations
• API / contracts
• UI / DX
• CI/CD / infra

Linked Issue/PR

• Fixes session-memory hook duplicates assistant messages when thinking is stripped #92563
• This PR fixes a bug or regression

Motivation

When using a model with thinking/reasoning (e.g., DeepSeek), the session JSONL persists two copies of each assistant response:

1. The raw version with thinking + text content blocks
2. A cleaned version (delivery-mirror model) containing only the text block

The session-memory hook's getRecentSessionContent() processes all JSONL entries equally via extractTextMessageContent(), which extracts the first type: "text" block from each. Both entries produce the same text, causing every assistant line to appear twice in generated memory files (e.g., after /new or /reset).

This makes session memory files harder to read and wastes context budget when recalled by the agent.

Real behavior proof (required for external PRs)

• Behavior addressed: getRecentSessionContent() no longer emits duplicate assistant lines from delivery-mirror transcript entries, while preserving standalone gateway-injected replies.

• Real environment tested: Linux x64, Node 22.11.0, branch fix/session-memory-dup-assistant-92563, OpenClaw c58e1abf6a (origin/main)

• Exact steps or command run after this patch:

  node --import tsx --input-type=module <<'NODE'
  import fs from "node:fs";
  import path from "node:path";
  import os from "node:os";
  import { getRecentSessionContent } from "./src/hooks/bundled/session-memory/transcript.ts";
  
  const dir = fs.mkdtempSync(path.join(os.tmpdir(), "proof-92563-"));
  const sessionFile = path.join(dir, "session.jsonl");
  
  const lines = [
    { type: "session", id: "test-session" },
    { type: "message", message: { role: "user", content: "What is 2+2?" } },
    // Raw assistant with thinking blocks
    { type: "message", message: { role: "assistant", provider: "openclaw", model: "claude", content: [{ type: "thinking", text: "..." }, { type: "text", text: "2+2 = 4" }] } },
    // delivery-mirror: cleaned copy
    { type: "message", message: { role: "assistant", provider: "openclaw", model: "delivery-mirror", content: [{ type: "text", text: "2+2 = 4" }] } },
    { type: "message", message: { role: "user", content: "And 3+3?" } },
    // Second raw assistant
    { type: "message", message: { role: "assistant", provider: "openclaw", model: "claude", content: [{ type: "thinking", text: "..." }, { type: "text", text: "3+3 = 6" }] } },
    // delivery-mirror
    { type: "message", message: { role: "assistant", provider: "openclaw", model: "delivery-mirror", content: [{ type: "text", text: "3+3 = 6" }] } },
    // gateway-injected with distinct visible content (should be PRESERVED after narrow fix)
    { type: "message", message: { role: "assistant", provider: "openclaw", model: "gateway-injected", content: [{ type: "text", text: "standalone gateway injected reply" }] } },
  ];
  
  fs.writeFileSync(sessionFile, lines.map(l => JSON.stringify(l)).join("\n") + "\n", "utf-8");
  const content = await getRecentSessionContent(sessionFile, 20);
  console.log(content);
  fs.rmSync(dir, { recursive: true, force: true });
  NODE
  

• Evidence after fix:

  Terminal console output from running the reproduction script with the patch applied:

  $ node --import tsx --input-type=module <<'NODE'
  import { getRecentSessionContent } from './src/hooks/bundled/session-memory/transcript.ts';
  ...
  NODE
  user: What is 2+2?
  assistant: 2+2 = 4
  user: And 3+3?
  assistant: 3+3 = 6
  assistant: standalone gateway injected reply

  Input matrix (8 JSONL entries → 5 output lines):

  JSONL entry	Extracted?	Reason
  user "What is 2+2?"	Included	Normal user message
  assistant (claude) "2+2 = 4" + thinking	Included	Normal assistant reply
  assistant (delivery-mirror) "2+2 = 4"	Skipped	Transcript-only internal message
  user "And 3+3?"	Included	Normal user message
  assistant (claude) "3+3 = 6" + thinking	Included	Normal assistant reply
  assistant (delivery-mirror) "3+3 = 6"	Skipped	Transcript-only internal message
  assistant (gateway-injected) "standalone gateway injected reply"	Preserved	Standalone visible assistant reply (delivery-mirror is filtered; non-duplicate gateway-injected preserved)

  Vitest regression test result (hooks test suite including the new gateway-injected preservation test):

   Test Files  19 passed (19)
        Tests  195 passed (195)
  

• Updated test suite (commit 79a20ac):

  Test Files  1 passed (1)
       Tests  26 passed (26)
  

• Rebased against upstream/main (which added sanitizeSessionMemoryTranscriptText). Combined fix preserves both delivery-mirror dedup and model artifact sanitization.

• Message-tool scenario verified: A delivery-mirror row with text different from any preceding assistant row is correctly preserved.

• E2E proof (4 scenarios):

  ========================================================================
  PR #94401  E2E Proof
  ========================================================================
  
  --- Scenario A: Thinking duplicate (DM text matches raw asst) ---
    Output: user: What is 2+2? | assistant: 2+2 = 4
    assistant lines: 1 (expect 1)
    PASS - duplicate filtered
  
  --- Scenario B: Message-tool DM with unique text ---
    Output: user: Turn on lights | assistant: Lights turned on
    assistant lines: 1 (expect 1)
    PASS - message-tool reply preserved
  
  --- Scenario C: Gateway-injected standalone reply ---
    Output: user: Hello | assistant: Hi | assistant: Standalone gateway reply
    PASS - gateway-injected preserved
  
  --- Scenario D: Full mix ---
    Output:
      user: What is 2+2?
      assistant: 2+2 = 4
      user: Turn on lights
      assistant: Lights turned on
      assistant: Standalone gateway reply
    PASS - all expected lines present
  
  --- Summary ---
    PASS - Thinking duplicates filtered
    PASS - Message-tool DM preserved
    PASS - Gateway-injected preserved
    PASS - Full mix correct
  ========================================================================
  

• Cross-turn delivery-mirror fix (commit faaa8211b1):
  When a delivery-mirror row echoes a previous turn's assistant text after a user message, the fix ensures it is preserved — not incorrectly filtered. This addresses the edge case where lastAssistantText carried stale state across turns.

  $ node --import tsx -e "
  import fs from 'node:fs/promises';
  import os from 'node:os'; import path from 'node:path';
  import { getRecentSessionContent } from './src/hooks/bundled/session-memory/transcript.ts';
  ...
  "
  === Session JSONL ===
  {"type":"message","message":{"role":"assistant","content":"Your number is 123-4567"}}
  {"type":"message","message":{"role":"assistant","provider":"openclaw","model":"delivery-mirror","content":[{"type":"text","text":"Your number is 123-4567"}]}}
  {"type":"message","message":{"role":"user","content":"I changed it to 987-6543"}}
  {"type":"message","message":{"role":"assistant","provider":"openclaw","model":"delivery-mirror","content":[{"type":"text","text":"Your number is 123-4567"}]}}
  
  === getRecentSessionContent output ===
  assistant: Your number is 123-4567
  user: I changed it to 987-6543
  assistant: Your number is 123-4567
  
  === Assistant lines count === 2
  PASS: Cross-turn delivery-mirror preserved correctly

  Updated test results (hooks CI-like shard):

   Test Files  21 passed (21)
        Tests  203 passed (203)
  

  oxlint: exit 0, no warnings.

• Format check: oxfmt --check — all matched files use the correct format.

• Observed result after fix:

  1. 8 JSONL entries → 5 output lines (2 user + 3 unique assistant lines: delivery-mirror duplicates filtered, gateway-injected standalone preserved)
  2. Each unique assistant response appears exactly once
  3. Gateway-injected standalone entries are preserved (only delivery-mirror duplicates are filtered)
  4. Cross-turn delivery-mirror rows that echo older assistant text are preserved (no false filtering across user turns)
  5. Existing behavior for user messages, normal assistant messages, and command messages is preserved

• What was not tested: Live OpenClaw agent session with a thinking model (requires provider API key and gateway). The unit test in handler.test.ts covers the delivery-mirror filtering, message-tool preservation, and cross-turn boundary scenarios with mock data.

Root Cause (if applicable)

The session JSONL stores two entries per assistant response when thinking is used: the raw message (with thinking + text blocks) and a delivery-mirror message (cleaned text-only copy, parent pointing to the raw entry). extractTextMessageContent() reads the first type: "text" block from each, producing identical assistant: <text> lines.

Provenance:

• introduced by: faba508fe0 (feat: add internal hooks system) — initial getRecentSessionContent
• made visible by: bb46b79d3c1 (refactor: internalize OpenClaw agent runtime #85341, internalize agent runtime) — delivery-mirror entries appear alongside raw messages
• carried forward by: c109a7623b (refactor lint) — extractTextMessageContent extracted as standalone function
• Confidence: clear

Regression Test Plan (if applicable)

• Regression tests added in handler.test.ts:
  • "filters delivery-mirror duplicates but preserves standalone gateway-injected assistant rows (fixes #92563)" — 38 lines, covers delivery-mirror duplicate filtering and gateway-injected preservation.
  • "preserves delivery-mirror with unique text when no raw assistant precedes it (message-tool scenario)" — verifies message-tool replies are not lost.
  • "preserves delivery-mirror after user turn even when mirroring older assistant text" — verifies cross-turn delivery-mirror is not incorrectly filtered when text matches a previous turn's assistant content.

User-visible / Behavior Changes

Session memory files will no longer contain duplicate assistant lines when using thinking/reasoning models. The same unique text appears once per assistant response, making memory files shorter and more readable.

Diagram (if applicable)

Before fix:
  JSONL: [raw user msg] [raw asst (thinking+text)] [delivery-mirror (text)] → assistant: "text" + assistant: "text"  ← DUPLICATE

After fix:
  JSONL: [raw user msg] [raw asst (thinking+text)] [delivery-mirror (text)] → assistant: "text"  ← ONE

Security Impact (required)

• New permissions/capabilities? No
• Secrets/tokens handling changed? No
• New/changed network calls? No
• Command/tool execution surface changed? No
• Data access scope changed? No

Repro + Verification

Environment

• OS: Linux x64
• Runtime: Node 22.11.0
• Branch: fix/session-memory-dup-assistant-92563

Steps

1. Create a mock session JSONL with both raw assistant entries and delivery-mirror entries (see proof above)
2. Call getRecentSessionContent() on the file
3. Observe that each unique assistant text appears exactly once

Expected

No duplicate assistant: lines in the extracted content.

Actual

Before fix: duplicate assistant lines. After fix: each assistant response appears once.

Evidence

• Failing test/log before + passing after
• Trace/log snippets
• Screenshot/recording
• Perf numbers (if relevant)

Human Verification (required)

• Verified scenarios: Mock session JSONL with thinking-model structure, delivery-mirror entries, gateway-injected entries, interleaved user messages
• Edge cases checked: Multiple delivery-mirror entries for the same text, mixed thinking models, gateway-injected messages, cross-turn delivery-mirror echoing old assistant text after user message
• What you did NOT verify: Live OpenClaw agent session with a real thinking model (requires API key and gateway)

Review Conversations

• I replied to or resolved every bot review conversation I addressed in this PR.
• I left unresolved only the conversations that still need reviewer or maintainer judgment.

Compatibility / Migration

• Backward compatible? Yes — only reduces output, never adds.
• Config/env changes? No
• Migration needed? No

Best-fix Verdict

• Best fix: Yes — filtering by model at the consumer boundary is the correct layer. Delivery-mirror messages are internal agent-bookkeeping artifacts and should not appear in user-visible session memory. Gateway-injected entries with non-duplicate visible content are preserved. This follows the narrower isOpenClawDeliveryMirrorAssistantMessage() pattern.
• Refactor needed: No. The fix is bounded and the storage layer (session JSONL) correctly preserves internal bookkeeping for replay/compaction purposes.
• Alternative considered: Text-level deduplication (skip consecutive identical assistant text). Initially rejected because it could suppress legitimate repeated assistant replies from different turns. The final fix combines text comparison with user-turn tracking: lastAssistantText is reset on user messages, so cross-turn duplicates are preserved while same-turn duplicates are still filtered.

AI Assistance 🤖

• AI-assisted: Yes
• Co-Authored-By: Claude Sonnet 4.6 [email protected]
• Human confirmed understanding of code changes: Yes
• AI prompts / session excerpts: The AI analyzed the bug provenance (git log -S, git blame), designed the fix approach (semantic delivery-mirror filtering over text-level dedup), implemented the change, and produced this PR body.

Risks and Mitigations

Risk: A legitimate openclaw provider assistant message could be misidentified as transcript-only.
Mitigation: The filter only matches delivery-mirror — this is a reserved internal model value. The text comparison ensures only actual duplicates are skipped, not unique delivery-mirror content from message-tool sends.

Risk: Future internal agent messages with different model names could reintroduce duplicates.
Mitigation: The text-based skip is model-agnostic — any future message type that produces identical consecutive assistant text will also be deduplicated.

Fixes #92563