Skip to content

Embedded runner: freshly streamed thinking signatures intermittently invalid on replay (Anthropic); recovery wrapper never fires because error text is genericized #92201

Description

@CarlCapital

Title: Embedded runner: freshly streamed thinking signatures intermittently invalid on replay (Anthropic); recovery wrapper never fires because error text is genericized

Summary

On 2026.6.5, the embedded agent runner (Slack plugin) intermittently persists Anthropic thinking blocks whose signatures fail validation when history is replayed — permanently poisoning the session: every subsequent message fails with invalid_request_error: messages.N.content.0: Invalid signatureinthinking block, surfaced to the user as LLM request failed: provider rejected the request schema or tool payload.

Additionally, wrapAnthropicStreamWithRecovery — which exists precisely for this and would auto-heal by stripping thinking blocks and retrying — never fires on this path: zero [session-recovery] log lines across all incidents. The error reaching the recovery predicate appears to be the genericized "LLM request failed: provider rejected the request schema or tool payload." text (raw provider error attached separately as rawError= in logs), which THINKING_BLOCK_ERROR_PATTERN cannot match.

Environment

  • OpenClaw 2026.6.5 (ghcr.io/openclaw/openclaw:latest pulled 2026-06-10), embedded runner via Slack plugin
  • Provider: anthropic, model claude-sonnet-4-6, thinkingLevel high
  • First occurrence ever ~17h after upgrading the image; 3 separate poisonings of the same session within ~2h on 2026-06-11; at peak, every fresh thinking block in the affected thread was bad (one reply works → next message bricks)

Evidence isolating the layer

  1. Replay/persistence exonerated: extracted each stored thinkingSignature from the session jsonl and replayed each block individually against api.anthropic.com (same key/model): blocks generated earlier that day validate; blocks generated from 13:36 UTC onward fail. Stored signature and thinking text are byte-identical to the post-assembly trajectory log → nothing mutated after assembly.
  2. Anthropic exonerated: direct curl→API generate+replay with the same key/model, both non-streaming and streaming (manual thinking_delta/signature_delta concatenation), validated 4/4 while OpenClaw was concurrently producing invalid pairs.
  3. → corruption happens between the wire and the assembled message inside the runner's stream assembly. Note the build already contains the signature_delta append fix from fix(agents): concatenate signature_delta chunks in transport stream #87582 (block.thinkingSignature = (block.thinkingSignature || "") + delta.signature), so this is something else — possibly related to 2026.6.5's deferred stream-start / message_start changes.

Two requested fixes

  1. Find/fix the stream-assembly corruption (related: Bug: anthropic-transport-stream overwrites signature_delta instead of appending, causing permanent session corruption #87574, Compaction/pruning corrupts thinking blocks → Anthropic API rejection #30157, Anthropic thinking block 'signature' field lost during session persistence — causes API rejection #43691).
  2. Make shouldRecoverAnthropicThinkingError see the raw provider error (e.g. include rawError in the matched text, or match before genericizing) so sessions self-heal as designed even when assembly produces a bad block.

Workaround we applied

Force dropReasoningFromHistory for anthropic in the embedded runner (build-time patch) — past thinking is never replayed, so a bad stored signature is harmless; thinking generation unaffected. Happy to provide session jsonl excerpts (sans content) / probe scripts.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1High-priority user-facing bug, regression, or broken workflow.clawsweeper:fix-shape-clearClawSweeper found a clear likely implementation shape for this issue.clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.impact:auth-providerAuth, provider routing, model choice, or SecretRef resolution may break.impact:message-lossChannel message delivery can be lost, duplicated, or misrouted.impact:session-stateSession, memory, transcript, context, or agent state can drift or corrupt.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions