Skip to content

fix(llm): preserve structured tool result text across providers#97450

Closed
snowzlmbot wants to merge 0 commit into
openclaw:mainfrom
snowzlmbot:fix/96857-provider-context-placeholders
Closed

fix(llm): preserve structured tool result text across providers#97450
snowzlmbot wants to merge 0 commit into
openclaw:mainfrom
snowzlmbot:fix/96857-provider-context-placeholders

Conversation

@snowzlmbot

@snowzlmbot snowzlmbot commented Jun 28, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

Preserves structured non-image tool-result content in provider replay/model-visible payloads instead of dropping it into empty or image-placeholder-style output. This addresses the provider-serializer slice of #96857 across OpenAI Chat Completions, OpenAI Responses / transport replay, Anthropic Messages / transport replay, Google / Vertex, and Mistral conversion paths.

This PR intentionally does not claim to resolve the broader provider-failover, truncation, persistent session-degradation, or empty non-image result placeholder paths tracked separately in #96857 and related PRs.

What Problem This Solves

Fixes an issue where provider serializers only preserved explicit type: "text" tool-result blocks when building model-visible replay context. Structured non-image results such as MCP resource blocks, JSON status payloads, and runtime adapter output could be silently dropped before reaching the model, so the agent could receive an empty result or a placeholder instead of the actual structured output.

The affected paths include OpenAI Chat Completions, OpenAI Responses / runtime transport replay, Anthropic Messages / runtime transport replay, Google / Vertex, and Mistral provider conversion.

Why This Change Was Made

The fix extracts a shared bounded structured tool-result serializer and uses it in provider replay plus the existing runtime visible-output structured fallback. That keeps the provider and runtime behavior aligned while ensuring structured payloads are safe before replay: sensitive fields are redacted, opaque/binary values are omitted, circular references are handled, and very large payloads are capped.

The provider extraction helper also applies a final aggregate cap after merging all text/structured chunks, so many individually bounded blocks cannot create an oversized model-visible provider payload.

User Impact

Agents can now read ordinary structured tool output in affected provider contexts instead of losing it during replay. Existing text blocks remain preferred, and image/audio payloads continue using the existing multimodal handling rather than being serialized as text.

Implementation

  • Adds src/logging/tool-result-serialization.ts as the shared bounded structured tool-result serializer and exports the shared truncation helper used by provider aggregation.
  • Uses the serializer from src/llm/providers/tool-result-text.ts before provider replay, with a final cap on the merged provider text.
  • Reuses the same serializer from src/agents/embedded-agent-subscribe.tools.ts for runtime structured-result visible output.
  • Preserves text blocks and skips image/audio blocks for text extraction.
  • Keeps media-only tool-result blocks (image, image_url, and audio) out of provider text replay so multimodal/media payloads do not become model-visible JSON text.
  • Treats both media_type, mime_type, and mimeType metadata as media MIME aliases when deciding whether structured data should be omitted as binary.
  • Redacts sensitive keys such as authorization, cookies, API keys, access/refresh/id tokens, client secrets, private keys, passwords, and generic token/secret fields.
  • Omits opaque replay-only fields such as encrypted_content / encrypted_stdout and binary fields such as blobs or media data payloads.
  • Handles circular structured payloads and caps both per-block serialization and final merged provider text.
  • Keeps actual data URI tokens redacted without rewriting ordinary prose such as metadata:ready or data: is ordinary prose.
  • Restores CHANGELOG.md; release-owned changelog content is not changed by this PR.

Evidence

  • MIME alias binary guard: src/llm/providers/tool-result-text.test.ts covers snake-case mime_type on structured resources and proves binary data is omitted before provider replay.

  • Media-only replay guard: src/llm/providers/tool-result-text.test.ts covers image, image_url, and audio blocks together and proves only the structured non-media resource remains in provider text extraction.

  • Code path proof — OpenAI Chat Completions provider payload: src/llm/providers/openai-completions.test.ts captures the actual Chat Completions request payload via onPayload and verifies structured tool output is replayed as tool text after sanitization. The assertions prove:

    • JSON/resource tool-result blocks are serialized into the provider messages[].content field.
    • authorization, token, and password values are redacted to ***.
    • binary application/octet-stream data is replaced with [binary omitted: ... chars].
    • opaque encrypted_content is replaced with [opaque data omitted: ... chars].
    • circular references serialize as [Circular].
    • oversized single structured output is capped with …(truncated)….
    • multiple individually-under-cap structured blocks are final-capped after merge before entering the provider payload.

  • Code path proof — OpenAI Responses provider conversion: src/llm/providers/openai-responses-shared.test.ts exercises convertResponsesMessages() and verifies the actual function_call_output.output payload contains sanitized structured text instead of raw secrets/binary/opaque data.

  • Affected runtime path alignment: src/agents/embedded-agent-subscribe.tools.ts delegates structured fallback serialization to the same shared serializer used by provider replay, preserving runtime visible-output semantics while sharing redaction, omission, circular handling, and per-block caps.

  • Shared sanitizer and aggregate-cap proof: src/llm/providers/tool-result-text.test.ts covers structured-only blocks, mixed text/structured/image blocks, ordinary data: prose preservation, actual data URI token redaction, nested sensitive field redaction, opaque/binary omission, circular references, per-block length caps, shared-helper aggregate truncation, and many-block aggregate truncation after merge.

  • External replay artifact status: If maintainers still require a dependency-backed terminal/log artifact before merge, it should be attached outside the patch diff and linked here without expanding the code surface.

  • Local lightweight validation on current head 4a92599d3ff07724e955627e594b5310be12c738:

git diff --check
node --experimental-strip-types --check src/agents/anthropic-transport-stream.test.ts
node --experimental-strip-types --check src/agents/anthropic-transport-stream.ts
node --experimental-strip-types --check src/agents/embedded-agent-subscribe.tools.ts
node --experimental-strip-types --check src/agents/openai-transport-stream.test.ts
node --experimental-strip-types --check src/agents/openai-transport-stream.ts
node --experimental-strip-types --check src/llm/providers/anthropic.ts
node --experimental-strip-types --check src/llm/providers/google-shared.convert.test.ts
node --experimental-strip-types --check src/llm/providers/google-shared.ts
node --experimental-strip-types --check src/llm/providers/mistral.test.ts
node --experimental-strip-types --check src/llm/providers/mistral.ts
node --experimental-strip-types --check src/llm/providers/openai-completions.test.ts
node --experimental-strip-types --check src/llm/providers/openai-completions.ts
node --experimental-strip-types --check src/llm/providers/openai-responses-shared.test.ts
node --experimental-strip-types --check src/llm/providers/openai-responses-shared.ts
node --experimental-strip-types --check src/llm/providers/tool-result-text.test.ts
node --experimental-strip-types --check src/llm/providers/tool-result-text.ts
node --experimental-strip-types --check src/logging/tool-result-serialization.ts

All listed lightweight checks passed locally. This checkout has no node_modules, so focused Vitest/oxlint were not run locally; GitHub Actions remain the dependency-backed validation path for this PR.

Labels considered

  • agents — affected behavior is model-visible agent/provider replay.
  • P1 — fixes loss of ordinary structured tool output in affected provider paths.
  • merge-risk: 🚨 security-boundary — relevant because provider replay now serializes structured payloads, so redaction/omission/caps are part of the patch.
  • merge-risk: 🚨 message-delivery — relevant because agent-visible tool output can affect message/tool workflows.

Breaking changes

None.

Risk

Low/medium. This changes model-visible serialization for structured non-image tool-result blocks. Risk is bounded by preserving explicit text-first behavior, continuing to skip image/audio payloads for text extraction, and applying shared redaction/omission/circular/per-block/final-aggregate cap handling before provider replay.

Rollback

Revert this PR to restore provider serializers to the previous text-only extraction behavior. No migration, persistent state format change, dependency change, or configuration rollback is required.

Docs updated

No user-facing docs were changed. CHANGELOG.md is not part of this PR diff; release notes can be handled by the release process or squash/merge message.

Related issue / PR

Not tested / Limitations

@openclaw-barnacle openclaw-barnacle Bot added agents Agent runtime and tooling size: L triage: needs-pr-context Candidate: external PR body lacks required problem context or evidence. labels Jun 28, 2026
@snowzlmbot

snowzlmbot commented Jun 28, 2026

Copy link
Copy Markdown
Contributor Author

@clawsweeper review-pr

@clawsweeper

clawsweeper Bot commented Jun 28, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

@clawsweeper

clawsweeper Bot commented Jun 28, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Codex review: needs real behavior proof before merge. Reviewed June 28, 2026, 8:04 PM ET / 00:04 UTC.

Summary
The PR adds a shared structured tool-result serializer and wires it into provider and transport replay paths so structured non-image tool-result blocks become sanitized model-visible text.

PR surface: Source +195, Tests +760. Total +955 across 18 files.

Reproducibility: yes. Source inspection shows current provider converters still filter tool-result replay to explicit text blocks, and current main has focused tests for the empty non-image fallback that this PR head would regress if carried forward unchanged.

Review metrics: 1 noteworthy metric.

  • Provider replay fanout: 7 converters changed. A shared serializer or fallback mistake would affect several provider families and transport replay paths, not one isolated conversion.

Stored data model
Persistent data-model change detected: serialized state: src/agents/anthropic-transport-stream.test.ts, serialized state: src/llm/providers/tool-result-text.ts, serialized state: src/logging/tool-result-serialization.test.ts, serialized state: src/logging/tool-result-serialization.ts, vector/embedding metadata: src/llm/providers/tool-result-text.test.ts. Confirm migration or upgrade compatibility proof before merge.

Root-cause cluster
Relationship: fixed_by_candidate
Canonical: #96857
Summary: This PR is a candidate partial fix for the provider serializer/replay slice of the broader tool-result placeholder degradation issue.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🧂 unranked krab
Proof: 🧂 unranked krab
Patch quality: 🧂 unranked krab
Result: blocked until real behavior proof is added.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • Resolve conflicts while preserving (no output) for empty non-image OpenAI replay paths.
  • Attach redacted real provider replay proof showing structured text preservation plus secret/binary/media/opaque omission.
  • After proof is added, update the PR body so ClawSweeper re-reviews automatically; if it does not, ask a maintainer to comment @clawsweeper re-review.

Proof guidance:

  • [P1] Needs real behavior proof before merge: The PR body lists tests, static checks, and CI-style evidence, but it also says no live external provider request was made; the contributor should add redacted current-head terminal/log/live replay output and update the PR body to trigger a fresh review.

Risk before merge

  • [P2] The branch is currently conflicting, and carrying the PR-head fallback through conflict resolution would make empty non-image OpenAI tool results replay as (see attached image) again instead of (no output).
  • [P1] The diff serializes more structured tool payload data into model-visible provider replay, so redaction, binary/media omission, and aggregate caps need real current-head replay proof beyond unit/static checks.
  • [P1] This PR only covers the provider serializer slice; the broader failover, truncation, and session-degradation symptoms remain tracked at Normal tool text outputs can degrade to “(see attached image)” placeholders in agent context #96857.

Maintainer options:

  1. Resolve The Replay Fallback Conflict (recommended)
    Update the OpenAI completions, Responses shared, and managed transport paths so empty non-image results keep (no output) while image-bearing results keep image placeholders.
  2. Attach Current-Head Replay Proof
    Add redacted terminal, log, live output, or linked artifact proof showing structured replay text is preserved while secrets, binary/media data, opaque fields, and empty non-image results behave correctly.
  3. Treat This As A Partial Fix
    If maintainers land this serializer slice, keep the broader placeholder degradation issue open for failover, truncation, and session-state cases not proven here.

Next step before merge

  • [P1] The PR needs maintainer/contributor handling for a dirty branch plus author-supplied real behavior proof; repair automation cannot provide the contributor's real setup proof.

Security
Cleared: No supply-chain change or concrete sanitizer bypass was found; provider-boundary redaction proof remains a merge risk rather than a separate security finding.

Review findings

  • [P1] Preserve no-output fallback when replay text is empty — src/llm/providers/openai-completions.ts:1140
Review details

Best possible solution:

Rebase or resolve conflicts onto current main, preserve #97423 no-output fallback, keep the shared structured serializer, attach redacted current-head provider replay proof, and leave the broader report open for remaining symptoms.

Do we have a high-confidence way to reproduce the issue?

Yes. Source inspection shows current provider converters still filter tool-result replay to explicit text blocks, and current main has focused tests for the empty non-image fallback that this PR head would regress if carried forward unchanged.

Is this the best way to solve the issue?

No as-is. The shared serializer is the right layer for the provider-serializer slice, but this head must preserve current main's empty non-image fallback and add real replay proof before it is the best fix.

Full review comments:

  • [P1] Preserve no-output fallback when replay text is empty — src/llm/providers/openai-completions.ts:1140
    Current main reserves (see attached image) for image-bearing tool results and sends (no output) for empty non-image results. This PR head falls back to (see attached image) whenever extractToolResultText() returns an empty string, with the same pattern in Responses shared conversion and managed OpenAI transport replay, so resolving the conflict this way would reintroduce the placeholder bug fixed by fix: avoid image placeholder for empty text tool results #97423.
    Confidence: 0.9

Overall correctness: patch is incorrect
Overall confidence: 0.88

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 305fa9c4ddbe.

Label changes

Label justifications:

  • P1: The PR targets a broken model-visible agent/provider replay workflow and currently has a blocking fallback regression versus main.
  • merge-risk: 🚨 message-delivery: The diff changes how tool-result content is delivered into provider/model replay across OpenAI, Anthropic, Google/Vertex, and Mistral paths.
  • merge-risk: 🚨 security-boundary: Structured tool payloads now cross an external model-provider boundary, so redaction, media omission, and caps are merge-critical.
  • rating: 🧂 unranked krab: Overall readiness is 🧂 unranked krab; proof is 🧂 unranked krab and patch quality is 🧂 unranked krab.
  • status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs real behavior proof before merge: The PR body lists tests, static checks, and CI-style evidence, but it also says no live external provider request was made; the contributor should add redacted current-head terminal/log/live replay output and update the PR body to trigger a fresh review.
Evidence reviewed

PR surface:

Source +195, Tests +760. Total +955 across 18 files.

View PR surface stats
Area Files Added Removed Net
Source 10 358 163 +195
Tests 8 761 1 +760
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 18 1119 164 +955

What I checked:

  • Repository policy read: Root AGENTS.md and scoped src/agents/AGENTS.md were read fully; the review policy requires current-main comparison, sibling-surface review, and direct Codex source inspection for Codex-related provider/runtime behavior. (AGENTS.md:1, 305fa9c4ddbe)
  • Live PR state: GitHub reports the PR as CONFLICTING/DIRTY at head 887e20f, so conflict resolution must preserve newer main behavior before merge. (887e20f898b2)
  • Current main no-output behavior: Current main sends EMPTY_TOOL_RESULT_TEXT for empty non-image OpenAI completions tool results and reserves IMAGE_TOOL_RESULT_TEXT for image-bearing results. (src/llm/providers/openai-completions.ts:1146, 305fa9c4ddbe)
  • Current main regression coverage: Current main has focused tests asserting update_plan-style empty non-image tool results replay as (no output) in OpenAI completions, Responses shared conversion, and managed OpenAI transport replay. (src/llm/providers/openai-completions.test.ts:251, 305fa9c4ddbe)
  • PR head completions regression: At PR head, OpenAI completions uses (see attached image) whenever extracted text is empty, without checking whether the tool result actually contains an image. (src/llm/providers/openai-completions.ts:1140, 887e20f898b2)
  • PR head sibling fallback regression: The same empty-text image-placeholder fallback appears in PR-head OpenAI Responses shared conversion and managed OpenAI transport replay. (src/llm/providers/openai-responses-shared.ts:404, 887e20f898b2)

Likely related people:

  • scribe-dandelion-cult: Authored the merged empty non-image OpenAI tool-result fallback fix that current main uses to distinguish (no output) from image placeholders. (role: introduced current fallback behavior; confidence: high; commits: c556ded7269a, dd4e2abfb551; files: src/llm/providers/openai-completions.ts, src/llm/providers/openai-responses-shared.ts, src/agents/openai-transport-stream.ts)
  • vincentkoc: Merged the current-main fallback PR, so they are a good routing candidate for preserving that behavior during conflict resolution. (role: merger of adjacent fallback fix; confidence: medium; commits: dd4e2abfb551; files: src/llm/providers/openai-completions.ts, src/llm/providers/openai-responses-shared.ts, src/agents/openai-transport-stream.ts)
  • steipete: Merged and contributed hardening commits to the adjacent structured visible-output fix that this PR generalizes into provider replay. (role: recent visible-output hardening contributor and merger; confidence: medium; commits: 8d168c836af3, db76698b0d48; files: src/agents/embedded-agent-subscribe.tools.ts)
  • snowzlmbot: Although this PR author is external, they also authored the merged visible structured tool-result fallback and this branch's shared provider serializer. (role: adjacent merged feature contributor; confidence: medium; commits: 8d168c836af3, 887e20f898b2; files: src/agents/embedded-agent-subscribe.tools.ts, src/llm/providers/tool-result-text.ts, src/logging/tool-result-serialization.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@openclaw-barnacle openclaw-barnacle Bot removed the triage: needs-pr-context Candidate: external PR body lacks required problem context or evidence. label Jun 28, 2026
@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. P1 High-priority user-facing bug, regression, or broken workflow. merge-risk: 🚨 message-delivery 🚨 May drop, duplicate, misroute, suppress, or wrongly target messages. labels Jun 28, 2026
@snowzlmbot

snowzlmbot commented Jun 28, 2026

Copy link
Copy Markdown
Contributor Author

Updated the PR body with the final problem statement and CI/evidence summary. @clawsweeper review-pr

@clawsweeper clawsweeper Bot mentioned this pull request Jun 28, 2026
Merged
@snowzlmbot
snowzlmbot force-pushed the fix/96857-provider-context-placeholders branch from 1587650 to a5bc35b Compare June 28, 2026 10:12
@snowzlmbot

snowzlmbot commented Jun 28, 2026

Copy link
Copy Markdown
Contributor Author

Updated again after rebasing this PR branch onto the latest upstream/main and re-running current-head proof for the ClawSweeper findings in #97450 (comment).

What changed in the pushed branch:

Current-head proof summary:

$ git rev-parse --short=12 HEAD
a5bc35bac491
$ git diff --check upstream/main...HEAD
passed
$ git diff --name-only upstream/main...HEAD | grep CHANGELOG.md
no output (CHANGELOG.md restored; it is not in this PR diff)
$ node --experimental-strip-types /tmp/pr97450-proof.mjs
structured-only output: {"type":"resource","uri":"memory://result.json","mimeType":"application/json","data":{"ok":true,"items":2}}
mixed image+structured output: {"type":"resource","status":"metadata:ready","note":"data: is ordinary prose here"}
data URI token output: {"type":"resource","preview":"thumbnail=[redacted data URI] done"}

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Jun 28, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

@openclaw-barnacle openclaw-barnacle Bot added the triage: needs-pr-context Candidate: external PR body lacks required problem context or evidence. label Jun 28, 2026
@clawsweeper clawsweeper Bot added the merge-risk: 🚨 security-boundary 🚨 May affect sandboxing, authorization, credentials, or sensitive data. label Jun 28, 2026
@clawsweeper clawsweeper Bot added rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. and removed rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. labels Jun 28, 2026
@snowzlmbot

snowzlmbot commented Jun 28, 2026

Copy link
Copy Markdown
Contributor Author

Updated PR body and pushed provider/runtime sanitizer proof for current head 5ee5552: added authored What Problem This Solves / Evidence sections, shared bounded structured tool-result serialization, and affected OpenAI provider payload assertions for sensitive-field redaction, opaque/binary omission, circular handling, and truncation. @clawsweeper review-pr

@openclaw-barnacle openclaw-barnacle Bot added triage: needs-pr-context Candidate: external PR body lacks required problem context or evidence. and removed triage: needs-pr-context Candidate: external PR body lacks required problem context or evidence. labels Jun 28, 2026
@openclaw-barnacle openclaw-barnacle Bot added the scripts Repository scripts label Jun 28, 2026
@snowzlmbot

snowzlmbot commented Jun 28, 2026

Copy link
Copy Markdown
Contributor Author

Current head c788cc7: updated PR Evidence with the committed affected-path replay proof script (scripts/proofs/provider-tool-result-replay-proof.ts), exact reproduction command, and edited OpenAI Chat Completions / OpenAI Responses replay-output excerpt showing retained structured text plus sensitive-field redaction, opaque/binary omission, data URI redaction, image exclusion, and final aggregate truncation before provider replay. @clawsweeper review-pr

@clawsweeper clawsweeper Bot added rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. and removed rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. labels Jun 28, 2026
@snowzlmbot

snowzlmbot commented Jun 28, 2026

Copy link
Copy Markdown
Contributor Author

Current head c788cc7: refreshed PR Evidence after adding the replay proof artifact and clearing CI. The body now points to this head and includes the proof script path, reproduction command, edited OpenAI Chat Completions / OpenAI Responses replay-output excerpt, and broader #96857 out-of-scope/open wording. @clawsweeper review-pr

@clawsweeper clawsweeper Bot added the merge-risk: 🚨 automation 🚨 May affect CI, automerge, proof capture, label sync, or maintainer automation. label Jun 28, 2026
@snowzlmbot
snowzlmbot force-pushed the fix/96857-provider-context-placeholders branch from e8eddd4 to c788cc7 Compare June 28, 2026 15:49
@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. and removed rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. labels Jun 28, 2026
@snowzlmbot
snowzlmbot force-pushed the fix/96857-provider-context-placeholders branch from c788cc7 to a344a56 Compare June 28, 2026 15:56
@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. and removed status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. merge-risk: 🚨 automation 🚨 May affect CI, automerge, proof capture, label sync, or maintainer automation. labels Jun 28, 2026
@openclaw-barnacle openclaw-barnacle Bot removed the scripts Repository scripts label Jun 28, 2026
@clawsweeper clawsweeper Bot added status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. and removed proof: sufficient ClawSweeper judged the real behavior proof convincing. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. labels Jun 28, 2026
@snowzlmbot

snowzlmbot commented Jun 28, 2026

Copy link
Copy Markdown
Contributor Author

Focused rollback cleanup for current head a344a56: the PR branch is restored to the high-quality provider serializer patch surface, and proof-script/workflow-only artifacts have been removed from the PR diff/body. The body now reflects that any additional dependency-backed replay transcript should be attached externally without expanding the code surface. I am waiting for the remaining checks/review state before requesting another review.

@harjothkhara

harjothkhara commented Jun 28, 2026

Copy link
Copy Markdown
Contributor

One additional edge beyond the MIME-alias finding: the new provider replay helper only skips type: "image" before serializing every other non-text block (src/llm/providers/tool-result-text.ts:25-37). That means historical/adapter-shaped media blocks such as type: "audio" or type: "image_url" can now be turned into model-visible JSON text instead of staying on the media-only path. The old visible-output structured fallback deliberately skipped text, image, image_url, and audio (src/agents/embedded-agent-subscribe.tools.ts:350-357), and the PR body says image/audio payloads continue using existing multimodal handling rather than being serialized as text.

This matters for replay quality and the security boundary: audio happens to redact the raw data because carriesBinaryData() sees type === "audio", but it still injects an audio placeholder object into provider text; image_url is not skipped or classified by that check, so URL/data-URI-shaped image blocks can be serialized too. Please make the provider helper skip the same media-only types (image, image_url, audio) or pass skipTypes through to serializeStructuredToolResultBlock, and add a focused provider/helper regression so media-only blocks do not become replay text.

@snowzlmbot

snowzlmbot commented Jun 28, 2026

Copy link
Copy Markdown
Contributor Author

Current head a344a56: PR diff/body have been cleaned back to the focused provider serializer patch surface. Workflow/proof-script-only artifacts are no longer in the diff/body; latest Real behavior proof and check-lint are successful for this head. Please re-review the clean patch state. @clawsweeper review-pr

@snowzlmbot

snowzlmbot commented Jun 28, 2026

Copy link
Copy Markdown
Contributor Author

Current head 4a92599: media-only replay guard and MIME alias binary guard are now pushed and documented. The PR diff remains source/test-only (no workflow/proof-script artifacts), provider text replay skips image/image_url/audio, and mime_type is treated as a binary MIME alias. Please re-review this clean current head. @clawsweeper review-pr

@snowzlmbot snowzlmbot closed this Jun 29, 2026
@snowzlmbot
snowzlmbot force-pushed the fix/96857-provider-context-placeholders branch from 887e20f to 87db23e Compare June 29, 2026 00:33
@snowzlmbot

snowzlmbot commented Jun 29, 2026

Copy link
Copy Markdown
Contributor Author

Rebased this PR on the latest openclaw/main, resolved the replay-path conflicts, rebuilt the serializer slice, and updated the PR body with current-head proof for the review findings in #97450 (comment).

Highlights now covered in the PR body:

  • latest-main rebase point and current head SHA
  • mediaType, contentType, and content_type binary-omission coverage in the shared serializer tests
  • current-head redacted terminal proof showing structured text preserved while binary/opaque/secret/media payloads are omitted/redacted
  • current GitHub check links

@clawsweeper review-pr

@clawsweeper

clawsweeper Bot commented Jun 29, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper could not start a re-review for this item.

Reason: re-review requires an open issue or PR.

@github-actions github-actions Bot mentioned this pull request Jun 29, 2026
Open
@clawsweeper clawsweeper Bot mentioned this pull request Jun 29, 2026
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

agents Agent runtime and tooling merge-risk: 🚨 message-delivery 🚨 May drop, duplicate, misroute, suppress, or wrongly target messages. merge-risk: 🚨 security-boundary 🚨 May affect sandboxing, authorization, credentials, or sensitive data. P1 High-priority user-facing bug, regression, or broken workflow. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. size: XS status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@snowzlmbot @harjothkhara