Skip to content

fix: allow trusted host-read html after outbound staging#90579

Merged
steipete merged 1 commit into
openclaw:mainfrom
wangwllu:fix/trusted-outbound-html-staging
Jul 22, 2026
Merged

fix: allow trusted host-read html after outbound staging#90579
steipete merged 1 commit into
openclaw:mainfrom
wangwllu:fix/trusted-outbound-html-staging

Conversation

@wangwllu

@wangwllu wangwllu commented Jun 5, 2026

Copy link
Copy Markdown
Contributor

What Problem This Solves

Fixes #90557. Trusted generated HTML reports pass the initial host-read check under the OpenClaw temp root, but normal reply delivery copies them into media/outbound and a channel adapter later reloads that staged path with host-read enabled. The staged path is outside the trusted temp root, so the second load rejects it and the outbound reply can be lost.

The trust decision must survive a process boundary. Normal reply delivery can persist a write-ahead queue intent before platform I/O, and gateway startup reloads pending delivery payloads for retry. An in-memory marker would therefore be insufficient for restart recovery.

Why This Change Was Made

When resolveOutboundAttachmentFromUrl stages HTML that already passed the generated-HTML trust gate, it records the staged file's canonical realpath, SHA-256, byte size, provenance kind, version, and creation time in the shared SQLite state DB. loadWebMedia accepts outbound HTML only when containment, the provenance row, and the exact buffer already read all agree.

This does not trust the outbound directory by path. The existing regular-file, no-symlink, single-link, text-shape, and HTML-document checks remain in force. Outbound precedence also prevents an overlapping configured temp/state root from bypassing provenance. Marker-write failure removes the staged file and propagates; media cleanup prunes stale rows in bounded batches.

User Impact

Generated MEDIA:/tmp/openclaw/*.html reports can be delivered after outbound staging instead of being rejected by the channel adapter's second host-read check. Arbitrary, unmarked, or byte-replaced HTML under media/outbound remains rejected.

Evidence

Exact head: 28fe171dd9666352e19c59443e32cf8896734be3, rebased on d823cdc79496877883d8bba75691eac2e633fb99.

node scripts/run-vitest.mjs src/media/outbound-attachment.test.ts src/media/web-media.test.ts
# 2 files passed, 86 tests passed

node scripts/generate-kysely-types.mjs --verify
# exit 0

git diff --check
# exit 0

.agents/skills/autoreview/scripts/autoreview --mode branch --base origin/main --stream-engine-output
# autoreview clean: no accepted/actionable findings reported

Regression coverage includes exact-byte acceptance, same-size replacement rejection, unmarked outbound rejection, overlapping-root precedence, marker-write cleanup, stale-row pruning, transient filesystem-error retention, and refusal to mark paths outside outbound staging.

The original report reproduces the first trusted load followed by the rejected staged reload. Independent Linux reproduction and a real Telegram Bot API send in the PR discussion confirm the same gate and end-to-end delivery behavior.

Contributor credit is preserved in the commit as Co-authored-by: wanglu241 <[email protected]>.

@openclaw-barnacle openclaw-barnacle Bot added size: XS triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels Jun 5, 2026
@clawsweeper

clawsweeper Bot commented Jun 5, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed July 12, 2026, 6:45 AM ET / 10:45 UTC.

Summary
The PR adds exact-byte SQLite provenance so trusted generated HTML remains authorized after outbound staging and subsequent host-read reloads.

PR surface: Source +195, Tests +394, Generated +19. Total +608 across 8 files.

Reproducibility: yes. Current source traces trusted temporary-root HTML through outbound staging to a second host-read rejection, and independent macOS and Linux production reports reproduce the same behavior.

Review metrics: 1 noteworthy metric.

  • Persistent trust metadata: 1 SQLite table added. The additive table becomes part of the local HTML authorization lifecycle and merits explicit maintainer notice.

Stored data model
Persistent data-model change detected: database schema: src/state/openclaw-state-schema.generated.ts, database schema: src/state/openclaw-state-schema.sql, serialized state: src/media/web-media.test.ts, serialized state: src/state/openclaw-state-db.generated.d.ts, serialized state: src/state/openclaw-state-schema.generated.ts, serialized state: src/state/openclaw-state-schema.sql. Confirm migration or upgrade compatibility proof before merge.

Root-cause cluster
Relationship: fixed_by_candidate
Canonical: #90557
Summary: This PR is the active candidate fix for the canonical outbound-staged trusted HTML regression.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🦞 diamond lobster
Proof: 🦞 diamond lobster
Patch quality: 🦞 diamond lobster
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Risk before merge

  • [P1] Merging intentionally makes persisted exact-byte provenance an accepted trust signal for local HTML outside the original temporary root.
  • [P2] Provenance rows are reconciled by the media-retention pass rather than bounded independently by age; installations where maintenance does not run could accumulate rows, although this is non-blocking hardening rather than a demonstrated correctness defect.

Maintainer options:

  1. Approve the exact-byte trust lifecycle (recommended)
    Accept provenance-matched staged HTML as trusted and merge after normal exact-head gates.
  2. Decline trust outside the temporary root
    Pause or close this PR and preserve root-only trust through a different delivery architecture.

Next step before merge

  • [P2] The patch needs explicit owner acceptance of the permanent staged-HTML trust boundary; no narrow automated repair remains.

Maintainer decision needed

  • Question: Should OpenClaw accept SHA-256 plus byte-size provenance in the shared state database as the permanent trust mechanism for generated HTML copied into media/outbound and later reloaded with host-read enabled?
  • Rationale: The implementation is correct, narrowly scoped, and fail-closed, but changing what evidence authorizes local HTML outside the original trusted root is a security-policy choice requiring maintainer intent.
  • Likely owner: steipete — They authored and merged the existing trusted generated HTML boundary and are the strongest available owner for extending it.
  • Options:
    • Accept content-bound provenance (recommended): Approve the exact-byte staged trust lifecycle and merge after the current-head mergeability and required checks settle.
    • Keep temp-root-only trust: Reject this boundary extension and redesign delivery so trusted HTML is never reauthorized from outbound staging.

Security
Cleared: The exact-head diff is content-bound, path-contained, read-once, and fail-closed, with no concrete security or supply-chain defect found.

Review details

Best possible solution:

Merge the focused content-bound implementation after the trust-boundary owner explicitly accepts persisted exact-byte provenance; track row-bounding and broader send-stage graceful degradation separately if desired.

Do we have a high-confidence way to reproduce the issue?

Yes. Current source traces trusted temporary-root HTML through outbound staging to a second host-read rejection, and independent macOS and Linux production reports reproduce the same behavior.

Is this the best way to solve the issue?

Yes technically. Content-bound read-once provenance is the narrowest maintainable repair found and is substantially safer than trusting the outbound directory by path; only security-policy acceptance remains.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 75f0960b71be.

Label changes

Label justifications:

  • P1: Independent production evidence shows the current regression silently and permanently loses complete Telegram replies.
  • merge-risk: 🚨 message-delivery: The changed second-load trust gate directly determines whether staged generated HTML and its reply are delivered.
  • merge-risk: 🚨 security-boundary: The PR authorizes exact provenance-matched HTML bytes outside the original trusted temporary root.
  • rating: 🦞 diamond lobster: Overall readiness is 🦞 diamond lobster; proof is 🦞 diamond lobster and patch quality is 🦞 diamond lobster.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (live_output): Deterministic production-helper before/after output plus independent live Telegram Bot API delivery and tamper rejection convincingly show the fixed behavior; the earlier non-delta Mantis run is not relied upon.
  • proof: sufficient: Contributor real behavior proof is sufficient. Deterministic production-helper before/after output plus independent live Telegram Bot API delivery and tamper rejection convincingly show the fixed behavior; the earlier non-delta Mantis run is not relied upon.
  • mantis: telegram-visible-proof: Mantis should capture Telegram visible proof. The change visibly determines whether Telegram receives the generated HTML document, and independent live-send evidence demonstrates that result.
Evidence reviewed

PR surface:

Source +195, Tests +394, Generated +19. Total +608 across 8 files.

View PR surface stats
Area Files Added Removed Net
Source 4 209 14 +195
Tests 2 395 1 +394
Docs 0 0 0 0
Config 0 0 0 0
Generated 2 19 0 +19
Other 0 0 0 0
Total 8 623 15 +608

What I checked:

  • Content-bound host-read trust: Outbound HTML trust requires the loaded buffer's size and SHA-256 to match persisted provenance after text and HTML-shape validation, so authorization and use operate on the same read. (src/media/web-media.ts:469, a79ecb27ec7f)
  • Marker creation boundary: Marker creation resolves the staged path, refuses paths outside the real outbound directory, and stores the digest and byte count in a synchronous state transaction. (src/media/web-media.ts:394, a79ecb27ec7f)
  • Staging failure behavior: Only media already identified as trusted generated HTML receives provenance; a marker-write failure removes the staged file best-effort and propagates the error instead of returning an unusable path. (src/media/outbound-attachment.ts:31, a79ecb27ec7f)
  • Provenance lifecycle: The normal media-retention pass reconciles provenance after filesystem pruning, and the gateway maintenance caller catches and logs cleanup failures without crashing the process. (src/media/store.ts:218, a79ecb27ec7f)
  • Upgrade behavior: The provenance table is additive and is created through the normal shared-state schema initialization when an existing database is opened; no manual operator migration is required. (src/state/openclaw-state-db.ts:1535, a79ecb27ec7f)
  • Regression coverage: Added tests cover marked delivery, unmarked rejection, replaced bytes, symlink escape, recorded digest fields, cleanup, and marker-write failure. (src/media/web-media.test.ts:703, a79ecb27ec7f)

Likely related people:

  • steipete: Authored and merged the original trusted generated HTML boundary and remains the clearest history owner for the affected media trust and staging paths. (role: introduced behavior and recent area contributor; confidence: high; commits: 68d0c0f2f5c4, 55254c3a0be8; files: src/media/web-media.ts, src/media/outbound-attachment.ts, src/media/store.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.
Review history (3 earlier review cycles)
  • reviewed 2026-07-01T03:22:03.007Z sha 96f7e87 :: needs maintainer review before merge. :: none
  • reviewed 2026-07-09T02:46:39.119Z sha 0d2cbad :: needs maintainer review before merge. :: none
  • reviewed 2026-07-09T02:53:54.723Z sha 0d2cbad :: needs maintainer review before merge. :: none

@openclaw-barnacle openclaw-barnacle Bot added proof: supplied External PR includes structured after-fix real behavior proof. and removed triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels Jun 5, 2026
@clawsweeper clawsweeper Bot added rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. mantis: telegram-visible-proof Mantis should capture Telegram visible proof. P1 High-priority user-facing bug, regression, or broken workflow. merge-risk: 🚨 security-boundary 🚨 May affect sandboxing, authorization, credentials, or sensitive data. labels Jun 5, 2026
@clawsweeper clawsweeper Bot added rating: 🌊 off-meta tidepool PR readiness rating does not apply to this item. and removed rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. mantis: telegram-visible-proof Mantis should capture Telegram visible proof. labels Jun 5, 2026
@wangwllu

wangwllu commented Jun 6, 2026

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Jun 6, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. mantis: telegram-visible-proof Mantis should capture Telegram visible proof. and removed rating: 🌊 off-meta tidepool PR readiness rating does not apply to this item. labels Jun 6, 2026
@wangwllu

wangwllu commented Jun 6, 2026

Copy link
Copy Markdown
Contributor Author

Patch ready for maintainer review

The patch has converged through three iterations addressing each ClawSweeper round:

Iteration Approach ClawSweeper verdict
07aa73b0 Whole media/outbound as trusted root rejected — too broad
a3367d531b .trust.json sidecar rejected — AGENTS.md L75-76 (no JSON sidecar runtime state)
ace52b0c1f (head) SQLite outbound_media_provenance table + cleanup platinum hermit patch quality, all patch-side findings cleared

The current head:

  • Uses a Kysely-accessed SQLite table on the shared state DB (matching AGENTS.md "SQLite only" canonical-storage policy).
  • Provenance rows are deleted in lockstep with the staged file via pruneStaleTrustedGeneratedHtmlMarkers() wired into cleanOldMedia (src/media/store.ts).
  • Negative test: an outbound .html written without markTrustedGeneratedHtmlPath is still rejected by the host-read trust gate.
  • Vitest green: web-media, outbound-attachment, reply-media-paths, openclaw-state-db (all touched suites pass).

Two remaining gates need maintainer judgement

1. Security boundary acceptance. ClawSweeper P1: "Merging expands the trusted host-read HTML boundary from temp-root files to outbound-staged files with a SQLite provenance marker, so maintainers need to accept that security lifecycle explicitly." The expansion is provenance-gated (only paths that the host-read gate already trusted at source time are marked), and the marker is best-effort cleared by the same cleanup pass that removes the file. Asking explicitly: is this trust lifecycle acceptable, or is there a tighter binding you'd prefer?

2. Real behavior proof. ClawSweeper has been asking for a Mantis Telegram Desktop capture across all three iterations; only a maintainer can trigger it. If you'd like to fire it, the suggested invocation (per ClawSweeper's own guidance) is:

@openclaw-mantis telegram desktop proof: verify a generated MEDIA:/tmp/openclaw/*.html report is delivered as a Telegram document after outbound staging.

@steipete — pinging since you authored the original trusted-generated-HTML host-read seam (commit 68d0c0f) and have the strongest context for the trust-boundary acceptance call.

@clawsweeper clawsweeper Bot added rating: 🌊 off-meta tidepool PR readiness rating does not apply to this item. and removed rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels Jun 6, 2026
@clawsweeper clawsweeper Bot added rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. and removed rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. labels Jun 24, 2026
@wangwllu

Copy link
Copy Markdown
Contributor Author

Targeted follow-up after the latest ClawSweeper verdict

ClawSweeper has now re-reviewed the current head (2dcfc944ac0739391228bdf624b54ea617181cf4). The two previous code findings are addressed:

  • outbound provenance is now content-bound (sha256 + size_bytes) and verified against the already-read loadWebMedia buffer, so there is no hash-then-read TOCTOU window;
  • marker write failures now best-effort unlink the staged file and rethrow instead of silently returning an untrusted staged path.

The remaining gate is proof / maintainer acceptance of the intentional trust-boundary expansion:

Accept outbound trusted-HTML provenance lifecycle — the patch authorizes outbound-staged HTML when SQLite provenance and content digest match, extending trusted local HTML host-read behavior beyond the temp root.

Why the previous Mantis proof did not exercise this bug

The previous Mantis artifact showed both baseline and candidate delivering HTML. That is consistent with the earlier diagnosis: the Mantis SUT did not enable the host-read gate for the second staged reload path, so both lanes bypassed isTrustedGeneratedHostReadHtmlPath and therefore could not show the pre-fix failure.

For this regression, the proof stimulus must explicitly hit this sequence:

  1. generated report HTML is written under resolvePreferredOpenClawTmpDir();
  2. the original /tmp/openclaw/*.html source is loaded as trusted generated HTML;
  3. resolveOutboundAttachmentFromUrl stages a copy into media/outbound and writes provenance;
  4. the Telegram send/reply path reloads the staged outbound path via loadWebMedia(..., hostReadCapability: true);
  5. baseline rejects at the second host-read HTML gate, while this PR accepts only if the staged bytes match the recorded digest and then delivers the Telegram document.

Requested targeted proof

@openclaw-mantis telegram desktop proof: please verify PR #90579 with host-read enabled on the staged reload path. The stimulus should drive a generated MEDIA:/tmp/openclaw/*.html report through outbound staging and then through the Telegram channel adapter's second loadWebMedia(stagedPath, { hostReadCapability: true }) reload before delivery. Expected result: baseline/main rejects the staged outbound .html at the trusted-host-read gate (or logs the OutboundDeliveryError), while this PR delivers the Telegram document and logs/records that the staged path matched its SQLite provenance digest.

Useful acceptance signals for the artifact/logs:

  • baseline/main: the second load of media/outbound/*.html reaches the host-read HTML gate and rejects because the path is no longer under the temp root;
  • candidate (2dcfc944): the same staged path has an outbound_media_provenance row with sha256 + size_bytes, the reload hashes the already-read buffer and matches it, then Telegram sends the document;
  • a negative/tamper check is not required for the video proof because it is covered by the new regression test: rejects an outbound-staged HTML whose bytes were replaced after the provenance marker (content-bound trust).

Maintainer/security acceptance ask

If the targeted proof is sufficient, could a maintainer or security owner explicitly accept this lifecycle?

The proposed boundary is intentionally narrow: temp-root generated HTML remains unchanged; outbound-staged HTML is trusted only when (a) it was staged from a previously trusted generated-HTML source, (b) the staged realpath has a SQLite provenance row, and (c) the bytes read during host-read match the recorded sha256 + size_bytes. Marker failures fail closed, and stale rows are pruned with the media-store cleanup.

@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed proof: 🎥 video Contributor real behavior proof includes video or recording evidence. rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels Jun 24, 2026
@wangwllu
wangwllu force-pushed the fix/trusted-outbound-html-staging branch from 2dcfc94 to 4f4dccc Compare June 26, 2026 02:43
wangwllu pushed a commit to wangwllu/openclaw that referenced this pull request Jun 26, 2026
- Remove dead `clearTrustedGeneratedHtmlMarker` export (lifecycle already
  owned by `pruneStaleTrustedGeneratedHtmlMarkers` via `cleanOldMedia`).
- `markTrustedGeneratedHtmlPath` now asserts the realpath is inside the
  outbound staging dir; tightens the API contract independently of the
  reader-side containment check.
- Outbound attachment staging tolerates ENOENT race on marker write
  (file unlinked between save and mark): warn + continue; downstream
  gate still rejects if the file actually vanished.
- `hasTrustedGeneratedHtmlMarker` catches state-DB failures (locked,
  corrupt, permissions) and returns false, so a broken DB degrades
  to the existing path-not-allowed gate rejection rather than a
  thrown non-LocalMediaAccessError.
- Add `logVerbose` for trust-gate decisions and prune sweep counts;
  the original bug surfaced as a silent drop, observability would
  have shortened the diagnostic loop.
- Add regression test for symlink-escape: marker call throws and the
  reader-side gate rejects even with a forged provenance row.

Addresses post-review concerns from independent review at
openclaw#90579 .
@wangwllu

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

Rebased onto current main (was based on a stale fork point → showed CONFLICTING). Single 1-line import conflict in src/media/web-media.ts resolved as the union (extractOriginalFilename + getMediaDir; both are used post-rebase). All 9 commits replayed cleanly; focused diff unchanged (8 files, src/media/* + generated src/state/*). New head 4f4dccc48f, now MERGEABLE/CLEAN with CI green. Please refresh the review on the new head.

@clawsweeper

clawsweeper Bot commented Jun 26, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

@clawsweeper clawsweeper Bot added status: 🔁 re-review loop A fresh ClawSweeper review was explicitly requested after the latest review. and removed status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. labels Jun 30, 2026
@wangwllu
wangwllu force-pushed the fix/trusted-outbound-html-staging branch from 4f4dccc to 96f7e87 Compare July 1, 2026 03:13
wangwllu pushed a commit to wangwllu/openclaw that referenced this pull request Jul 1, 2026
- Remove dead `clearTrustedGeneratedHtmlMarker` export (lifecycle already
  owned by `pruneStaleTrustedGeneratedHtmlMarkers` via `cleanOldMedia`).
- `markTrustedGeneratedHtmlPath` now asserts the realpath is inside the
  outbound staging dir; tightens the API contract independently of the
  reader-side containment check.
- Outbound attachment staging tolerates ENOENT race on marker write
  (file unlinked between save and mark): warn + continue; downstream
  gate still rejects if the file actually vanished.
- `hasTrustedGeneratedHtmlMarker` catches state-DB failures (locked,
  corrupt, permissions) and returns false, so a broken DB degrades
  to the existing path-not-allowed gate rejection rather than a
  thrown non-LocalMediaAccessError.
- Add `logVerbose` for trust-gate decisions and prune sweep counts;
  the original bug surfaced as a silent drop, observability would
  have shortened the diagnostic loop.
- Add regression test for symlink-escape: marker call throws and the
  reader-side gate rejects even with a forged provenance row.

Addresses post-review concerns from independent review at
openclaw#90579 .
@wangwllu

wangwllu commented Jul 2, 2026

Copy link
Copy Markdown
Contributor Author

Hi @vincentkoc — sorry for the direct tag; I'm trying to route this to the right maintainer rather than bump the PR blindly. This PR only touches the media outbound/trust-gate path, and I noticed you've recently worked on outbound-attachment.ts / store.ts, so I wanted to ask whether you're the right person for this area or if there's a preferred reviewer I should redirect to. CI is green and the PR body is up to date; as far as I can tell there isn't a CODEOWNERS-required reviewer for these files. Happy to be redirected if someone else owns this seam.

@wanpidan1234-design

wanpidan1234-design commented Jul 8, 2026

Copy link
Copy Markdown

Independent production repro of #90557 + security review of this fix

Context: hit this bug in production today and root-caused it independently before finding #90557 — our conclusion matches the issue exactly. Sharing the repro as a second environment data point, plus a review of the fix design. (Reviewed at commit 96f7e87.)

Independent reproduction (Linux, v2026.6.8, Telegram direct chat)

Environment: OpenClaw 2026.6.8 (844f405), Node v22.22.1, Linux 6.17 (GCP), Telegram bot account, direct chat, single-agent setup. /tmp/openclaw is mode 0700, owned by the gateway uid, so resolvePreferredOpenClawTmpDir() resolves to it (verified by importing dist/tmp-openclaw-dir-*.js directly).

  1. Agent reply emits MEDIA:/tmp/openclaw/report.html (regular file, nlink=1, valid UTF-8, <!doctype html shape, 551 KB) → gateway log:
    telegram final reply failed: OutboundDeliveryError: hostReadCapability permits only validated plain-text documents and trusted generated HTML reports for local reads
    
  2. Confirmed the split-brain mechanism with a harness that imports the shipped dist modules (buildOutboundMediaLoadOptions + agent-scoped readFile + loadWebMedia):
  3. Two impact confirmations for the labels on [Bug]: Telegram outbound rejects trusted generated HTML after staging strips trusted-root marker (PR #87982 missed e2e case) #90557:
    • impact:message-loss is worse than "attachment dropped": when the rejection happens at the channel-adapter load (send time), the entire reply text is lost, not just the media. Normalize-stage drops degrade gracefully (⚠️ Media failed. appended); send-stage rejections do not.
    • The failed delivery then hits the reconnect drain as unrecoverable:
      Delivery entry <uuid> delivery state is send_attempt_started; refusing blind replay without adapter reconciliation
      
      so the message is permanently gone even after reconnect.

Review of the fix design (content-bound provenance)

Read the full diff. The core design — bind trust to sha256+size of the staged bytes in outbound_media_provenance, keyed by realpath, verified against the exact buffer being delivered — is the right shape, and notably better than extending path-based trust to the staging dir:

  • No TOCTOU between verify and use: isTrustedGeneratedHostReadHtml hashes params.buffer, i.e. the same bytes that get delivered, not a separate read. The replacement-attack test (rejects an outbound-staged HTML whose bytes were replaced after the provenance marker) covers the obvious attack.
  • Fail-closed everywhere I traced: marker lookup failure → untrusted; DB open failure → untrusted; marker-write failure → staged file unlinked and error propagated (no marker-less trusted file, no orphaned trusted marker). Crash between stage and mark leaves a file without a row → rejected. Correct default in every branch.
  • Symlink/hardlink escapes: markTrustedGeneratedHtmlPath refuses realpaths outside the outbound root; reader side keeps the nlink !== 1 and isSymbolicLink() rejects and re-resolves realpath before the row lookup; the forged-row + symlink test covers the nastier combination.
  • Trust transitivity is sound: re-staging an already-staged outbound file re-marks the new copy only after the loaded buffer was hash-verified against the existing row, so the chain stays content-bound.

Three minor points, none blocking:

  1. Row growth between retention passes. Rows are only reconciled inside cleanOldMedia. On installs where the retention pass rarely runs, outbound_media_provenance grows one row per trusted HTML send. A created_at_ms-based TTL fallback inside pruneStaleTrustedGeneratedHtmlMarkers (delete rows older than the media TTL regardless of file state) would bound it cheaply.
  2. WHERE realpath IN (…) chunking. pruneStaleTrustedGeneratedHtmlMarkers builds a single IN list from all stale rows. Fine at realistic volumes, but chunking (e.g. 500/batch) would make it immune to SQLite bound-variable limits on pathological tables.
  3. Send-stage failure mode (follow-up, not this PR): as observed above, a send-time host-read rejection still destroys the whole reply. Even with this fix merged, any future rejection path at the adapter load will have the same blast radius. Worth a follow-up issue: degrade send-stage media rejections to text-with-warning, matching the normalize-stage behavior. Happy to file it if useful.

CI is green on the PR head (check-test-types / check-lint / build-artifacts / QA Smoke), and the added tests cover replacement, forged-row, symlink, prune-lockstep, and marker-write-failure paths. We could not rebuild the PR locally (disk constraints on the repro box), so this review relies on CI for the unit suite and on the dist-level harness above for the behavioral before-state.

Verdict: the design is correct and fail-closed; would like to see this merged. The production data above hopefully helps the security review along — this bug currently causes silent, unrecoverable loss of entire replies on a default-ish Linux + Telegram setup.


Disclosure: AI-assisted — the repro harness, log analysis, and review draft were produced with an AI agent under my direction; I reviewed the findings and evidence before posting. Flagging per the repo's AI-transparency guideline.


Update (same day): empirical confirmation that the second gate is the only blocker.

To unblock our own instance we deployed a minimal local variant of this fix onto the shipped 2026.6.8 dist — path-trust for the outbound staging dir only (no provenance table; deliberately weaker than this PR, single-user box). Result, same Linux + Telegram setup as the repro above:

  • MEDIA:*.html reply → staged → send-stage host-read re-check passes → telegram outbound send ok … operation=sendDocument, attachment delivered end-to-end;
  • negative control: HTML outside the temp root / staging dir is still rejected.

This is not a test of this PR's code — but it empirically confirms the PR's core premise: once the staged copy is recognized as trusted, delivery works end-to-end; there is no additional hidden gate downstream of the one this PR fixes. We'll drop our local variant in favor of this PR's content-bound design as soon as it ships.

(Same AI-assisted disclosure as above applies.)

@wanpidan1234-design

Copy link
Copy Markdown

Follow-up: independent local verification of this PR (built at 96f7e87)

Earlier I posted an independent repro of #90557 (comment). I've now built this branch locally and verified the fix at four levels:

1. PR unit testssrc/media/outbound-attachment.test.ts + src/media/web-media.test.ts: 86/86 pass.

2. src/media/ suite — 626 pass / 17 fail; all 17 failures are in modules this PR does not touch (ffmpeg exec, audio transcode, proxied fetch, QR terminal — pre-existing environment issues on my host, cross-checked against gh pr diff --name-only). Every file this PR touches passes.

3. Independent no-mock verification — my own test (real fs + real SQLite + real loadWebMedia, no stubs), four scenarios:

  • Trusted tmp-root HTML → staged via resolveOutboundAttachmentFromUrl → send-side re-read of the staged copy is allowed (this is the exact step that silently dropped whole replies in [Bug]: Telegram outbound rejects trusted generated HTML after staging strips trusted-root marker (PR #87982 missed e2e case) #90557) ✅
  • Same staged file without a provenance row → still rejected (path-not-allowed; the security boundary is not weakened) ✅
  • Tampering with staged bytes after marking → rejected (sha256 content binding holds) ✅
  • Fault injection (state DB made unwritable) → marker write failure propagates loudly and the staged file is cleaned up — no orphaned file, no deferred silent drop ✅

4. Real-send E2E — using this PR build against the live Telegram Bot API (isolated OPENCLAW_STATE_DIR, no gateway process): full chain stage → provenance write → trust-gate re-read → sendDocument succeeded and the .html document was actually delivered to a real chat. On the pre-fix release this same flow is silently rejected at the trust gate.

One tip for other verifiers: don't start a second gateway against the same bot token to test this — it competes for getUpdates and steals update offsets from your production instance. Calling resolveOutboundAttachmentFromUrl + sendMessageTelegram directly is sufficient and side-effect free.

From my side this fixes the production silent-drop we hit, and the worst-case path is now loud instead of silent. LGTM.

@wangwllu

wangwllu commented Jul 9, 2026

Copy link
Copy Markdown
Contributor Author

Rebased onto current main (was BEHIND/CONFLICTING)

main had moved ahead, so this PR went CONFLICTING. Rebased the 9 commits onto the current main tip; head is now 0d2cbad534, MERGEABLE.

Conflict surface was generated schema only — the two hand-written source areas (web-media.ts, store.ts, outbound-attachment.ts) auto-merged cleanly. The only conflicts were in the SQLite state schema, because main added new tables (worktrees, skill_*, audit_events, clawhub_*, …) in the same tail region where this PR appends outbound_media_provenance. Resolved as the union (keep both), then regenerated the derived files via pnpm db:kysely:gen rather than hand-editing them — db:kysely --check passes, so .generated.ts / .d.ts are in sync with .sql.

Focused diff is unchanged: still the same 8 files (src/media/* + generated src/state/*), no unrelated changes pulled in.

Local verification on the rebased head:

  • src/media/web-media.test.ts (81) + src/media/store.test.ts (47) + src/media/outbound-attachment.test.ts (5) → 133/133 pass.
  • tsgo prod typecheck: zero errors in the changed surface (src/media, src/state).

Re: the three non-blocking suggestions from @wanpidan1234-design (created_at_ms TTL fallback in prune, IN (...) chunking, and send-stage degrade-to-text) — all reasonable; I'd prefer to keep this PR's reviewed diff intact and handle them as a follow-up rather than re-open the reviewed surface here. The send-stage degrade one is explicitly a separate message-loss concern (its own blast radius), so it belongs in its own issue.

@wangwllu

wangwllu commented Jul 9, 2026

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

@wangwllu

Copy link
Copy Markdown
Contributor Author

Rebased onto current main (was CONFLICTING/DIRTY)

main advanced past the 2026-07-09 rebase point, so this PR flipped to CONFLICTING. Rebased the 9 commits onto the current main tip (0350b87f06); new head a79ecb27ec, back to MERGEABLE.

Conflict surface was generated schema only — the hand-written sources (web-media.ts, outbound-attachment.ts, store.ts) auto-merged cleanly. The only conflicts were in the SQLite state schema, where main appended new tables (session_groups, worker_environments, worker_transcript_*, fleet_cells, …) in the same tail region where this PR appends outbound_media_provenance. Resolved as the union (kept both), then regenerated the derived files via db:kysely:gen rather than hand-editing them (db:kysely:check passes, so .generated.ts / .d.ts are in sync with .sql).

Focused diff unchanged: still the same 8 files (src/media/* + generated src/state/*), no unrelated changes pulled in.

Local verification on the rebased head (a79ecb27ec):

  • web-media.test.ts (81) + store.test.ts (48) + outbound-attachment.test.ts (5) → 134/134 pass.
  • tsgo:core prod typecheck: 0 errors across the changed surface (src/media, src/state).

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Jul 12, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

@openclaw-barnacle

Copy link
Copy Markdown

This assigned pull request has been automatically marked as stale after being open for 27 days.
Please add updates or it will be closed.

@clawsweeper

clawsweeper Bot commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

ClawSweeper status: review started.

I am starting a fresh review of this pull request: fix: allow trusted host-read html after outbound staging This is item 1/1 in the current shard. Shard 0/1.

This placeholder means the worker is alive and reading the current context. I will edit this same comment with the actual review when the claws are done clicking.

Crustacean status: shell secured, claws on keyboard, evidence pebbles being sorted.

@steipete

Copy link
Copy Markdown
Contributor

Merged via squash.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

mantis: telegram-visible-proof Mantis should capture Telegram visible proof. merge-risk: 🚨 message-delivery 🚨 May drop, duplicate, misroute, suppress, or wrongly target messages. merge-risk: 🚨 security-boundary 🚨 May affect sandboxing, authorization, credentials, or sensitive data. P1 High-priority user-facing bug, regression, or broken workflow. proof: sufficient ClawSweeper judged the real behavior proof convincing. proof: supplied External PR includes structured after-fix real behavior proof. rating: 🦞 diamond lobster Very strong PR readiness with only minor maintainer review expected. size: L stale Marked as stale due to inactivity status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Telegram outbound rejects trusted generated HTML after staging strips trusted-root marker (PR #87982 missed e2e case)

3 participants