fix(tlon): bound urbit scry JSON response reads by hugenshen · Pull Request #100376 · openclaw/openclaw

hugenshen · 2026-07-05T16:19:33Z

What Problem This Solves

Tlon Urbit scry reads could cause out-of-memory failures when a configured ship endpoint returned an oversized JSON body on the success path. Poke error bodies were already bounded, but scry success responses still used unbounded response.json().

Why This Change Was Made

scryUrbitPath now uses the shared readProviderJsonResponse helper, which caps successful reads at 16 MiB and cancels the upstream body on overflow. The error label retains the exact scry path, and the existing finally continues to release guarded-fetch resources on success and failure.

The maintainer pass removed duplicate 18 MiB unit-stream scaffolding while retaining the real loopback HTTP boundary scenario. Urbit documents scry as authenticated HTTP GET returning the requested data; this patch changes only the local response bound.

User Impact

Oversized Tlon scry responses now fail with a bounded, path-specific error instead of risking gateway OOM. Normal small JSON and existing non-success HTTP behavior are unchanged.

Evidence

Exact head: 110e0daea1ad6abfbaa9a77a22f56f704a537b42.
Sanitized public-network AWS Crabbox cbx_8ae94556bc3b, run run_3863cc067902: no IAM role, Tailscale, or hydration; 5/5 focused tests passed on behavior-equivalent head 3ccce220b4924b6bd17ef5f8557642a0c17c7eb8.
Source-blind before/after real loopback HTTP proof:
- before: current main buffered and parsed all 18,874,368 payload bytes;
- after: the patch failed at the 16,777,216-byte ceiling and cancellation stopped the server after 17,235,968 bytes.
Exact-head hosted CI run 28763080955: all required checks passed after attempt 2 cleared an unrelated TUI PTY timing flake.
The direct rebased head descends current main and contains only the three Tlon files; tree hash 260853298300c82388679cbeef32a104081fffda.
Fresh post-sync autoreview: clean, no actionable findings, confidence 0.97.
Contributor-credit changelog entries for fix(agents): preserve streamed assistant text when Claude CLI result event is empty #90450 (@totobusnello), fix(channels): normalize phone identities with stray plus signs #100467 (@morluto), and this fix (@hugenshen) are deferred to one final maintainer changelog PR after this runtime-only branch lands.
Targeted oxfmt and git diff --check: pass.
External hosted Urbit credentials were unavailable; loopback exercised the production urbitFetch and guarded response-reader path.

Risk checklist

User-visible behavior: yes, oversized scry responses now fail early.
Config/environment/migration: no change.
Security/auth/network/tool execution: narrows the remote response boundary; no auth or SSRF policy change.
Highest risk: unusually large legitimate scry payloads.
Mitigation: shared 16 MiB provider JSON contract, normal-response proof, path-specific diagnostic.

Related context

fix(extensions): bound JSON response reads to prevent OOM #97558 contains a broader overlapping response-bounds batch; after this focused PR lands, its Tlon scry slice should be removed while distinct work remains.
Official Urbit contracts: https://docs.urbit.org/urbit-os/kernel/eyre/external-api-ref and https://docs.urbit.org/build-on-urbit/tools/js-libs/http-api-guide

AI-assisted disclosure

AI-assisted maintainer rewrite and review
Source-blind before/after behavior proof
Fresh structured autoreview

clawsweeper · 2026-07-05T16:21:12Z

Codex review: needs maintainer review before merge. Reviewed July 5, 2026, 10:12 PM ET / 02:12 UTC.

Summary
The PR replaces Tlon scry success parsing with the shared bounded provider JSON reader and adds malformed, normal, and oversized response coverage.

PR surface: Source +3, Tests +53. Total +56 across 3 files.

Reproducibility: yes. Current main reaches unbounded response.json() on successful Tlon scry responses, and the PR body plus maintainer comments provide before/after loopback HTTP proof for the same path.

Review metrics: 1 noteworthy metric.

Bounded success-read paths: 1 changed from unbounded to 16 MiB. This is the compatibility boundary maintainers should notice before merging the Tlon scry fix.

Stored data model
Persistent data-model change detected: serialized state: extensions/tlon/src/urbit/error-body-boundary.test.ts. Confirm migration or upgrade compatibility proof before merge.

Merge readiness
Overall: 🐚 platinum hermit
Proof: 🦞 diamond lobster
Patch quality: 🐚 platinum hermit
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

Resolve or classify the current compact-shard failure before landing.

Risk before merge

[P1] Successful Tlon scry JSON responses over 16 MiB now fail early instead of being accepted, so maintainers should own that compatibility boundary even though the PR discussion records acceptance of the shared ceiling.
[P1] Live checks were not fully green at review time; this is a merge-gate issue rather than a concrete code finding from the diff.

Maintainer options:

Land With Shared Ceiling (recommended)
Accept that successful Tlon scry responses above 16 MiB now fail early and land once normal merge gates are green.
Pause For Tlon-Specific Policy
Ask for a Tlon-specific size policy only if maintainers believe legitimate scry responses need a different supported ceiling.

Next step before merge

No automated repair is needed; maintainers need to land or rerun/clear current checks while accepting the shared 16 MiB Tlon scry response ceiling.

Security
Cleared: The diff narrows a remote response-body boundary and does not change dependencies, workflows, permissions, secrets, package metadata, or code execution paths.

Review details

Best possible solution:

Land this focused Tlon owner-local fix after normal merge gates are green, then remove the overlapping Tlon slice from the broader response-bounds PR while leaving its unrelated plugin work reviewable.

Do we have a high-confidence way to reproduce the issue?

Yes. Current main reaches unbounded response.json() on successful Tlon scry responses, and the PR body plus maintainer comments provide before/after loopback HTTP proof for the same path.

Is this the best way to solve the issue?

Yes. The patch keeps the fix inside the Tlon owner boundary and reuses the existing SDK bounded JSON helper instead of adding a Tlon-specific reader or waiting for a broader multi-plugin sweep.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against d2d5911c3296.

Label changes

Label justifications:

P2: This is a normal-priority Tlon reliability fix with limited blast radius and focused coverage.
merge-risk: 🚨 compatibility: Successful Tlon scry responses above the shared 16 MiB cap change from accepted input to an early failure.
rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (logs): The PR body and maintainer comments include source-blind loopback HTTP proof showing current main buffered an oversized scry response and the patched path failed at the 16 MiB cap with cancellation.
proof: sufficient: Contributor real behavior proof is sufficient. The PR body and maintainer comments include source-blind loopback HTTP proof showing current main buffered an oversized scry response and the patched path failed at the 16 MiB cap with cancellation.

Evidence reviewed

PR surface:

Source +3, Tests +53. Total +56 across 3 files.

View PR surface stats

Area	Files	Added	Removed	Net
Source	1	10	7	+3
Tests	2	66	13	+53
Docs	0	0	0	0
Config	0	0	0	0
Generated	0	0	0	0
Other	0	0	0	0
Total	3	76	20	+56

What I checked:

Current main behavior: Successful Tlon scry responses still reach native response.json() after the HTTP status check, so current main has the unbounded success-body read this PR targets. (extensions/tlon/src/urbit/channel-ops.ts:96, d2d5911c3296)
PR implementation: The PR head imports readProviderJsonResponse and uses it for successful scry responses with a path-specific label, while preserving the existing finally release. (extensions/tlon/src/urbit/channel-ops.ts:102, 110e0daea1ad)
Shared reader contract: readProviderJsonResponse defaults to the shared 16 MiB cap and reports labeled malformed JSON or overflow errors. (src/agents/provider-http-errors.ts:315, d2d5911c3296)
Overflow cancellation contract: readResponseWithLimit cancels the reader when the next chunk would exceed the cap and releases the lock afterward. (src/infra/http-body.ts:228, d2d5911c3296)
PR regression coverage: The PR adds loopback HTTP coverage for normal scry JSON and an oversized streaming success response that now fails with the path-specific 16 MiB error. (extensions/tlon/src/urbit/error-body-boundary.test.ts:103, 110e0daea1ad)
Runtime callers checked: Current production callers are the Tlon storage configuration path and the SSE client scry method, so the bounded reader sits in the owner-local shared Tlon scry helper rather than one caller. (extensions/tlon/src/tlon-api.ts:167, d2d5911c3296)

Likely related people:

steipete: Peter Steinberger authored the Tlon Urbit request-helper refactors and the current PR commits that preserved diagnostics and tightened proof. (role: feature-history owner and recent PR maintainer; confidence: high; commits: d0f64c955e03, de103773c79d, 2da7772da4b2; files: extensions/tlon/src/urbit/channel-ops.ts, extensions/tlon/src/urbit/fetch.ts, extensions/tlon/src/urbit/error-body-boundary.test.ts)
Alix-007: Alix-007 introduced the shared readProviderJsonResponse helper pattern that this PR applies to the Tlon scry path. (role: shared bounded-reader contributor; confidence: medium; commits: 2592f8a51a4e; files: src/agents/provider-http-errors.ts)
vincentkoc: Vincent Koc recently touched the same Tlon runtime seam in the current history, making them relevant for adjacent Tlon runtime context. (role: recent adjacent contributor; confidence: medium; commits: 0e54440ecc39; files: extensions/tlon/src/urbit/channel-ops.ts)

What the crustacean ranks mean

🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works

ClawSweeper keeps one durable marker-backed review comment per issue or PR.
Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
Maintainers can also comment @clawsweeper review to request a fresh review only.
Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

Review history (8 earlier review cycles)

reviewed 2026-07-05T16:24:14.677Z sha 0c3a7fc :: needs real behavior proof before merge. :: [P2] Preserve the failing scry path in JSON errors
reviewed 2026-07-05T16:30:19.405Z sha 0c3a7fc :: needs real behavior proof before merge. :: [P2] Preserve the scry path in JSON errors
reviewed 2026-07-06T00:15:15.258Z sha b745eef :: needs real behavior proof before merge. :: none
reviewed 2026-07-06T00:21:54.888Z sha b745eef :: needs maintainer review before merge. :: none
reviewed 2026-07-06T00:42:05.993Z sha 46ea2b0 :: needs changes before merge. :: [P3] Remove the release-owned changelog entry
reviewed 2026-07-06T01:17:31.137Z sha 343355a :: needs changes before merge. :: [P3] Remove the release-owned changelog additions
reviewed 2026-07-06T01:42:59.587Z sha 7db10da :: needs maintainer review before merge. :: none
reviewed 2026-07-06T02:06:08.129Z sha 110e0da :: needs maintainer review before merge. :: none

steipete · 2026-07-06T00:13:28Z

Land-ready maintainer proof for exact head 14571c2482309d47decdf1458d2d7ce39e56ca4f:

Confirmed every successful Tlon scry currently reaches unbounded Response.json(). The shared readProviderJsonResponse is the best owner-boundary fix because it already owns byte accounting, early stream cancellation, decoding, parsing, and stable bounded errors.
Fixed the submitted diagnostic regression: malformed and oversized errors retain the exact scry path. The regression stream is pull-driven and verifies early cancellation rather than pre-buffering its full fixture.
Added real loopback HTTP coverage through urbitFetch for a normal scry and a streaming oversized success response, alongside focused malformed/overflow/release tests.
Fresh structured AutoReview on the clean final diff: clean, no accepted/actionable findings.
Exact-head hosted CI: https://github.com/openclaw/openclaw/actions/runs/28761814647

Maintainer decision: accept the repository's shared 16 MiB successful-JSON ceiling for this operator-configured endpoint. Known gap: no production Urbit credentials were available; loopback proof exercises the real Tlon HTTP request/read path. Thanks @hugenshen for the focused fix.

steipete · 2026-07-06T00:49:34Z

Maintainer closeout at exact head 110e0daea1ad6abfbaa9a77a22f56f704a537b42:

Replaced Tlon's eager unbounded success-body buffering with the shared provider JSON reader, capped at 16 MiB with response cancellation on overflow.
Kept Urbit protocol handling owner-local; no generic core policy or provider-specific fallback was added.
Sanitized AWS Crabbox cbx_8ae94556bc3b: public network, no instance role, no Tailscale, no hydration.
Focused run run_3863cc067902: 5/5 bounded-response tests passed on the behavior-equivalent head.
Source-blind loopback HTTP proof: before, an 18,874,368-byte success response was fully buffered and parsed; after, the request failed at the 16 MiB limit and canceled the response after 17,235,968 bytes were sent.
Exact-head hosted CI run 28763080955: all required checks passed after attempt 2 cleared an unrelated TUI PTY timing flake.
The direct rebased head descends current main and contains only the three Tlon files; tree hash 260853298300c82388679cbeef32a104081fffda.
Fresh post-sync autoreview: clean, no actionable findings, confidence 0.97.
Official Urbit external API and HTTP API documentation reviewed; neither defines a bounded success-response body.
Contributor-credit changelog entries for fix(agents): preserve streamed assistant text when Claude CLI result event is empty #90450 (@totobusnello), fix(channels): normalize phone identities with stray plus signs #100467 (@morluto), and this fix (@hugenshen) follow in one final maintainer changelog PR.

Duplicate convergence: #97558 contains a broader overlapping Tlon slice. After this focused fix lands, that slice should be removed while its unrelated work remains independently reviewable.

steipete · 2026-07-06T02:25:34Z

Landed as 19f5a4a0bb030f9fb482bf37fedfd3b29d9b0f79.

The final merge contains only the three Tlon files: the shared 16 MiB bounded JSON reader now owns successful scry response reads, oversized streams are cancelled, path-specific diagnostics remain, and guarded-fetch resources still release in finally.

Proof: sanitized AWS Crabbox run run_3863cc067902 passed 5/5 focused tests; source-blind loopback proved the 18 MiB before/after boundary and cancellation; exact-head CI run 28763080955 passed on attempt 2 after an unrelated TUI PTY timing flake; fresh autoreview was clean at 0.97 confidence. Thanks @hugenshen for the original fix.

* fix(tlon): bound urbit scry JSON response reads * fix(tlon): preserve bounded scry diagnostics * refactor(tlon): tighten scry boundary proof * docs(changelog): stabilize Tlon response entry * docs(changelog): close out maintainer batch * chore(changelog): defer batch closeout --------- Co-authored-by: NIO <[email protected]> Co-authored-by: Peter Steinberger <[email protected]>

openclaw-barnacle Bot added channel: tlon Channel integration: tlon size: S labels Jul 5, 2026

steipete self-assigned this Jul 5, 2026

steipete force-pushed the fix/bound-tlon-scry-json branch from 0c3a7fc to b745eef Compare July 5, 2026 23:56

steipete force-pushed the fix/bound-tlon-scry-json branch 2 times, most recently from 3ccce22 to 46ea2b0 Compare July 6, 2026 00:33

openclaw-barnacle Bot added scripts Repository scripts size: M gateway Gateway runtime and removed size: S labels Jul 6, 2026

steipete force-pushed the fix/bound-tlon-scry-json branch from 7b51402 to 1de4034 Compare July 6, 2026 00:59

openclaw-barnacle Bot added size: S and removed gateway Gateway runtime scripts Repository scripts size: M labels Jul 6, 2026

steipete force-pushed the fix/bound-tlon-scry-json branch from 1de4034 to 343355a Compare July 6, 2026 01:06

clawsweeper Bot removed proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. labels Jul 6, 2026

steipete mentioned this pull request Jul 6, 2026

fix(channels): normalize phone identities with stray plus signs #100467

Merged

3 tasks

steipete force-pushed the fix/bound-tlon-scry-json branch 3 times, most recently from 0a97072 to 7db10da Compare July 6, 2026 01:34

openclaw-barnacle Bot added channel: telegram Channel integration: telegram app: ios App: ios scripts Repository scripts agents Agent runtime and tooling size: XL and removed size: S labels Jul 6, 2026

NIO and others added 6 commits July 6, 2026 03:00

fix(tlon): bound urbit scry JSON response reads

7733b05

fix(tlon): preserve bounded scry diagnostics

2da7772

refactor(tlon): tighten scry boundary proof

9d94ffa

docs(changelog): stabilize Tlon response entry

7109771

docs(changelog): close out maintainer batch

9a5108f

chore(changelog): defer batch closeout

110e0da

steipete force-pushed the fix/bound-tlon-scry-json branch from 3b4a52b to 110e0da Compare July 6, 2026 02:01

openclaw-barnacle Bot added size: S and removed channel: telegram Channel integration: telegram app: ios App: ios scripts Repository scripts agents Agent runtime and tooling size: XL labels Jul 6, 2026

steipete merged commit 19f5a4a into openclaw:main Jul 6, 2026
160 of 167 checks passed

This was referenced Jul 6, 2026

fix(extensions): bound JSON response reads to prevent OOM #97558

Closed

docs(changelog): credit recent contributor fixes #100562

Merged

github-actions Bot mentioned this pull request Jul 6, 2026

📡 Upstream Digest — 2026-07-06 02:26 UTC curtismercier/openclaw-mods#1220

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

fix(tlon): bound urbit scry JSON response reads#100376

fix(tlon): bound urbit scry JSON response reads#100376
steipete merged 6 commits into
openclaw:mainfrom
hugenshen:fix/bound-tlon-scry-json

hugenshen commented Jul 5, 2026 •

edited by steipete

Loading

Uh oh!

clawsweeper Bot commented Jul 5, 2026 •

edited

Loading

Uh oh!

steipete commented Jul 6, 2026 •

edited

Loading

Uh oh!

steipete commented Jul 6, 2026 •

edited

Loading

Uh oh!

Uh oh!

steipete commented Jul 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

hugenshen commented Jul 5, 2026 • edited by steipete Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Problem This Solves

Why This Change Was Made

User Impact

Evidence

Risk checklist

Related context

AI-assisted disclosure

Uh oh!

clawsweeper Bot commented Jul 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

steipete commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

steipete commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

steipete commented Jul 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

hugenshen commented Jul 5, 2026 •

edited by steipete

Loading

clawsweeper Bot commented Jul 5, 2026 •

edited

Loading

steipete commented Jul 6, 2026 •

edited

Loading

steipete commented Jul 6, 2026 •

edited

Loading