Skip to content

fix(web-fetch): respect NO_PROXY when useTrustedEnvProxy is enabled#94541

Closed
cxbAsDev wants to merge 2 commits into
openclaw:mainfrom
cxbAsDev:fix/web-fetch-no-proxy-env
Closed

fix(web-fetch): respect NO_PROXY when useTrustedEnvProxy is enabled#94541
cxbAsDev wants to merge 2 commits into
openclaw:mainfrom
cxbAsDev:fix/web-fetch-no-proxy-env

Conversation

@cxbAsDev

@cxbAsDev cxbAsDev commented Jun 18, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

When tools.web.fetch.useTrustedEnvProxy: true is set and NO_PROXY excludes a target host, web_fetch should bypass the proxy and connect directly. The proxy bypass logic in shouldUseEnvHttpProxyForUrl() already detects NO_PROXY matches, but after skipping proxy creation the request falls through to the strict SSRF pinned-DNS path which blocks private/internal IP addresses — making NO_PROXY exclusions ineffective for local services.

Fix: use exact-host trust via allowedHostnames for NO_PROXY-matched hosts. This allows private and local addresses (RFC1918, tailnet, localhost) to be reached directly without the proxy, while metadata and link-local addresses (169.254.x.x, cloud metadata IPs) remain blocked by the resolved-address guard in resolvePinnedHostnameWithPolicy. Does NOT set allowPrivateNetwork: true which would skip all SSRF checks.

Fixes #93807.

Changes

  • src/infra/net/fetch-guard.ts: exact-host trust via allowedHostnames for NO_PROXY hosts, preserving metadata/link-local guard
  • src/infra/net/fetch-guard.ssrf.test.ts: +2 metadata/link-local regression tests for NO_PROXY-matched hosts
  • docs/tools/web-fetch.md: updated NO_PROXY behavior description

Real behavior proof (required for external PRs)

Behavior addressed: web_fetch with useTrustedEnvProxy: true ignores NO_PROXY exclusions for private-network targets. After fix, NO_PROXY-matched hosts use exact-host trust (private IPs allowed, metadata/link-local still blocked).

Real environment tested: Linux, Node 24, OpenClaw PR branch 149e547

Exact steps or command run after fix:

node scripts/run-vitest.mjs run src/infra/net/fetch-guard.ssrf.test.ts

Evidence after fix:

Test Files  1 passed (1)
     Tests  90 passed (90)

Includes 2 new metadata/link-local regression tests:

  • blocks link-local resolved addresses for NO_PROXY-matched hosts (metadata guard) — DNS rebinding to 169.254.169.254 is blocked
  • blocks cloud metadata resolved addresses for NO_PROXY-matched hosts (metadata guard) — DNS rebinding to 100.100.100.200 is blocked

Private IPs (192.168.x.x) and localhost are allowed for NO_PROXY-matched hosts, proving exact-host trust works.

Observed result after fix: NO_PROXY-matched hosts can reach private networks directly. Metadata/link-local IPs remain blocked. The allowPrivateNetwork: true flag is NOT used — exact-host trust via allowedHostnames preserves the resolved-address guard.

What was not tested: A live web_fetch call against a real private NO_PROXY target was not tested — requires a production proxy setup.

Tests and validation

  • fetch-guard.ssrf.test.ts: 90/90 passed ✅ (incl. 2 metadata guard tests)
  • Docs: web-fetch.md updated

Risk checklist

Did user-visible behavior change? Yes — NO_PROXY hosts can now reach private IPs directly.
Did config, environment, or migration behavior change? No.
Did security, auth, secrets, network, or tool execution behavior change? Yes — SSRF security boundary: exact-host trust replaces full allowPrivateNetwork, preserving metadata/link-local blocking.

@cxbAsDev @claude
fix(web-fetch): respect NO_PROXY when useTrustedEnvProxy is enabled
304c68e 
When tools.web.fetch.useTrustedEnvProxy is true, NO_PROXY env var excludes
were previously ignored because the request fell through to the strict SSRF
pinned-DNS path which blocks private-network addresses.

Add a trustedNoProxyBypass gate that relaxes SSRF private-network checks
for NO_PROXY-matched hosts while keeping hostname allowlist and DNS pinning
active. The NO_PROXY host is added to allowedHostnames and
allowPrivateNetwork is set to true, scoped only to the matched target.

Closes openclaw#93807

Co-Authored-By: Claude <[email protected]>
@openclaw-barnacle openclaw-barnacle Bot added size: S triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. proof: supplied External PR includes structured after-fix real behavior proof. and removed triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels Jun 18, 2026
@clawsweeper

clawsweeper Bot commented Jun 18, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Codex review: needs real behavior proof before merge. Reviewed June 26, 2026, 11:29 PM ET / 03:29 UTC.

Summary
The branch adds exact-host trust for NO_PROXY-matched trusted env-proxy web_fetch requests, adds SSRF regression tests, and updates the web_fetch docs.

PR surface: Source +30, Tests +148, Docs +3. Total +181 across 3 files.

Reproducibility: yes. source-level: current main and v2026.6.10 skip EnvHttpProxyAgent when NO_PROXY matches, then use strict pinned-DNS SSRF checks that block private/local targets. I did not run a live proxy plus local-service reproduction in this read-only review.

Review metrics: 2 noteworthy metrics.

  • NO_PROXY trust path: 1 added. A previously strict fallback becomes exact-host private/local reachability in trusted env-proxy mode.
  • Live proof coverage: 5 mocked guard cases, 0 live web_fetch proofs. Security-sensitive network behavior needs user-path proof beyond unit guard tests before merge.

Stored data model
Persistent data-model change detected: vector/embedding metadata: docs/tools/web-fetch.md. Confirm migration or upgrade compatibility proof before merge.

Root-cause cluster
Relationship: fixed_by_candidate
Canonical: #93807
Summary: This PR is one candidate fix for the canonical web_fetch trusted-env-proxy NO_PROXY issue; sibling candidate PRs remain open and no canonical fix has merged.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🧂 unranked krab
Proof: 🧂 unranked krab
Patch quality: 🦐 gold shrimp
Result: blocked until real behavior proof from a real setup is added.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • [P1] Add redacted live web_fetch or gateway proof with HTTP(S)_PROXY plus a NO_PROXY-matched private/local target and a non-NO_PROXY external/proxied target.
  • Get maintainer security/product approval for making NO_PROXY grant exact-host private/local web_fetch access.
  • Pick this or another linked PR as the canonical implementation before landing.

Proof guidance:

  • [P1] Needs real behavior proof before merge: The PR body supplies unit-test output only and explicitly says no live private NO_PROXY web_fetch target was tested, so the contributor should add redacted live proof before merge and then update the PR body for re-review.

Risk before merge

  • [P1] Existing users with broad NO_PROXY entries could see previously blocked private/internal web_fetch destinations become reachable after upgrade.
  • [P1] The diff intentionally relaxes a web_fetch SSRF boundary; metadata/link-local unit coverage helps, but maintainer security/product approval is still needed.
  • [P1] The external PR proof is limited to unit/mocked guard output and explicitly says no live private NO_PROXY web_fetch target was tested.
  • [P1] Multiple open candidate PRs target web_fetch useTrustedEnvProxy ignores NO_PROXY env variable #93807, so maintainers still need to choose one canonical implementation direction.

Maintainer options:

  1. Require Live Proof And Security Approval (recommended)
    Before merge, require redacted live web_fetch or gateway proof for HTTP(S)_PROXY plus a NO_PROXY-matched private/local target and maintainer approval of the exact-host SSRF policy change.
  2. Preserve Strict Fallback
    Maintainers can decide that NO_PROXY should remain proxy-bypass-only and leave private/local targets on the strict pinned-DNS SSRF path.
  3. Accept Exact-Host Local Access
    Maintainers can intentionally accept the new direct-access behavior with the added metadata/link-local coverage, docs update, and one canonical PR chosen for the issue.

Next step before merge

  • [P1] Needs maintainer security/product judgment plus contributor live proof; automation cannot choose the SSRF policy or prove the contributor's network setup.

Security
Needs attention: The diff intentionally changes the web_fetch SSRF boundary and needs maintainer security approval before merge.

Review details

Best possible solution:

Land one canonical shared fetch-guard policy only after maintainer security approval and redacted live web_fetch proof, or preserve the shipped strict fallback if NO_PROXY should remain proxy-bypass-only.

Do we have a high-confidence way to reproduce the issue?

Yes, source-level: current main and v2026.6.10 skip EnvHttpProxyAgent when NO_PROXY matches, then use strict pinned-DNS SSRF checks that block private/local targets. I did not run a live proxy plus local-service reproduction in this read-only review.

Is this the best way to solve the issue?

Unclear until maintainers approve the security policy. If NO_PROXY local access is approved, the shared fetch-guard exact-host trust shape is cleaner than wrapper-only or full allowPrivateNetwork alternatives.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against cee2aca40912.

Label changes

Label justifications:

  • P2: This is a bounded web_fetch proxy/local-service behavior change with active candidate fixes but no evidence of a broad outage.
  • merge-risk: 🚨 compatibility: Existing broad NO_PROXY configurations could make previously blocked private/internal web_fetch destinations reachable after upgrade.
  • merge-risk: 🚨 security-boundary: The diff intentionally changes SSRF handling for NO_PROXY-matched trusted-env-proxy web_fetch targets.
  • rating: 🧂 unranked krab: Overall readiness is 🧂 unranked krab; proof is 🧂 unranked krab and patch quality is 🦐 gold shrimp.
  • status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs real behavior proof before merge: The PR body supplies unit-test output only and explicitly says no live private NO_PROXY web_fetch target was tested, so the contributor should add redacted live proof before merge and then update the PR body for re-review.
Evidence reviewed

PR surface:

Source +30, Tests +148, Docs +3. Total +181 across 3 files.

View PR surface stats
Area Files Added Removed Net
Source 1 40 10 +30
Tests 1 148 0 +148
Docs 1 6 3 +3
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 3 194 13 +181

Security concerns:

  • [medium] NO_PROXY expands private web_fetch reachability — src/infra/net/fetch-guard.ts:534
    Adding exact-host trust for NO_PROXY matches means broad existing NO_PROXY settings can make private/local targets directly reachable through web_fetch; metadata/link-local coverage helps, but the policy change still needs maintainer approval.
    Confidence: 0.9

What I checked:

  • Repository policy read: Root AGENTS.md and docs/AGENTS.md were read; the compatibility/security-boundary guidance affects this review because the PR changes proxy fallback and SSRF behavior, and docs guidance applies to docs/tools/web-fetch.md. (AGENTS.md:1, cee2aca40912)
  • PR-head policy change: At PR head, trustedNoProxyBypass detects NO_PROXY matches in TRUSTED_ENV_PROXY mode and adds the parsed hostname to allowedHostnames, which makes private/local direct access possible without setting allowPrivateNetwork. (src/infra/net/fetch-guard.ts:529, 149e5470c4cf)
  • PR-head regression coverage: The diff adds mocked guard tests for NO_PROXY-matched private IP and localhost access, non-NO_PROXY private blocking, and metadata/link-local blocking for NO_PROXY-matched hosts. (src/infra/net/fetch-guard.ssrf.test.ts:2115, 149e5470c4cf)
  • Current-main strict fallback: Current main only creates EnvHttpProxyAgent when shouldUseEnvHttpProxyForUrl is true; NO_PROXY matches therefore fall through to pinned DNS and the normal SSRF policy path. (src/infra/net/fetch-guard.ts:507, cee2aca40912)
  • Latest-release documented behavior: Latest release v2026.6.10 documents NO_PROXY targets as falling back to the normal strict path with local DNS pinning, so this PR changes shipped/user-visible behavior. Public docs: docs/tools/web-fetch.md. (docs/tools/web-fetch.md:174, aa69b12d0086)
  • Current-main NO_PROXY gate: shouldUseEnvHttpProxyForUrl already treats NO_PROXY matches as proxy bypass by returning false when matchesNoProxy is true; the unresolved behavior is whether that bypass should also grant private/local reachability. (src/infra/net/proxy-env.ts:122, cee2aca40912)

Likely related people:

  • steipete: Commit history links this account to the web_fetch trusted env-proxy opt-in and proxy/guard mode unification in the affected surface. (role: feature introducer and adjacent refactor author; confidence: high; commits: 66336bf7c846, c973b053a5e2, 53aa5232bc00; files: src/agents/tools/web-fetch.ts, src/agents/tools/web-guarded-fetch.ts, src/infra/net/fetch-guard.ts)
  • Kaspre: Recent SSRF exact-origin and metadata/link-local hardening commits directly affect whether exact-host NO_PROXY trust is safe. (role: security boundary contributor; confidence: high; commits: 44840007d42d, fd2a9adbe6b0; files: src/infra/net/ssrf.ts, src/infra/net/fetch-guard.ts, src/infra/net/fetch-guard.ssrf.test.ts)
  • vincentkoc: Recent history connects this account to proxy-env helpers, guarded fetch behavior, and the web_fetch docs surface touched by this PR. (role: recent area contributor; confidence: high; commits: e098eb735ff7, 6ee8e194c027, 21728777dfa0; files: src/infra/net/proxy-env.ts, src/infra/net/fetch-guard.ts, docs/tools/web-fetch.md)
  • cluster2600: The trusted env-proxy dispatch behavior that skips local DNS pinning before proxy dispatch traces to this author, with steipete as committer. (role: trusted proxy dispatch behavior author; confidence: medium; commits: d7c3210cd6f5; files: src/infra/net/fetch-guard.ts, src/infra/net/fetch-guard.ssrf.test.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@clawsweeper clawsweeper Bot added rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. P2 Normal backlog priority with limited blast radius. merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. merge-risk: 🚨 security-boundary 🚨 May affect sandboxing, authorization, credentials, or sensitive data. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. and removed rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. labels Jun 18, 2026
@cxbAsDev

cxbAsDev commented Jun 18, 2026

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Jun 18, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

This was referenced Jun 19, 2026
Open
Closed
Open
@cxbAsDev @claude
fix(web-fetch): use exact-host trust for NO_PROXY, preserving metadat…
149e547 
…a guard

Replace allowPrivateNetwork with exact-host trust via allowedHostnames
so NO_PROXY-matched private/local addresses (RFC1918, tailnet, localhost)
are reachable directly, but metadata and link-local addresses (169.254.x.x,
cloud metadata IPs) remain blocked by the resolved-address guard in
resolvePinnedHostnameWithPolicy.

- Remove allowPrivateNetwork: true from the NO_PROXY effective policy
- Add metadata/link-local regression tests for NO_PROXY-matched hosts
- Update web_fetch docs to describe exact-host trust behavior

Refs openclaw#94541, openclaw#93807

Co-Authored-By: Claude <[email protected]>
@openclaw-barnacle openclaw-barnacle Bot added the docs Improvements or additions to documentation label Jun 23, 2026
@clawsweeper clawsweeper Bot added rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. and removed rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. labels Jun 23, 2026
@nxmxbbd nxmxbbd mentioned this pull request Jun 23, 2026
Open
@clawsweeper clawsweeper Bot mentioned this pull request Jun 25, 2026
Open
@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. and removed rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. labels Jun 26, 2026
@cxbAsDev

cxbAsDev commented Jun 27, 2026

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Jun 27, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

@cxbAsDev cxbAsDev closed this Jun 30, 2026
@clawsweeper clawsweeper Bot mentioned this pull request Jul 6, 2026
Closed
@cxbAsDev
cxbAsDev deleted the fix/web-fetch-no-proxy-env branch July 15, 2026 03:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

docs Improvements or additions to documentation merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. merge-risk: 🚨 security-boundary 🚨 May affect sandboxing, authorization, credentials, or sensitive data. P2 Normal backlog priority with limited blast radius. proof: supplied External PR includes structured after-fix real behavior proof. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. size: S status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

web_fetch useTrustedEnvProxy ignores NO_PROXY env variable

1 participant

@cxbAsDev