Skip to content

Commit 66336bf

Browse files
steipetesteipete
committed
fix: add trusted env proxy opt-in for web fetch
1 parent bd2f856 commit 66336bf

16 files changed

Lines changed: 212 additions & 13 deletions

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,7 @@ Docs: https://docs.openclaw.ai
4040
- Config/messages: coerce boolean `messages.visibleReplies` and `messages.groupChat.visibleReplies` values to the documented enum modes so an intuitive toggle no longer invalidates config and drops channel startup. Fixes #75390. Thanks @scottgl9.
4141
- Agents/network: allow trusted web-search providers and configured model-provider hosts to work behind Surge/Clash/sing-box fake-IP DNS by accepting RFC 2544 and IPv6 ULA synthetic answers only for the request's scoped hostname, without broad private-network access. Refs #76530 and #76549. Thanks @zqchris.
4242
- Providers: honor env-proxy settings for guarded provider model fetches when no explicit dispatcher policy is configured, preserving explicit transport overrides. Fixes #70453. (#72480) Thanks @mjamiv.
43+
- Web fetch: add a default-off `tools.web.fetch.useTrustedEnvProxy` opt-in for proxy-only environments so `web_fetch` can let an operator-controlled HTTP(S) proxy resolve DNS while preserving default strict DNS pinning and hostname policy checks. Refs #58034 and #62560. Thanks @cosmicnet and @mjamiv.
4344
- Feishu: accept and honor `channels.feishu.blockStreaming` at the top level and per account, while keeping the legacy default off so Feishu cards no longer reject documented config or silently drop block replies. Fixes #75555. Thanks @vincentkoc.
4445
- Gateway/update: avoid `launchctl kickstart -k` immediately after fresh macOS update bootstraps, and unlink dangling global plugin-runtime symlinks during packaged postinstall and `doctor --fix` so upgrades no longer SIGTERM the newly booted Gateway or leave bundled plugin imports pointed at pruned `plugin-runtime-deps` trees. Completes #76261 and fixes #76466. (#76929)
4546
- Google Chat: normalize custom Google auth transport headers before google-auth/gaxios interceptors run, restoring webhook token verification when certificate retrieval expects Fetch `Headers`. Fixes #76742. Thanks @donbowman.

‎docs/.generated/config-baseline.sha256‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
b4cce06ca8c16774e277551ba027591289762ed9cf2490c993fec2051ac19c61 config-baseline.json
2-
bfb7ade43e58c630d0480eaa215ef22bf0d5030136c3e24cdd2c2a4c73d1b663 config-baseline.core.json
1+
056760c0a86627641d8e2993cc0cc987820dc4289c40c67dc8c2c1e8970c1849 config-baseline.json
2+
5b5ebd95939d75496597d9858a375e27544812d0f79dc3b4bf87c794ada2ba08 config-baseline.core.json
33
7b207901b595ad527026b1f357f63a5cd33123a72eeb66bdac24a8f2e8bb1ac8 config-baseline.channel.json
44
055fae0d0067a751dc10125af7421da45633f73519c94c982d02b0c4eb2bdf67 config-baseline.plugin.json

‎docs/tools/web-fetch.md‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -72,6 +72,7 @@ Truncate output to this many characters.
7272
timeoutSeconds: 30,
7373
cacheTtlMinutes: 15,
7474
maxRedirects: 3,
75+
useTrustedEnvProxy: false, // let a trusted HTTP(S) env proxy resolve DNS
7576
readability: true, // use Readability extraction
7677
userAgent: "Mozilla/5.0 ...", // override User-Agent
7778
ssrfPolicy: {
@@ -142,6 +143,22 @@ Current runtime behavior:
142143
- If Readability is disabled, `web_fetch` skips straight to the selected
143144
provider fallback. If no provider is available, it fails closed.
144145

146+
## Trusted Env Proxy
147+
148+
If your deployment requires `web_fetch` to go through a trusted outbound
149+
HTTP(S) proxy, set `tools.web.fetch.useTrustedEnvProxy: true`.
150+
151+
In this mode, OpenClaw still applies hostname-based SSRF checks before sending
152+
the request, but it lets the proxy resolve DNS instead of doing local DNS
153+
pinning. Enable this only when the proxy is operator-controlled and enforces
154+
outbound policy after DNS resolution.
155+
156+
<Note>
157+
If no HTTP(S) proxy env var is configured, or the target host is excluded by
158+
`NO_PROXY`, `web_fetch` falls back to the normal strict path with local DNS
159+
pinning.
160+
</Note>
161+
145162
## Limits and safety
146163

147164
- `maxChars` is clamped to `tools.web.fetch.maxCharsCap`
@@ -153,6 +170,9 @@ Current runtime behavior:
153170
for trusted fake-IP proxy stacks; leave them unset unless your proxy owns
154171
those synthetic ranges and enforces its own destination policy
155172
- Redirects are checked and limited by `maxRedirects`
173+
- `useTrustedEnvProxy` is an explicit opt-in and should only be enabled for
174+
operator-controlled proxies that still enforce outbound policy after DNS
175+
resolution
156176
- `web_fetch` is best-effort -- some sites need the [Web Browser](/tools/browser)
157177

158178
## Tool profiles

‎src/agents/tools/web-fetch.ssrf.test.ts‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,7 @@ function setMockFetch(
3636

3737
function createWebFetchToolForTest(params?: {
3838
firecrawlApiKey?: string;
39+
useTrustedEnvProxy?: boolean;
3940
ssrfPolicy?: { allowRfc2544BenchmarkRange?: boolean; allowIpv6UniqueLocalRange?: boolean };
4041
cacheTtlMinutes?: number;
4142
}) {
@@ -58,6 +59,7 @@ function createWebFetchToolForTest(params?: {
5859
web: {
5960
fetch: {
6061
cacheTtlMinutes: params?.cacheTtlMinutes ?? 0,
62+
useTrustedEnvProxy: params?.useTrustedEnvProxy,
6163
ssrfPolicy: params?.ssrfPolicy,
6264
...(params?.firecrawlApiKey ? { provider: "firecrawl" } : {}),
6365
},
@@ -89,6 +91,7 @@ describe("web_fetch SSRF protection", () => {
8991
global.fetch = priorFetch;
9092
lookupMock.mockClear();
9193
vi.restoreAllMocks();
94+
vi.unstubAllEnvs();
9295
});
9396

9497
it("blocks localhost hostnames before fetch/firecrawl", async () => {
@@ -202,4 +205,18 @@ describe("web_fetch SSRF protection", () => {
202205
const stricterTool = createWebFetchToolForTest({ cacheTtlMinutes: 1 });
203206
await expectBlockedUrl(stricterTool, url, /private|internal|blocked/i);
204207
});
208+
209+
it("still blocks dangerous hostnames when trusted env proxy is explicitly enabled", async () => {
210+
vi.stubEnv("HTTP_PROXY", "http://127.0.0.1:7890");
211+
vi.stubEnv("http_proxy", "http://127.0.0.1:7890");
212+
const fetchSpy = setMockFetch();
213+
const tool = createWebFetchToolForTest({
214+
useTrustedEnvProxy: true,
215+
cacheTtlMinutes: 1,
216+
});
217+
218+
await expectBlockedUrl(tool, "http://localhost/test", /Blocked hostname/i);
219+
expect(fetchSpy).not.toHaveBeenCalled();
220+
expect(lookupMock).not.toHaveBeenCalled();
221+
});
205222
});

‎src/agents/tools/web-fetch.ts‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -117,6 +117,10 @@ function resolveFetchReadabilityEnabled(fetch?: WebFetchConfig): boolean {
117117
return true;
118118
}
119119

120+
function resolveFetchUseTrustedEnvProxy(fetch?: WebFetchConfig): boolean {
121+
return fetch?.useTrustedEnvProxy === true;
122+
}
123+
120124
function resolveFetchMaxCharsCap(fetch?: WebFetchConfig): number {
121125
const raw =
122126
fetch && "maxCharsCap" in fetch && typeof fetch.maxCharsCap === "number"
@@ -273,6 +277,7 @@ type WebFetchRuntimeParams = {
273277
userAgent: string;
274278
readabilityEnabled: boolean;
275279
config?: OpenClawConfig;
280+
useTrustedEnvProxy: boolean;
276281
ssrfPolicy?: {
277282
allowRfc2544BenchmarkRange?: boolean;
278283
allowIpv6UniqueLocalRange?: boolean;
@@ -392,6 +397,7 @@ async function maybeFetchProviderWebFetchPayload(
392397
async function runWebFetch(params: WebFetchRuntimeParams): Promise<Record<string, unknown>> {
393398
const allowRfc2544BenchmarkRange = params.ssrfPolicy?.allowRfc2544BenchmarkRange === true;
394399
const allowIpv6UniqueLocalRange = params.ssrfPolicy?.allowIpv6UniqueLocalRange === true;
400+
const useTrustedEnvProxy = params.useTrustedEnvProxy;
395401
const ssrfPolicy: SsrFPolicy | undefined =
396402
allowRfc2544BenchmarkRange || allowIpv6UniqueLocalRange
397403
? {
@@ -400,7 +406,7 @@ async function runWebFetch(params: WebFetchRuntimeParams): Promise<Record<string
400406
}
401407
: undefined;
402408
const cacheKey = normalizeCacheKey(
403-
`fetch:${params.url}:${params.extractMode}:${params.maxChars}${allowRfc2544BenchmarkRange ? ":allow-rfc2544" : ""}${allowIpv6UniqueLocalRange ? ":allow-ipv6-ula" : ""}`,
409+
`fetch:${params.url}:${params.extractMode}:${params.maxChars}${allowRfc2544BenchmarkRange ? ":allow-rfc2544" : ""}${allowIpv6UniqueLocalRange ? ":allow-ipv6-ula" : ""}${useTrustedEnvProxy ? ":trusted-env-proxy" : ""}`,
404410
);
405411
const cached = readCache(FETCH_CACHE, cacheKey);
406412
if (cached) {
@@ -428,6 +434,7 @@ async function runWebFetch(params: WebFetchRuntimeParams): Promise<Record<string
428434
maxRedirects: params.maxRedirects,
429435
timeoutSeconds: params.timeoutSeconds,
430436
lookupFn: params.lookupFn,
437+
useEnvProxy: useTrustedEnvProxy,
431438
policy: ssrfPolicy,
432439
init: {
433440
headers: {
@@ -661,6 +668,7 @@ export function createWebFetchTool(options?: {
661668
userAgent,
662669
readabilityEnabled,
663670
config: options?.config,
671+
useTrustedEnvProxy: resolveFetchUseTrustedEnvProxy(fetch),
664672
ssrfPolicy: fetch?.ssrfPolicy,
665673
lookupFn: options?.lookupFn,
666674
resolveProviderFallback,

‎src/agents/tools/web-tools.fetch.test.ts‎

Lines changed: 26 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -331,7 +331,7 @@ describe("web_fetch extraction fallbacks", () => {
331331
expect(details?.warning).toContain("Response body truncated");
332332
});
333333

334-
it("keeps DNS pinning for untrusted web_fetch URLs even when HTTP_PROXY is configured", async () => {
334+
it("keeps DNS pinning for web_fetch by default even when HTTP_PROXY is configured", async () => {
335335
vi.stubEnv("HTTP_PROXY", "http://127.0.0.1:7890");
336336
const mockFetch = installMockFetch((input: RequestInfo | URL) =>
337337
Promise.resolve({
@@ -353,6 +353,31 @@ describe("web_fetch extraction fallbacks", () => {
353353
expect(requestInit?.dispatcher).not.toBeInstanceOf(EnvHttpProxyAgent);
354354
});
355355

356+
it("uses env proxy dispatch for web_fetch when trusted env proxy is explicitly enabled", async () => {
357+
vi.stubEnv("HTTP_PROXY", "http://127.0.0.1:7890");
358+
const mockFetch = installMockFetch((input: RequestInfo | URL) =>
359+
Promise.resolve({
360+
ok: true,
361+
status: 200,
362+
headers: makeFetchHeaders({ "content-type": "text/plain" }),
363+
text: async () => "proxy body",
364+
url: resolveRequestUrl(input),
365+
} as Response),
366+
);
367+
const tool = createFetchTool({
368+
firecrawl: { enabled: false },
369+
useTrustedEnvProxy: true,
370+
});
371+
372+
await tool?.execute?.("call", { url: "https://example.com/proxy" });
373+
374+
const requestInit = mockFetch.mock.calls[0]?.[1] as
375+
| (RequestInit & { dispatcher?: unknown })
376+
| undefined;
377+
expect(requestInit?.dispatcher).toBeDefined();
378+
expect(requestInit?.dispatcher).toBeInstanceOf(EnvHttpProxyAgent);
379+
});
380+
356381
// NOTE: Test for wrapping url/finalUrl/warning fields requires DNS mocking.
357382
// The sanitization of these fields is verified by external-content.test.ts tests.
358383

‎src/config/schema.base.generated.ts‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8759,6 +8759,12 @@ export const GENERATED_BASE_CONFIG_SCHEMA: BaseConfigSchemaResponse = {
87598759
description:
87608760
"Use Readability to extract main content from HTML (fallbacks to basic HTML cleanup).",
87618761
},
8762+
useTrustedEnvProxy: {
8763+
type: "boolean",
8764+
title: "Web Fetch Trusted Env Proxy",
8765+
description:
8766+
"Route web_fetch through a trusted HTTP(S) env proxy and let the proxy resolve DNS. Enable only when that proxy is operator-controlled and enforces outbound policy after DNS resolution.",
8767+
},
87628768
ssrfPolicy: {
87638769
type: "object",
87648770
properties: {
@@ -25987,6 +25993,11 @@ export const GENERATED_BASE_CONFIG_SCHEMA: BaseConfigSchemaResponse = {
2598725993
help: "Use Readability to extract main content from HTML (fallbacks to basic HTML cleanup).",
2598825994
tags: ["tools"],
2598925995
},
25996+
"tools.web.fetch.useTrustedEnvProxy": {
25997+
label: "Web Fetch Trusted Env Proxy",
25998+
help: "Route web_fetch through a trusted HTTP(S) env proxy and let the proxy resolve DNS. Enable only when that proxy is operator-controlled and enforces outbound policy after DNS resolution.",
25999+
tags: ["tools"],
26000+
},
2599026001
"tools.web.fetch.ssrfPolicy": {
2599126002
label: "Web Fetch SSRF Policy",
2599226003
help: "Scoped SSRF policy overrides for web_fetch. Keep this narrow and opt in only for known local-network proxy environments.",

‎src/config/schema.help.ts‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -834,6 +834,8 @@ export const FIELD_HELP: Record<string, string> = {
834834
"tools.web.fetch.userAgent": "Override User-Agent header for web_fetch requests.",
835835
"tools.web.fetch.readability":
836836
"Use Readability to extract main content from HTML (fallbacks to basic HTML cleanup).",
837+
"tools.web.fetch.useTrustedEnvProxy":
838+
"Route web_fetch through a trusted HTTP(S) env proxy and let the proxy resolve DNS. Enable only when that proxy is operator-controlled and enforces outbound policy after DNS resolution.",
837839
"tools.web.fetch.ssrfPolicy":
838840
"Scoped SSRF policy overrides for web_fetch. Keep this narrow and opt in only for known local-network proxy environments.",
839841
"tools.web.fetch.ssrfPolicy.allowRfc2544BenchmarkRange":

‎src/config/schema.labels.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -296,6 +296,7 @@ export const FIELD_LABELS: Record<string, string> = {
296296
"tools.web.fetch.maxRedirects": "Web Fetch Max Redirects",
297297
"tools.web.fetch.userAgent": "Web Fetch User-Agent",
298298
"tools.web.fetch.readability": "Web Fetch Readability Extraction",
299+
"tools.web.fetch.useTrustedEnvProxy": "Web Fetch Trusted Env Proxy",
299300
"tools.web.fetch.ssrfPolicy": "Web Fetch SSRF Policy",
300301
"tools.web.fetch.ssrfPolicy.allowRfc2544BenchmarkRange":
301302
"Web Fetch Allow RFC 2544 Benchmark Range",

‎src/config/schema.test.ts‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -390,6 +390,18 @@ describe("config schema", () => {
390390
});
391391
});
392392

393+
it("accepts web fetch trusted env proxy opt-in in the runtime zod schema", () => {
394+
const parsed = ToolsSchema.parse({
395+
web: {
396+
fetch: {
397+
useTrustedEnvProxy: true,
398+
},
399+
},
400+
});
401+
402+
expect(parsed?.web?.fetch?.useTrustedEnvProxy).toBe(true);
403+
});
404+
393405
it("rejects allowPrivateNetwork on media-understanding request config", () => {
394406
expect(() =>
395407
ToolsSchema.parse({

0 commit comments

Comments
 (0)