[Feature/Bug]: web_fetch does not work behind corporate HTTP proxy (DNS NXDOMAIN); needs opt-in env-proxy mode + optional allowlist for internal URLs #47598
Description
Bug type
Behavior bug (incorrect output/state without crash)
Summary
In a corporate network where direct DNS resolution fails (NXDOMAIN) but outbound HTTPS works via an HTTP proxy (CONNECT), OpenClaw’s web_fetch tool fails with:
getaddrinfo ENOTFOUND <host>
Even whenHTTP_PROXY/HTTPS_PROXY(and lowercase variants) are set and confirmed working with curl.
This appears to be because web_fetch uses a strict SSRF-safe fetch mode with DNS pinning and does not route arbitrary URLs through env proxy. As a result, web_fetch is unusable in corporate environments that require proxy egress and/or block external DNS.
- public web pages (via proxy)
- selected internal URLs (via proxy), with a safe/explicit allowlist
Steps to reproduce
- Set proxy env vars (example)
HTTP_PROXY=http://127.0.0.1:3128/HTTPS_PROXY=http://127.0.0.1:3128/(optionally NO_PROXY=...)
- Confirm direct DNS fails but proxy HTTP works:
PS> nslookup molty.me
Server: <corp-dns>
Address: <corp-dns-ip>
*** molty.me was not found: Non-existent domain.
PS> curl -i molty.me
HTTP/1.0 308 Permanent Redirect
Location: https://molty.me/
- Ask an agent to use
web_fetch:
Expected behavior
- When
HTTP_PROXY/HTTPS_PROXYis set (and proxy supports CONNECT),web_fetchshould be able to fetch public URLs through the proxy (or at least provide an explicit opt-in setting to do so). - For internal URLs: allow an explicit allowlist-based mode (operator-controlled) so
web_fetchcan access internal resources safely through the proxy.
Actual behavior
web_fetch fails with DNS resolution errors:web_fetch failed: getaddrinfo ENOTFOUND molty.meweb_fetch failed: getaddrinfo ENOTFOUND www.molty.me
This happens even though curl succeeds via proxy.
OpenClaw version
2026.3.13 (61d171a)
Operating system
Windows 11
Install method
npm install -g openclaw@latest
Model
openai/chatgpt-5.4
Provider / routing chain
openclaw -> px local proxy -> corporate proxy -> internet
Config file / key location
No response
Additional provider/model setup details
No response
Logs, screenshots, and evidence
Impact and severity
No response
Additional information
Additional context / why preload hacks don’t solve it
I can make Node’s global fetch() proxy-aware (e.g. by installing undici EnvHttpProxyAgent as the global dispatcher), and that helps other components.
However, web_fetch appears to supply its own guarded/pinned dispatcher for SSRF protection, which bypasses the global dispatcher and thus bypasses env proxy behavior.
Why this matters
Corporate environments commonly:
- block external DNS
- require all egress via HTTP proxy
Without proxy support, web_fetch is effectively unusable.
Proposed fix / design suggestion
Add an explicit, opt-in config option for web_fetch to use env proxy mode, e.g. one of:
tools.web.fetch.networkMode: "strict" | "trusted_env_proxy"(default"strict")
or
tools.web.fetch.useEnvProxy: true|false(defaultfalse)
Additionally, for internal URLs, provide an operator-controlled allowlist + explicit security switch, e.g.
tools.web.fetch.allowPrivateNetwork: false(default)tools.web.fetch.allowedHostnames: ["*.corp.example", "intranet.example"](optional)
so internal access is never enabled accidentally.
Security note: This should be opt-in with loud warnings because proxy mode breaks DNS pinning and changes the SSRF threat model.
Workarounds tried
- Firecrawl fallback is not viable due to cost.
- Running Gateway on a separate machine/network with normal DNS/egress works, but defeats the “run locally inside corp network” use case.
Related issues (similar proxy class)