Skip to content

Commit a411be2

Browse files
anandgupta42claude
andauthored
fix: [AltimateAI#838] webfetch 404 cache — longer TTL + URL normalization (AltimateAI#839)
* fix: [AltimateAI#838] webfetch 404 cache — extend TTL + normalize URLs across tracking-param variations Telemetry-2026-05-21 showed 486 residual webfetch 404 errors (down 78% from March's 2,222 — the existing failure cache is doing its job). The remaining cases mostly escape because of two gaps: 1. TTL was 5 minutes. Real sessions often run 30+ minutes; the same dead URL re-hit the network after the first 5-minute window. 2. Cache key was the raw URL string. LLM-generated URLs frequently include tracking params (utm_source, fbclid, gclid, etc.) or fragments — each variation got its own cache entry, so a known-bad URL with different tracking params re-hit the network on every call. Changes - Extend FAILURE_CACHE_TTL from 5 minutes to 30 minutes. URLs that 404 rarely self-heal within a session; the longer window catches the slow-burn case without measurably increasing the risk of caching a transient failure. - Add `normalizeUrlForCache()` and use it for every cache read/write. Strips universally-tracking params (utm_*, ref*, fbclid, gclid, msclkid, yclid, twclid, li_fat_id, mc_cid, mc_eid, _ga, _gl, igshid, mibextid, __cf_chl_*), strips URL fragments, lowercases the hostname, and sorts remaining params. Conservative list — only params known to be tracking-only are stripped, so functional params (page, id, q, etc.) survive untouched. - Export `_resetFailureCache` and `_failureCacheSize` for tests. Tests New test/tool/webfetch-cache.test.ts (15 tests): - strips utm_*, fbclid, gclid, msclkid family - preserves functional params (page, id, q, etc.) - sorts params for stable cache key - strips fragments - lowercases hostname (case-insensitive per RFC 3986) - preserves path case (paths ARE case-sensitive per HTTP) - collapses Cloudflare __cf_chl_* challenge tokens - 6 variations of the same logical URL collapse to one cache key - different functional params and paths do NOT collapse - invalid URL strings pass through unchanged Validation - `tsgo --noEmit`: pass. - `bun test test/tool/webfetch-cache.test.ts test/tool/webfetch.test.ts`: 19 pass / 0 fail (15 new + 4 existing). - Marker check: pass — altimate_change markers applied. Expected telemetry impact - Further reduction in residual webfetch 404 errors (currently 486 in 14d). Specifically helps long sessions revisiting dead doc URLs multiple times, and LLMs pasting the same URL with different tracking suffixes. - No change for first-time fetches of legitimately new URLs. Closes AltimateAI#838 Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * fix: [AltimateAI#838] address consensus review — preserve `ref` param, value-aware sort, LRU re-touch Second-pass on PR AltimateAI#839 after code review. Three follow-ups in scope. 1. Preserve `ref` / `ref_src` / `ref_url` (MAJOR per reviewer) `ref` is functional on GitHub raw URLs and git-hosting APIs — `?ref=main` vs `?ref=v2.0` selects a different branch/tag and resolves to different content. Stripping it would suppress legitimate fetches when the agent (which fetches GitHub heavily) tries different refs of the same path. Removed `ref`, `ref_src`, `ref_url` from TRACKING_PARAMS. Kept `referrer` (the full word, tracking-only). Added doc-block at the TRACKING_PARAMS declaration explaining the exclusion so the next reviewer doesn't re-add them. 2. Value-aware param sort (MINOR) Duplicate-key params like `?a=1&a=2` and `?a=2&a=1` produced different cache keys despite carrying the same logical content — sorting by key alone leaked insertion order between same-key entries. Changed the comparator to sort by (key, value) tuple. 3. LRU re-touch semantics (MINOR) `Map.set` on an existing key preserves the original insertion position, so a frequently-failing URL stayed at the head of the FIFO queue and was the first to be evicted under pressure. Wrong for an LRU-flavoured cache. Delete-then-set on the re-touch path so the entry moves to the tail. 4. TTL doc (MINOR) Added a comment explaining the deliberate choice to use one TTL (30 min) for all three of 404, 410, 451 rather than split. The 3-status grouping is a simplification, not equivalence. 5. `@internal` JSDoc tags (NIT) Marked `_resetFailureCache` and `_failureCacheSize` as `@internal` so future tree-shakers / IDE hints discourage their use from production code. The underscore convention was the only guard. 6. Dropped explicit `__cf_chl_*` entries (NIT) Cloudflare challenge tokens are caught by the `isTrackingParamPrefix` check; the two enumerated entries (`__cf_chl_tk`, `__cf_chl_jschl_tk__`) were redundant. Kept utm_* in both the explicit list and the prefix check — the duplication documents the known cases (typed) while the prefix catches future variants. Tests - 3 new tests pin the new contracts: - `ref` param is preserved across normalization, AND different `ref` values do NOT collapse (regression guard against re-adding `ref` to TRACKING_PARAMS). - `referrer` (full word) is still stripped to confirm the distinction is real. - Duplicate-key params produce the same cache key regardless of input order (`?a=1&a=2` ↔ `?a=2&a=1`). - Updated existing test "preserves functional params when mixed with tracking params" to use a different tracking param (since `ref` is no longer one). Validation - `tsgo --noEmit`: pass. - `bun test test/tool/webfetch-cache.test.ts test/tool/webfetch.test.ts`: 22 pass / 0 fail (3 new tests). - Marker check: pass. Reviewer note - DeepSeek second opinion was unavailable (kilo CLI timeout, no output within window). Single-reviewer Claude analysis. Closes review feedback for PR AltimateAI#839. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * fix: [AltimateAI#838] address re-review — strip userinfo + add HubSpot/Marketo/Adobe/Piwik tracking params Reviewer flagged two cache-key fragmentation cases not covered by the first pass: 1. Basic-auth userinfo (user:pass@host) was left in the cache key, letting the same logical URL with and without credentials occupy two slots. Also a hygiene win — telemetry never logs cache keys with creds. 2. The tracking-param set covered Google/Meta/Twitter ad ecosystems but missed three major B2B martech vendors that dominate the email-newsletter URLs LLMs love to retrieve: - HubSpot: _hsenc, _hsmi, hsCtaTracking - Marketo: mkt_tok - Adobe Analytics: s_cid, s_kwcid - Piwik/Matomo: pk_campaign, pk_kwd, pk_source, pk_medium, pk_content, piwik_campaign, piwik_keyword Deliberately omitted Adobe Target's at_* prefix — too risky for false positives (e.g., at_preview_* are functional). The named s_cid / s_kwcid params catch the high-volume cases. 5 new tests pin the contract; existing 19 still pass. Closes AltimateAI#838 Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * fix: [AltimateAI#838] address fourth-pass re-review — split 451 TTL + cover LRU re-touch + fix stale doc Reviewer surfaced four issues. Three actionable: 1. STALE HEADER DOC — header comment listed `ref` as a stripped param, but the exclusion block immediately below explains we deliberately keep it. Future readers would hit cognitive whiplash. Removed `ref` from the inline list; switched the cache description to "process- level" (more accurate — Map is module-scope, persists across logical sessions in daemon/MCP mode). 2. SPLIT 451 TTL (webfetch.ts) — 404/410 are resource-permanent (30 min stays right). 451 is observer-conditional: an agent that swaps networks (VPN, travel, hotspot) inside a long-cache window would be locked out of URLs that are reachable from the new vantage point. Added FAILURE_CACHE_TTL_451 = 5 min and ttlForStatus() helper. 3. LRU RE-TOUCH WAS UNTESTED — the most subtle change in the original PR (delete-then-set so frequently-failing URLs don't sit at the head of the FIFO) had zero test coverage. Exposed three test-only helpers (_cacheUrlFailureForTest, _failureCacheKeysInOrder, _isUrlCachedFailureForTest) and added 6 cache state-machine tests: - cacheUrlFailure → isUrlCachedFailure round-trip - only 404/410/451 are cached (other statuses ignored) - tracking-param variations all hit the same cache key - re-touch moves entry to tail (pins the LRU contract) - 451 + 410 entries coexist (TTL split is wired) - _resetFailureCache actually clears cached entries 29/29 tests pass; marker check clean. Closes AltimateAI#838 Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * fix: [AltimateAI#838] address coderabbit nit — Matomo test name + pin "exact-name, not prefix" contract Reviewer flagged that the test title "strips Piwik/Matomo tracking params (pk_*, piwik_*)" overstates the contract — TRACKING_PARAMS does exact-name matching, NOT pk_* / piwik_* prefix stripping. The title was documenting behavior that isn't guaranteed. Renamed to enumerate the named params we DO strip and added two counter-tests verifying that pk_unknown / piwik_unknown ARE preserved. This pins the "exact-name, not prefix" contract so a future refactor can't silently flip to prefix-stripping without breaking these. Restraint rationale documented inline: prefix-stripping `pk_` risks false positives on functional params from unrelated services that happen to start with the same two letters. 29/29 tests pass; marker check clean. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> --------- Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
1 parent fbc7f97 commit a411be2

2 files changed

Lines changed: 527 additions & 7 deletions

File tree

‎packages/opencode/src/tool/webfetch.ts‎

Lines changed: 190 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -15,16 +15,143 @@ const BROWSER_UA =
1515
// Status codes that warrant a retry with a different User-Agent
1616
const RETRYABLE_STATUSES = new Set([403, 406])
1717

18-
// altimate_change start — session-level URL failure cache (#471)
19-
// Prevents repeated fetches to URLs that already returned 404/410 in this session.
18+
// altimate_change start — process-level URL failure cache (#471)
19+
// Prevents repeated fetches to URLs that already returned 404/410 in this
20+
// process. Cache is module-scope so in daemon mode (`altimate serve`) and
21+
// long-running MCP servers it persists across logical sessions — fine for
22+
// 404/410 (resource-permanent) but worth flagging for 451 (geo/legal block,
23+
// observer-conditional). See cacheUrlFailure comment for TTL rationale.
24+
//
25+
// 2026-05-22 follow-up to telemetry-2026-05-21 (486 residual webfetch 404s,
26+
// down from March's 2,222 but still meaningful):
27+
// - Extended TTL from 5 min to 30 min. URLs that 404 rarely self-heal
28+
// within a session, and many sessions run longer than 5 min so the
29+
// short window was letting agents re-hit the same dead URL multiple
30+
// times in one session.
31+
// - Cache key is the URL with tracking-only params (utm_*, fbclid,
32+
// gclid, mc_*, _ga, _gl, igshid, mibextid, __cf_chl_*, plus HubSpot/
33+
// Marketo/Adobe/Matomo variants) stripped, so LLM-generated tracking
34+
// variations of a known-bad URL all hit cache. Functional query params
35+
// (page, id, q, ref, etc.) are preserved — see exclusions block below
36+
// for the rationale on `ref` specifically.
2037
const failedUrls = new Map<string, { status: number; timestamp: number }>()
21-
const FAILURE_CACHE_TTL = 5 * 60 * 1000 // 5 minutes
38+
const FAILURE_CACHE_TTL = 30 * 60 * 1000 // 30 minutes — 404/410 (resource-permanent)
39+
const FAILURE_CACHE_TTL_451 = 5 * 60 * 1000 // 5 minutes — 451 is observer-conditional (VPN swap, travel)
40+
41+
/**
42+
* Param names that are universally tracking-only — stripping them never
43+
* changes the document the URL points to. Conservative list; favoured over
44+
* a regex sweep because false positives (e.g. stripping `?page=2`) would
45+
* cause real fetches to incorrectly hit the cache.
46+
*/
47+
// Notable exclusions (do NOT add these to TRACKING_PARAMS):
48+
// - `ref` / `ref_src` / `ref_url`: functional on GitHub raw URLs and
49+
// git-hosting APIs (`?ref=main` vs `?ref=v2.0` selects a different
50+
// branch/tag). Stripping `ref` would suppress legitimate fetches of
51+
// different refs against the same path — a coding agent fetches GitHub
52+
// heavily, so this is a real false-positive risk.
53+
// - `referrer` (full word) is tracking-only and IS in the list; the
54+
// abbreviated `ref` is not.
55+
//
56+
// The utm_* family is enumerated explicitly here AND covered by the
57+
// `isTrackingParamPrefix` check below. The duplication is intentional —
58+
// the explicit list documents the named params we know about, the prefix
59+
// check catches any future utm_* variant we haven't enumerated.
60+
// Cloudflare challenge tokens (`__cf_chl_*`) are handled by the prefix
61+
// check exclusively; no explicit entries needed.
62+
const TRACKING_PARAMS = new Set([
63+
"utm_source",
64+
"utm_medium",
65+
"utm_campaign",
66+
"utm_term",
67+
"utm_content",
68+
"utm_id",
69+
"referrer",
70+
"fbclid",
71+
"gclid",
72+
"dclid",
73+
"msclkid",
74+
"yclid",
75+
"twclid",
76+
"li_fat_id",
77+
"mc_cid",
78+
"mc_eid",
79+
"_ga",
80+
"_gl",
81+
"igshid",
82+
"mibextid",
83+
"vero_conv",
84+
"vero_id",
85+
// HubSpot
86+
"_hsenc",
87+
"_hsmi",
88+
"hsCtaTracking",
89+
// Marketo
90+
"mkt_tok",
91+
// Adobe Analytics / SiteCatalyst
92+
"s_cid",
93+
"s_kwcid",
94+
// Piwik / Matomo
95+
"pk_campaign",
96+
"pk_kwd",
97+
"pk_source",
98+
"pk_medium",
99+
"pk_content",
100+
"piwik_campaign",
101+
"piwik_keyword",
102+
])
103+
104+
function isTrackingParamPrefix(name: string): boolean {
105+
return name.startsWith("__cf_chl_") || name.startsWith("utm_")
106+
}
107+
108+
/**
109+
* Normalize a URL for cache lookup. Strips tracking-only query params, sorts
110+
* the remaining params for stable ordering, and lowercases the host. Returns
111+
* the input unchanged if URL parsing fails.
112+
*/
113+
export function normalizeUrlForCache(url: string): string {
114+
try {
115+
const parsed = new URL(url)
116+
parsed.hostname = parsed.hostname.toLowerCase()
117+
// Strip userinfo (basic-auth `user:pass@`) — it doesn't change the
118+
// resource the URL points to, and leaving it in lets the same logical
119+
// URL with/without credentials occupy two cache slots. Also a minor
120+
// hygiene win for cache keys that get logged in telemetry.
121+
parsed.username = ""
122+
parsed.password = ""
123+
const kept: [string, string][] = []
124+
for (const [k, v] of parsed.searchParams) {
125+
if (TRACKING_PARAMS.has(k) || isTrackingParamPrefix(k)) continue
126+
kept.push([k, v])
127+
}
128+
// Sort by (key, value) so duplicate-key params produce a stable order:
129+
// `?a=1&a=2` and `?a=2&a=1` normalize to the same string. Sorting by key
130+
// alone leaked the original insertion order between same-key entries.
131+
kept.sort(([ak, av], [bk, bv]) => {
132+
if (ak !== bk) return ak < bk ? -1 : 1
133+
return av < bv ? -1 : av > bv ? 1 : 0
134+
})
135+
parsed.search = ""
136+
for (const [k, v] of kept) parsed.searchParams.append(k, v)
137+
// Also strip fragment — fragments don't change the fetched resource.
138+
parsed.hash = ""
139+
return parsed.toString()
140+
} catch {
141+
return url
142+
}
143+
}
144+
145+
function ttlForStatus(status: number): number {
146+
return status === 451 ? FAILURE_CACHE_TTL_451 : FAILURE_CACHE_TTL
147+
}
22148

23149
function isUrlCachedFailure(url: string): { status: number } | null {
24-
const entry = failedUrls.get(url)
150+
const key = normalizeUrlForCache(url)
151+
const entry = failedUrls.get(key)
25152
if (!entry) return null
26-
if (Date.now() - entry.timestamp > FAILURE_CACHE_TTL) {
27-
failedUrls.delete(url)
153+
if (Date.now() - entry.timestamp > ttlForStatus(entry.status)) {
154+
failedUrls.delete(key)
28155
return null
29156
}
30157
return { status: entry.status }
@@ -33,16 +160,72 @@ function isUrlCachedFailure(url: string): { status: number } | null {
33160
const MAX_CACHED_URLS = 500
34161

35162
function cacheUrlFailure(url: string, status: number): void {
163+
// TTL by status:
164+
// 404 (resource not found) + 410 (gone) — 30 min. Both are
165+
// resource-permanent; 410 is spec-permanent, 404 is overwhelmingly
166+
// permanent in practice (typos, removed docs).
167+
// 451 (unavailable for legal reasons) — 5 min. Observer-conditional:
168+
// if the agent swaps networks (VPN, travel, mobile hotspot) the URL
169+
// may become reachable from the new vantage point, so a long TTL is
170+
// wrong. 5 min keeps the immediate-retry suppression but lets a
171+
// network change unblock the URL.
36172
if (status === 404 || status === 410 || status === 451) {
173+
const key = normalizeUrlForCache(url)
174+
// Re-touch semantics: if the key is already in the cache, delete first
175+
// so the .set() reinserts at the tail of the FIFO. Otherwise a
176+
// frequently-failing URL would stay at the head and be the first
177+
// evicted under pressure — wrong order for an LRU-flavoured cache.
178+
if (failedUrls.has(key)) failedUrls.delete(key)
37179
if (failedUrls.size >= MAX_CACHED_URLS) {
38180
// Evict oldest entry (Map preserves insertion order)
39181
const oldest = failedUrls.keys().next().value
40182
if (oldest) failedUrls.delete(oldest)
41183
}
42-
failedUrls.set(url, { status, timestamp: Date.now() })
184+
failedUrls.set(key, { status, timestamp: Date.now() })
43185
}
44186
}
45187

188+
/**
189+
* Reset the failure cache.
190+
* @internal — only used by tests; do not call from production code.
191+
*/
192+
export function _resetFailureCache(): void {
193+
failedUrls.clear()
194+
}
195+
196+
/**
197+
* Read the current failure cache size.
198+
* @internal — only used by tests; do not call from production code.
199+
*/
200+
export function _failureCacheSize(): number {
201+
return failedUrls.size
202+
}
203+
204+
/**
205+
* Drive the cache state machine from tests — same code path as production.
206+
* @internal — only used by tests; do not call from production code.
207+
*/
208+
export function _cacheUrlFailureForTest(url: string, status: number): void {
209+
cacheUrlFailure(url, status)
210+
}
211+
212+
/**
213+
* Read cache key insertion order — the order entries would be evicted under
214+
* FIFO/LRU pressure. Returns normalized cache keys, not raw URLs.
215+
* @internal — only used by tests; do not call from production code.
216+
*/
217+
export function _failureCacheKeysInOrder(): string[] {
218+
return [...failedUrls.keys()]
219+
}
220+
221+
/**
222+
* Probe the cache for a URL — same code path as production lookups.
223+
* @internal — only used by tests; do not call from production code.
224+
*/
225+
export function _isUrlCachedFailureForTest(url: string): { status: number } | null {
226+
return isUrlCachedFailure(url)
227+
}
228+
46229
/** Strip query string from URL to avoid leaking auth tokens in error messages. */
47230
function sanitizeUrl(url: string): string {
48231
try {

0 commit comments

Comments
 (0)