Repository navigation
Commit 25be299
fix: [#838] webfetch 404 cache — extend TTL + normalize URLs across tracking-param variations
Telemetry-2026-05-21 showed 486 residual webfetch 404 errors (down 78%
from March's 2,222 — the existing failure cache is doing its job). The
remaining cases mostly escape because of two gaps:
1. TTL was 5 minutes. Real sessions often run 30+ minutes; the same
dead URL re-hit the network after the first 5-minute window.
2. Cache key was the raw URL string. LLM-generated URLs frequently
include tracking params (utm_source, fbclid, gclid, etc.) or
fragments — each variation got its own cache entry, so a known-bad
URL with different tracking params re-hit the network on every
call.
Changes
- Extend FAILURE_CACHE_TTL from 5 minutes to 30 minutes. URLs that
404 rarely self-heal within a session; the longer window catches
the slow-burn case without measurably increasing the risk of
caching a transient failure.
- Add `normalizeUrlForCache()` and use it for every cache read/write.
Strips universally-tracking params (utm_*, ref*, fbclid, gclid,
msclkid, yclid, twclid, li_fat_id, mc_cid, mc_eid, _ga, _gl,
igshid, mibextid, __cf_chl_*), strips URL fragments, lowercases
the hostname, and sorts remaining params. Conservative list — only
params known to be tracking-only are stripped, so functional
params (page, id, q, etc.) survive untouched.
- Export `_resetFailureCache` and `_failureCacheSize` for tests.
Tests
New test/tool/webfetch-cache.test.ts (15 tests):
- strips utm_*, fbclid, gclid, msclkid family
- preserves functional params (page, id, q, etc.)
- sorts params for stable cache key
- strips fragments
- lowercases hostname (case-insensitive per RFC 3986)
- preserves path case (paths ARE case-sensitive per HTTP)
- collapses Cloudflare __cf_chl_* challenge tokens
- 6 variations of the same logical URL collapse to one cache key
- different functional params and paths do NOT collapse
- invalid URL strings pass through unchanged
Validation
- `tsgo --noEmit`: pass.
- `bun test test/tool/webfetch-cache.test.ts test/tool/webfetch.test.ts`:
19 pass / 0 fail (15 new + 4 existing).
- Marker check: pass — altimate_change markers applied.
Expected telemetry impact
- Further reduction in residual webfetch 404 errors (currently 486
in 14d). Specifically helps long sessions revisiting dead doc URLs
multiple times, and LLMs pasting the same URL with different
tracking suffixes.
- No change for first-time fetches of legitimately new URLs.
Closes #838
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>1 parent 476d234 commit 25be299
2 files changed
Lines changed: 228 additions & 4 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
17 | 17 | | |
18 | 18 | | |
19 | 19 | | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
20 | 31 | | |
21 | | - | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
22 | 98 | | |
23 | 99 | | |
24 | | - | |
| 100 | + | |
| 101 | + | |
25 | 102 | | |
26 | 103 | | |
27 | | - | |
| 104 | + | |
28 | 105 | | |
29 | 106 | | |
30 | 107 | | |
| |||
34 | 111 | | |
35 | 112 | | |
36 | 113 | | |
| 114 | + | |
37 | 115 | | |
38 | 116 | | |
39 | 117 | | |
40 | 118 | | |
41 | 119 | | |
42 | | - | |
| 120 | + | |
43 | 121 | | |
44 | 122 | | |
45 | 123 | | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
46 | 134 | | |
47 | 135 | | |
48 | 136 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
0 commit comments