Repository navigation
Commit a411be2
fix: [AltimateAI#838] webfetch 404 cache — longer TTL + URL normalization (AltimateAI#839)
* fix: [AltimateAI#838] webfetch 404 cache — extend TTL + normalize URLs across tracking-param variations
Telemetry-2026-05-21 showed 486 residual webfetch 404 errors (down 78%
from March's 2,222 — the existing failure cache is doing its job). The
remaining cases mostly escape because of two gaps:
1. TTL was 5 minutes. Real sessions often run 30+ minutes; the same
dead URL re-hit the network after the first 5-minute window.
2. Cache key was the raw URL string. LLM-generated URLs frequently
include tracking params (utm_source, fbclid, gclid, etc.) or
fragments — each variation got its own cache entry, so a known-bad
URL with different tracking params re-hit the network on every
call.
Changes
- Extend FAILURE_CACHE_TTL from 5 minutes to 30 minutes. URLs that
404 rarely self-heal within a session; the longer window catches
the slow-burn case without measurably increasing the risk of
caching a transient failure.
- Add `normalizeUrlForCache()` and use it for every cache read/write.
Strips universally-tracking params (utm_*, ref*, fbclid, gclid,
msclkid, yclid, twclid, li_fat_id, mc_cid, mc_eid, _ga, _gl,
igshid, mibextid, __cf_chl_*), strips URL fragments, lowercases
the hostname, and sorts remaining params. Conservative list — only
params known to be tracking-only are stripped, so functional
params (page, id, q, etc.) survive untouched.
- Export `_resetFailureCache` and `_failureCacheSize` for tests.
Tests
New test/tool/webfetch-cache.test.ts (15 tests):
- strips utm_*, fbclid, gclid, msclkid family
- preserves functional params (page, id, q, etc.)
- sorts params for stable cache key
- strips fragments
- lowercases hostname (case-insensitive per RFC 3986)
- preserves path case (paths ARE case-sensitive per HTTP)
- collapses Cloudflare __cf_chl_* challenge tokens
- 6 variations of the same logical URL collapse to one cache key
- different functional params and paths do NOT collapse
- invalid URL strings pass through unchanged
Validation
- `tsgo --noEmit`: pass.
- `bun test test/tool/webfetch-cache.test.ts test/tool/webfetch.test.ts`:
19 pass / 0 fail (15 new + 4 existing).
- Marker check: pass — altimate_change markers applied.
Expected telemetry impact
- Further reduction in residual webfetch 404 errors (currently 486
in 14d). Specifically helps long sessions revisiting dead doc URLs
multiple times, and LLMs pasting the same URL with different
tracking suffixes.
- No change for first-time fetches of legitimately new URLs.
Closes AltimateAI#838
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* fix: [AltimateAI#838] address consensus review — preserve `ref` param, value-aware sort, LRU re-touch
Second-pass on PR AltimateAI#839 after code review. Three follow-ups in scope.
1. Preserve `ref` / `ref_src` / `ref_url` (MAJOR per reviewer)
`ref` is functional on GitHub raw URLs and git-hosting APIs —
`?ref=main` vs `?ref=v2.0` selects a different branch/tag and
resolves to different content. Stripping it would suppress
legitimate fetches when the agent (which fetches GitHub heavily)
tries different refs of the same path. Removed `ref`, `ref_src`,
`ref_url` from TRACKING_PARAMS. Kept `referrer` (the full word,
tracking-only). Added doc-block at the TRACKING_PARAMS declaration
explaining the exclusion so the next reviewer doesn't re-add them.
2. Value-aware param sort (MINOR)
Duplicate-key params like `?a=1&a=2` and `?a=2&a=1` produced
different cache keys despite carrying the same logical content —
sorting by key alone leaked insertion order between same-key
entries. Changed the comparator to sort by (key, value) tuple.
3. LRU re-touch semantics (MINOR)
`Map.set` on an existing key preserves the original insertion
position, so a frequently-failing URL stayed at the head of the
FIFO queue and was the first to be evicted under pressure.
Wrong for an LRU-flavoured cache. Delete-then-set on the
re-touch path so the entry moves to the tail.
4. TTL doc (MINOR)
Added a comment explaining the deliberate choice to use one TTL
(30 min) for all three of 404, 410, 451 rather than split. The
3-status grouping is a simplification, not equivalence.
5. `@internal` JSDoc tags (NIT)
Marked `_resetFailureCache` and `_failureCacheSize` as `@internal`
so future tree-shakers / IDE hints discourage their use from
production code. The underscore convention was the only guard.
6. Dropped explicit `__cf_chl_*` entries (NIT)
Cloudflare challenge tokens are caught by the `isTrackingParamPrefix`
check; the two enumerated entries (`__cf_chl_tk`, `__cf_chl_jschl_tk__`)
were redundant. Kept utm_* in both the explicit list and the prefix
check — the duplication documents the known cases (typed) while
the prefix catches future variants.
Tests
- 3 new tests pin the new contracts:
- `ref` param is preserved across normalization, AND different `ref`
values do NOT collapse (regression guard against re-adding `ref` to
TRACKING_PARAMS).
- `referrer` (full word) is still stripped to confirm the distinction
is real.
- Duplicate-key params produce the same cache key regardless of
input order (`?a=1&a=2` ↔ `?a=2&a=1`).
- Updated existing test "preserves functional params when mixed with
tracking params" to use a different tracking param (since `ref` is
no longer one).
Validation
- `tsgo --noEmit`: pass.
- `bun test test/tool/webfetch-cache.test.ts test/tool/webfetch.test.ts`:
22 pass / 0 fail (3 new tests).
- Marker check: pass.
Reviewer note
- DeepSeek second opinion was unavailable (kilo CLI timeout, no output
within window). Single-reviewer Claude analysis.
Closes review feedback for PR AltimateAI#839.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* fix: [AltimateAI#838] address re-review — strip userinfo + add HubSpot/Marketo/Adobe/Piwik tracking params
Reviewer flagged two cache-key fragmentation cases not covered by the
first pass:
1. Basic-auth userinfo (user:pass@host) was left in the cache key,
letting the same logical URL with and without credentials occupy
two slots. Also a hygiene win — telemetry never logs cache keys
with creds.
2. The tracking-param set covered Google/Meta/Twitter ad ecosystems
but missed three major B2B martech vendors that dominate the
email-newsletter URLs LLMs love to retrieve:
- HubSpot: _hsenc, _hsmi, hsCtaTracking
- Marketo: mkt_tok
- Adobe Analytics: s_cid, s_kwcid
- Piwik/Matomo: pk_campaign, pk_kwd, pk_source, pk_medium,
pk_content, piwik_campaign, piwik_keyword
Deliberately omitted Adobe Target's at_* prefix — too risky for false
positives (e.g., at_preview_* are functional). The named s_cid /
s_kwcid params catch the high-volume cases.
5 new tests pin the contract; existing 19 still pass.
Closes AltimateAI#838
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* fix: [AltimateAI#838] address fourth-pass re-review — split 451 TTL + cover LRU re-touch + fix stale doc
Reviewer surfaced four issues. Three actionable:
1. STALE HEADER DOC — header comment listed `ref` as a stripped param,
but the exclusion block immediately below explains we deliberately
keep it. Future readers would hit cognitive whiplash. Removed `ref`
from the inline list; switched the cache description to "process-
level" (more accurate — Map is module-scope, persists across logical
sessions in daemon/MCP mode).
2. SPLIT 451 TTL (webfetch.ts) — 404/410 are resource-permanent (30 min
stays right). 451 is observer-conditional: an agent that swaps
networks (VPN, travel, hotspot) inside a long-cache window would be
locked out of URLs that are reachable from the new vantage point.
Added FAILURE_CACHE_TTL_451 = 5 min and ttlForStatus() helper.
3. LRU RE-TOUCH WAS UNTESTED — the most subtle change in the original
PR (delete-then-set so frequently-failing URLs don't sit at the head
of the FIFO) had zero test coverage. Exposed three test-only helpers
(_cacheUrlFailureForTest, _failureCacheKeysInOrder,
_isUrlCachedFailureForTest) and added 6 cache state-machine tests:
- cacheUrlFailure → isUrlCachedFailure round-trip
- only 404/410/451 are cached (other statuses ignored)
- tracking-param variations all hit the same cache key
- re-touch moves entry to tail (pins the LRU contract)
- 451 + 410 entries coexist (TTL split is wired)
- _resetFailureCache actually clears cached entries
29/29 tests pass; marker check clean.
Closes AltimateAI#838
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* fix: [AltimateAI#838] address coderabbit nit — Matomo test name + pin "exact-name, not prefix" contract
Reviewer flagged that the test title "strips Piwik/Matomo tracking
params (pk_*, piwik_*)" overstates the contract — TRACKING_PARAMS does
exact-name matching, NOT pk_* / piwik_* prefix stripping. The title
was documenting behavior that isn't guaranteed.
Renamed to enumerate the named params we DO strip and added two
counter-tests verifying that pk_unknown / piwik_unknown ARE preserved.
This pins the "exact-name, not prefix" contract so a future refactor
can't silently flip to prefix-stripping without breaking these.
Restraint rationale documented inline: prefix-stripping `pk_` risks
false positives on functional params from unrelated services that
happen to start with the same two letters.
29/29 tests pass; marker check clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>1 parent fbc7f97 commit a411be2
2 files changed
Lines changed: 527 additions & 7 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
15 | 15 | | |
16 | 16 | | |
17 | 17 | | |
18 | | - | |
19 | | - | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
20 | 37 | | |
21 | | - | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
22 | 148 | | |
23 | 149 | | |
24 | | - | |
| 150 | + | |
| 151 | + | |
25 | 152 | | |
26 | | - | |
27 | | - | |
| 153 | + | |
| 154 | + | |
28 | 155 | | |
29 | 156 | | |
30 | 157 | | |
| |||
33 | 160 | | |
34 | 161 | | |
35 | 162 | | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
36 | 172 | | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
37 | 179 | | |
38 | 180 | | |
39 | 181 | | |
40 | 182 | | |
41 | 183 | | |
42 | | - | |
| 184 | + | |
43 | 185 | | |
44 | 186 | | |
45 | 187 | | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
46 | 229 | | |
47 | 230 | | |
48 | 231 | | |
| |||
0 commit comments