Repository navigation
Commit 7529453
fix: [#838] address consensus review — preserve
Second-pass on PR #839 after code review. Three follow-ups in scope.
1. Preserve `ref` / `ref_src` / `ref_url` (MAJOR per reviewer)
`ref` is functional on GitHub raw URLs and git-hosting APIs —
`?ref=main` vs `?ref=v2.0` selects a different branch/tag and
resolves to different content. Stripping it would suppress
legitimate fetches when the agent (which fetches GitHub heavily)
tries different refs of the same path. Removed `ref`, `ref_src`,
`ref_url` from TRACKING_PARAMS. Kept `referrer` (the full word,
tracking-only). Added doc-block at the TRACKING_PARAMS declaration
explaining the exclusion so the next reviewer doesn't re-add them.
2. Value-aware param sort (MINOR)
Duplicate-key params like `?a=1&a=2` and `?a=2&a=1` produced
different cache keys despite carrying the same logical content —
sorting by key alone leaked insertion order between same-key
entries. Changed the comparator to sort by (key, value) tuple.
3. LRU re-touch semantics (MINOR)
`Map.set` on an existing key preserves the original insertion
position, so a frequently-failing URL stayed at the head of the
FIFO queue and was the first to be evicted under pressure.
Wrong for an LRU-flavoured cache. Delete-then-set on the
re-touch path so the entry moves to the tail.
4. TTL doc (MINOR)
Added a comment explaining the deliberate choice to use one TTL
(30 min) for all three of 404, 410, 451 rather than split. The
3-status grouping is a simplification, not equivalence.
5. `@internal` JSDoc tags (NIT)
Marked `_resetFailureCache` and `_failureCacheSize` as `@internal`
so future tree-shakers / IDE hints discourage their use from
production code. The underscore convention was the only guard.
6. Dropped explicit `__cf_chl_*` entries (NIT)
Cloudflare challenge tokens are caught by the `isTrackingParamPrefix`
check; the two enumerated entries (`__cf_chl_tk`, `__cf_chl_jschl_tk__`)
were redundant. Kept utm_* in both the explicit list and the prefix
check — the duplication documents the known cases (typed) while
the prefix catches future variants.
Tests
- 3 new tests pin the new contracts:
- `ref` param is preserved across normalization, AND different `ref`
values do NOT collapse (regression guard against re-adding `ref` to
TRACKING_PARAMS).
- `referrer` (full word) is still stripped to confirm the distinction
is real.
- Duplicate-key params produce the same cache key regardless of
input order (`?a=1&a=2` ↔ `?a=2&a=1`).
- Updated existing test "preserves functional params when mixed with
tracking params" to use a different tracking param (since `ref` is
no longer one).
Validation
- `tsgo --noEmit`: pass.
- `bun test test/tool/webfetch-cache.test.ts test/tool/webfetch.test.ts`:
22 pass / 0 fail (3 new tests).
- Marker check: pass.
Reviewer note
- DeepSeek second opinion was unavailable (kilo CLI timeout, no output
within window). Single-reviewer Claude analysis.
Closes review feedback for PR #839.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>ref param, value-aware sort, LRU re-touch1 parent 25be299 commit 7529453
2 files changed
Lines changed: 82 additions & 9 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
37 | 37 | | |
38 | 38 | | |
39 | 39 | | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
40 | 55 | | |
41 | 56 | | |
42 | 57 | | |
43 | 58 | | |
44 | 59 | | |
45 | 60 | | |
46 | 61 | | |
47 | | - | |
48 | | - | |
49 | | - | |
50 | 62 | | |
51 | 63 | | |
52 | 64 | | |
| |||
61 | 73 | | |
62 | 74 | | |
63 | 75 | | |
64 | | - | |
65 | | - | |
66 | 76 | | |
67 | 77 | | |
68 | 78 | | |
| |||
85 | 95 | | |
86 | 96 | | |
87 | 97 | | |
88 | | - | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
89 | 105 | | |
90 | 106 | | |
91 | 107 | | |
| |||
110 | 126 | | |
111 | 127 | | |
112 | 128 | | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
113 | 135 | | |
114 | 136 | | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
115 | 142 | | |
116 | 143 | | |
117 | 144 | | |
| |||
121 | 148 | | |
122 | 149 | | |
123 | 150 | | |
124 | | - | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
125 | 155 | | |
126 | 156 | | |
127 | 157 | | |
128 | 158 | | |
129 | | - | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
130 | 163 | | |
131 | 164 | | |
132 | 165 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
51 | 51 | | |
52 | 52 | | |
53 | 53 | | |
| 54 | + | |
| 55 | + | |
54 | 56 | | |
55 | 57 | | |
56 | | - | |
| 58 | + | |
57 | 59 | | |
58 | 60 | | |
59 | 61 | | |
| |||
123 | 125 | | |
124 | 126 | | |
125 | 127 | | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
126 | 166 | | |
127 | 167 | | |
128 | 168 | | |
| |||
0 commit comments