Skip to content

Commit 7529453

Browse files
anandgupta42claude
andcommitted
fix: [#838] address consensus review — preserve ref param, value-aware sort, LRU re-touch
Second-pass on PR #839 after code review. Three follow-ups in scope. 1. Preserve `ref` / `ref_src` / `ref_url` (MAJOR per reviewer) `ref` is functional on GitHub raw URLs and git-hosting APIs — `?ref=main` vs `?ref=v2.0` selects a different branch/tag and resolves to different content. Stripping it would suppress legitimate fetches when the agent (which fetches GitHub heavily) tries different refs of the same path. Removed `ref`, `ref_src`, `ref_url` from TRACKING_PARAMS. Kept `referrer` (the full word, tracking-only). Added doc-block at the TRACKING_PARAMS declaration explaining the exclusion so the next reviewer doesn't re-add them. 2. Value-aware param sort (MINOR) Duplicate-key params like `?a=1&a=2` and `?a=2&a=1` produced different cache keys despite carrying the same logical content — sorting by key alone leaked insertion order between same-key entries. Changed the comparator to sort by (key, value) tuple. 3. LRU re-touch semantics (MINOR) `Map.set` on an existing key preserves the original insertion position, so a frequently-failing URL stayed at the head of the FIFO queue and was the first to be evicted under pressure. Wrong for an LRU-flavoured cache. Delete-then-set on the re-touch path so the entry moves to the tail. 4. TTL doc (MINOR) Added a comment explaining the deliberate choice to use one TTL (30 min) for all three of 404, 410, 451 rather than split. The 3-status grouping is a simplification, not equivalence. 5. `@internal` JSDoc tags (NIT) Marked `_resetFailureCache` and `_failureCacheSize` as `@internal` so future tree-shakers / IDE hints discourage their use from production code. The underscore convention was the only guard. 6. Dropped explicit `__cf_chl_*` entries (NIT) Cloudflare challenge tokens are caught by the `isTrackingParamPrefix` check; the two enumerated entries (`__cf_chl_tk`, `__cf_chl_jschl_tk__`) were redundant. Kept utm_* in both the explicit list and the prefix check — the duplication documents the known cases (typed) while the prefix catches future variants. Tests - 3 new tests pin the new contracts: - `ref` param is preserved across normalization, AND different `ref` values do NOT collapse (regression guard against re-adding `ref` to TRACKING_PARAMS). - `referrer` (full word) is still stripped to confirm the distinction is real. - Duplicate-key params produce the same cache key regardless of input order (`?a=1&a=2` ↔ `?a=2&a=1`). - Updated existing test "preserves functional params when mixed with tracking params" to use a different tracking param (since `ref` is no longer one). Validation - `tsgo --noEmit`: pass. - `bun test test/tool/webfetch-cache.test.ts test/tool/webfetch.test.ts`: 22 pass / 0 fail (3 new tests). - Marker check: pass. Reviewer note - DeepSeek second opinion was unavailable (kilo CLI timeout, no output within window). Single-reviewer Claude analysis. Closes review feedback for PR #839. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
1 parent 25be299 commit 7529453

2 files changed

Lines changed: 82 additions & 9 deletions

File tree

‎packages/opencode/src/tool/webfetch.ts‎

Lines changed: 41 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -37,16 +37,28 @@ const FAILURE_CACHE_TTL = 30 * 60 * 1000 // 30 minutes
3737
* a regex sweep because false positives (e.g. stripping `?page=2`) would
3838
* cause real fetches to incorrectly hit the cache.
3939
*/
40+
// Notable exclusions (do NOT add these to TRACKING_PARAMS):
41+
// - `ref` / `ref_src` / `ref_url`: functional on GitHub raw URLs and
42+
// git-hosting APIs (`?ref=main` vs `?ref=v2.0` selects a different
43+
// branch/tag). Stripping `ref` would suppress legitimate fetches of
44+
// different refs against the same path — a coding agent fetches GitHub
45+
// heavily, so this is a real false-positive risk.
46+
// - `referrer` (full word) is tracking-only and IS in the list; the
47+
// abbreviated `ref` is not.
48+
//
49+
// The utm_* family is enumerated explicitly here AND covered by the
50+
// `isTrackingParamPrefix` check below. The duplication is intentional —
51+
// the explicit list documents the named params we know about, the prefix
52+
// check catches any future utm_* variant we haven't enumerated.
53+
// Cloudflare challenge tokens (`__cf_chl_*`) are handled by the prefix
54+
// check exclusively; no explicit entries needed.
4055
const TRACKING_PARAMS = new Set([
4156
"utm_source",
4257
"utm_medium",
4358
"utm_campaign",
4459
"utm_term",
4560
"utm_content",
4661
"utm_id",
47-
"ref",
48-
"ref_src",
49-
"ref_url",
5062
"referrer",
5163
"fbclid",
5264
"gclid",
@@ -61,8 +73,6 @@ const TRACKING_PARAMS = new Set([
6173
"_gl",
6274
"igshid",
6375
"mibextid",
64-
"__cf_chl_tk",
65-
"__cf_chl_jschl_tk__",
6676
"vero_conv",
6777
"vero_id",
6878
])
@@ -85,7 +95,13 @@ export function normalizeUrlForCache(url: string): string {
8595
if (TRACKING_PARAMS.has(k) || isTrackingParamPrefix(k)) continue
8696
kept.push([k, v])
8797
}
88-
kept.sort(([a], [b]) => (a < b ? -1 : a > b ? 1 : 0))
98+
// Sort by (key, value) so duplicate-key params produce a stable order:
99+
// `?a=1&a=2` and `?a=2&a=1` normalize to the same string. Sorting by key
100+
// alone leaked the original insertion order between same-key entries.
101+
kept.sort(([ak, av], [bk, bv]) => {
102+
if (ak !== bk) return ak < bk ? -1 : 1
103+
return av < bv ? -1 : av > bv ? 1 : 0
104+
})
89105
parsed.search = ""
90106
for (const [k, v] of kept) parsed.searchParams.append(k, v)
91107
// Also strip fragment — fragments don't change the fetched resource.
@@ -110,8 +126,19 @@ function isUrlCachedFailure(url: string): { status: number } | null {
110126
const MAX_CACHED_URLS = 500
111127

112128
function cacheUrlFailure(url: string, status: number): void {
129+
// 404 (permanent), 410 (gone), and 451 (legal block) get the same TTL
130+
// here intentionally. 410 is spec-permanent (won't lift), 404 is
131+
// overwhelmingly permanent in practice (typos, removed docs), and 451
132+
// can lift but rarely within a single session. Caching all three at
133+
// 30 min is a deliberate simplification — split TTLs would be marginal
134+
// complexity for marginal benefit.
113135
if (status === 404 || status === 410 || status === 451) {
114136
const key = normalizeUrlForCache(url)
137+
// Re-touch semantics: if the key is already in the cache, delete first
138+
// so the .set() reinserts at the tail of the FIFO. Otherwise a
139+
// frequently-failing URL would stay at the head and be the first
140+
// evicted under pressure — wrong order for an LRU-flavoured cache.
141+
if (failedUrls.has(key)) failedUrls.delete(key)
115142
if (failedUrls.size >= MAX_CACHED_URLS) {
116143
// Evict oldest entry (Map preserves insertion order)
117144
const oldest = failedUrls.keys().next().value
@@ -121,12 +148,18 @@ function cacheUrlFailure(url: string, status: number): void {
121148
}
122149
}
123150

124-
/** Reset the failure cache (testing only). */
151+
/**
152+
* Reset the failure cache.
153+
* @internal — only used by tests; do not call from production code.
154+
*/
125155
export function _resetFailureCache(): void {
126156
failedUrls.clear()
127157
}
128158

129-
/** Read the current failure cache size (testing only). */
159+
/**
160+
* Read the current failure cache size.
161+
* @internal — only used by tests; do not call from production code.
162+
*/
130163
export function _failureCacheSize(): number {
131164
return failedUrls.size
132165
}

‎packages/opencode/test/tool/webfetch-cache.test.ts‎

Lines changed: 41 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -51,9 +51,11 @@ describe("normalizeUrlForCache", () => {
5151
})
5252

5353
test("preserves functional params when mixed with tracking params", () => {
54+
// `ref` is preserved (it's functional on GitHub etc.); `referrer` is stripped.
55+
// utm_* and fbclid are stripped.
5456
expect(
5557
normalizeUrlForCache(
56-
"https://example.com/post?id=42&utm_source=email&utm_campaign=monthly&ref=fred",
58+
"https://example.com/post?id=42&utm_source=email&utm_campaign=monthly&fbclid=abc&referrer=twitter",
5759
),
5860
).toBe("https://example.com/post?id=42")
5961
})
@@ -123,6 +125,44 @@ describe("normalizeUrlForCache", () => {
123125
normalizeUrlForCache("https://example.com/v2/api"),
124126
)
125127
})
128+
129+
test("preserves `ref` param (functional on GitHub raw URLs / git APIs)", () => {
130+
// GitHub: `?ref=main` vs `?ref=v2.0` selects different branches/tags.
131+
// Stripping `ref` would suppress legitimate fetches of different refs.
132+
// Caught by reviewer (2026-05-22). Same for ref_src and ref_url.
133+
expect(normalizeUrlForCache("https://api.github.com/repos/x/y/contents/foo.md?ref=main")).toBe(
134+
"https://api.github.com/repos/x/y/contents/foo.md?ref=main",
135+
)
136+
expect(normalizeUrlForCache("https://api.github.com/repos/x/y/contents/foo.md?ref=v2.0")).toBe(
137+
"https://api.github.com/repos/x/y/contents/foo.md?ref=v2.0",
138+
)
139+
// Different `ref` values must NOT collapse — they identify different
140+
// resources. Pinning this prevents a regression that would re-add ref
141+
// to TRACKING_PARAMS.
142+
expect(
143+
normalizeUrlForCache("https://api.github.com/repos/x/y/contents/foo.md?ref=main"),
144+
).not.toBe(normalizeUrlForCache("https://api.github.com/repos/x/y/contents/foo.md?ref=v2.0"))
145+
})
146+
147+
test("still strips `referrer` (full word, tracking-only)", () => {
148+
// The abbreviated `ref` is functional; the full word `referrer` is
149+
// a tracking header and stays in TRACKING_PARAMS.
150+
expect(normalizeUrlForCache("https://example.com/page?referrer=twitter")).toBe(
151+
"https://example.com/page",
152+
)
153+
})
154+
155+
test("duplicate-key params collapse to a stable order regardless of input order", () => {
156+
// Without value-aware sort, `?a=1&a=2` and `?a=2&a=1` would produce
157+
// different cache keys despite carrying the same logical content.
158+
expect(normalizeUrlForCache("https://example.com/x?a=1&a=2")).toBe(
159+
normalizeUrlForCache("https://example.com/x?a=2&a=1"),
160+
)
161+
// And the normalized form is deterministic:
162+
expect(normalizeUrlForCache("https://example.com/x?a=2&a=1")).toBe(
163+
"https://example.com/x?a=1&a=2",
164+
)
165+
})
126166
})
127167

128168
describe("failure cache helpers", () => {

0 commit comments

Comments
 (0)