fix(fatf-listing): Wayback Machine fallback for Cloudflare-blocked fetches#3413
Conversation
…tches FATF's www.fatf-gafi.org enforces a Cloudflare "Just a moment…" JS challenge that blocks both direct Node fetches and CONNECT-proxy (Decodo) requests with HTTP 403. PR #3407's seeder consequently writes nothing on every Railway tick — economic:fatf-listing:v1 shows status=EMPTY in /api/health because the key is never published. Direct probe today (2026-04-25): direct fetch with full Chrome browser headers (Accept-Language, Referer, Sec-Fetch-*) still 403s from a residential IP. The challenge requires JS execution, which neither Node fetch nor httpsProxyFetchRaw provides. Header tweaks won't help. Wayback Machine bypasses Cloudflare entirely (its crawler captures through different infrastructure). CDX API has multiple status:200 snapshots in 2026 (most recent 2026-04-03 with current Feb-2026-plenary content); FATF's 3x/year publishing cadence vs the seeder's 30-day bundle interval and 42-day STALE_SEED threshold means Wayback's 1-3 day capture lag is irrelevant. Three-tier fetch: direct -> CONNECT proxy -> Wayback. The first two are preserved unchanged so this seeder transparently recovers the moment FATF rotates off Cloudflare or whitelists Decodo egress. Wayback fetch uses the id_ URL modifier (/web/<timestamp>id_/<url>), which returns the original captured HTML byte-for-byte WITHOUT Wayback's banner injection or href/src rewriting. The seeder's existing parser (findPublicationLink, extractListedCountries) works unchanged. A regression test pins the modifier to defend against accidental cleanup that switches to the bare /web/timestamp/url form. Tests: +7 cases on fetchViaWayback covering happy path, CDX/snapshot HTTP errors, no-200-snapshots window, malformed timestamps, the id_ modifier pin, and the from-date floor in the CDX query. Bundle interval, validate, and seed-meta wiring unchanged.
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
Greptile SummaryThis PR adds a Wayback Machine third-tier fallback to
Confidence Score: 3/5Not safe to merge as-is — the Wayback fallback may silently serve a months-old snapshot instead of the most recent one. One P1 bug: scripts/seed-fatf-listing.mjs — specifically the Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[fetchHtml called] --> B[Tier 1: Direct fetch\nwith Chrome UA]
B -->|ok| Z[Return HTML]
B -->|fail| C{_proxyAuth\nconfigured?}
C -->|yes| D[Tier 2: CONNECT proxy\nhttpsProxyFetchRaw]
C -->|no| E[Log: no proxy,\nfall back to Wayback]
D -->|ok| Z
D -->|fail| E
E --> F[Tier 3: fetchViaWayback]
F --> G[CDX API query\nlimit=20 ascending ⚠️\nfilter=statuscode:200\nfrom=180d ago]
G -->|ok 200| H[Pick rows at last index\n⚠️ may be 20th-oldest\nnot most recent]
G -->|non-200| ERR[Throw CDX error]
H --> I[Fetch snapshot via id_ modifier\nno Wayback toolbar injection]
I -->|ok| Z
I -->|non-200| ERR2[Throw snapshot error]
ERR --> FINAL[Throw combined error\ndirect + proxy + wayback msgs]
ERR2 --> FINAL
Reviews (1): Last reviewed commit: "fix(fatf-listing): Wayback Machine fallb..." | Re-trigger Greptile |
| .toISOString() | ||
| .slice(0, 10) | ||
| .replace(/-/g, ''); | ||
| const cdxUrl = `${WAYBACK_CDX_URL}?url=${encodeURIComponent(url)}&filter=statuscode:200&output=json&from=${fromDate}&limit=20`; |
There was a problem hiding this comment.
limit=20 returns 20 oldest snapshots, not the 20 most recent
The CDX API returns results in ascending timestamp order by default, so limit=20 gives the 20 oldest snapshots since fromDate. rows[rows.length - 1] is therefore the 20th-oldest, not the most recent. The code comment says "the LAST row is most recent" — this is only true when there are ≤ 20 total captures in the window. The Wayback Machine crawls major sites like FATF's multiple times per week; over 180 days that easily exceeds 20 captures, meaning the code would silently use a months-old snapshot and miss the 2026-04-03 capture cited in the PR description.
Use limit=-1 (CDX shorthand for "return the single most recent match") to guarantee the latest snapshot is selected:
| const cdxUrl = `${WAYBACK_CDX_URL}?url=${encodeURIComponent(url)}&filter=statuscode:200&output=json&from=${fromDate}&limit=20`; | |
| const cdxUrl = `${WAYBACK_CDX_URL}?url=${encodeURIComponent(url)}&filter=statuscode:200&output=json&from=${fromDate}&limit=-1`; |
The rest of the function (rows.length < 2 guard, rows[rows.length - 1]) then works correctly with 1 returned data row.
| const cdxUrl = `${WAYBACK_CDX_URL}?url=${encodeURIComponent(url)}&filter=statuscode:200&output=json&from=${fromDate}&limit=20`; | |
| const cdxUrl = `${WAYBACK_CDX_URL}?url=${encodeURIComponent(url)}&filter=statuscode:200&output=json&from=${fromDate}&limit=-1`; |
| const cdxResp = await fetchFn(cdxUrl, { signal: AbortSignal.timeout(20_000) }); | ||
| if (!cdxResp.ok) throw new Error(`Wayback CDX HTTP ${cdxResp.status} for ${url}`); | ||
| const rows = await cdxResp.json(); | ||
| // CDX returns: [headerRow, ...snapshotRows]. Each snapshot row = | ||
| // [urlkey, timestamp, original, mimetype, statuscode, digest, length]. | ||
| // Snapshots are timestamp-ascending, so the LAST row is most recent. | ||
| if (!Array.isArray(rows) || rows.length < 2) { | ||
| throw new Error(`Wayback has no status-200 snapshots for ${url} since ${fromDate}`); | ||
| } | ||
| const latest = rows[rows.length - 1]; | ||
| const timestamp = latest[1]; | ||
| if (!/^\d{14}$/.test(timestamp)) { | ||
| throw new Error(`Wayback CDX returned malformed timestamp "${timestamp}" for ${url}`); | ||
| } | ||
| const snapshotUrl = `https://web.archive.org/web/${timestamp}id_/${url}`; | ||
| const snapResp = await fetchFn(snapshotUrl, { signal: AbortSignal.timeout(30_000) }); |
There was a problem hiding this comment.
Missing
User-Agent on Wayback fetches
AGENTS.md lists "Always include User-Agent header in server-side fetch calls" as a critical convention. Both the CDX request and the snapshot request omit the User-Agent header. While archive.org is unlikely to block headless requests, this is inconsistent with the rest of the seeder (which passes CHROME_UA to the FATF direct fetch) and with the repo convention.
| const cdxResp = await fetchFn(cdxUrl, { signal: AbortSignal.timeout(20_000) }); | |
| if (!cdxResp.ok) throw new Error(`Wayback CDX HTTP ${cdxResp.status} for ${url}`); | |
| const rows = await cdxResp.json(); | |
| // CDX returns: [headerRow, ...snapshotRows]. Each snapshot row = | |
| // [urlkey, timestamp, original, mimetype, statuscode, digest, length]. | |
| // Snapshots are timestamp-ascending, so the LAST row is most recent. | |
| if (!Array.isArray(rows) || rows.length < 2) { | |
| throw new Error(`Wayback has no status-200 snapshots for ${url} since ${fromDate}`); | |
| } | |
| const latest = rows[rows.length - 1]; | |
| const timestamp = latest[1]; | |
| if (!/^\d{14}$/.test(timestamp)) { | |
| throw new Error(`Wayback CDX returned malformed timestamp "${timestamp}" for ${url}`); | |
| } | |
| const snapshotUrl = `https://web.archive.org/web/${timestamp}id_/${url}`; | |
| const snapResp = await fetchFn(snapshotUrl, { signal: AbortSignal.timeout(30_000) }); | |
| const cdxResp = await fetchFn(cdxUrl, { headers: { 'User-Agent': CHROME_UA }, signal: AbortSignal.timeout(20_000) }); |
And similarly for the snapshot fetch on line 100.
PR #3413 review: with limit=20 the CDX query returned the OLDEST 20 captures within the from-date window (CDX default sort is timestamp- ascending, positive `limit=N` = first N rows = oldest N). Picking rows[length-1] then gave us the 20th-oldest capture, not the truly most-recent. FATF accumulates well over 20 captures per 180-day window so this would silently serve a stale archived list while a newer snapshot existed — breaking the whole "latest FATF state via Wayback" guarantee. CDX accepts negative `limit` to mean "last N captures by timestamp". `limit=-1` returns just the single most-recent capture, which is exactly what we need and also avoids fetching ~20× more rows than we use. Picking logic (rows[length-1]) is unchanged — with limit=-1 the response is exactly [headerRow, mostRecentSnapshot] so length-1 still points at the right row. Test additions: - Pin `limit=-1` in the URL-shape test (regression guard). - Negative-assertion that no positive `limit=N` is present (defense against a future cleanup that "tidies up" the syntax and silently reverts to the bug).
|
Reviewer was right — verified the bug against the live CDX API. With Fix in Tests:
Full /newpr verification re-run, all 8 checks green; |
Greptile P2 review finding on PR #3413. AGENTS.md:185 mandates "Always include User-Agent header in server-side fetch calls". The direct FATF fetch passes CHROME_UA but the Wayback fallback's two fetches (CDX query + snapshot) omitted it, breaking convention. archive.org doesn't currently block UA-less requests, but: (a) house-style consistency is the point, (b) a future Wayback rate limiter could reasonably enforce UA, (c) the direct fetch the seeder makes to fatf-gafi.org sets CHROME_UA, so a Wayback fallback that swaps out the UA is unexpected behavior. Test: +1 case asserts both calls receive a User-Agent header matching the canonical CHROME_UA pattern (Mozilla/5.0...). Pinned strongly enough to catch a regression that drops the header OR substitutes a placeholder/empty value. Also addresses Greptile P1 (limit=20 oldest-not-newest), already fixed in commit ab4797a -- Greptile's last-reviewed commit was the pre-fix a4f254e.
|
Greptile review responses: P1 — limit=20 returns 20 oldest snapshots — already fixed in commit ab4797a (pushed yesterday). Greptile's last-reviewed commit was the pre-fix a4f254e; a re-trigger should pick up the fix. P2 — Missing User-Agent on Wayback fetches — fixed in commit f74be5e (just pushed). AGENTS.md:185 mandates 'Always include User-Agent header in server-side fetch calls'. Both the CDX query and the snapshot fetch now pass CHROME_UA (matches the canonical UA the direct FATF fetch already uses). Test added: 'sends a User-Agent header on BOTH the CDX query and the snapshot fetch (AGENTS.md convention)'. Asserts both calls receive a non-empty UA matching /Mozilla/5.0/ — strong enough to catch a regression that drops the header OR substitutes a placeholder. Full /newpr verification re-run on f74be5e: 7195 tests passing (up from 7194), all 8 checks green. |
…from Railway (#3415) * fix(fatf-listing): proxy-fallback + describeErr for Wayback failures from Railway Production observation 2026-04-25T20:35 (post PR #3413 merge): the Wayback fallback path itself fails on every Railway tick. Direct CDX query times out at 20s, subsequent retries surface as "fetch failed" with no cause unwrapped. Local desktop probes complete in <2s, so the issue is Railway-egress-specific — the IP pool gets soft-rate-limited or routed slowly to archive.org. Pre-fix log: wayback=The operation was aborted due to timeout wayback=fetch failed wayback=fetch failed The bare "fetch failed" comes from undici's default error message and gives operators no actionable signal. Three changes: 1. CDX timeout 20s -> 45s. Generous ceiling that still keeps the seeder under bundle-runner's 120s timeoutMs and accommodates the added proxy-fallback hops. 2. Two-tier per-call inside fetchViaWayback: direct fetch first (fast when egress isn't rate-limited), CONNECT proxy fallback (Decodo's residential pool isn't bucketed alongside Railway IPs at archive.org). Same shape as the seeder's outer fetchHtml three-tier. Applied to BOTH the CDX query AND the snapshot fetch. 3. describeErr helper (mirrors seed-unrest-events.mjs:162) unwraps err.cause so production logs surface the actual failure mode (DNS / TCP reset / TLS abort) instead of undici's bare "fetch failed". Tests: +2 cases on fetchViaWayback (proxy-fallback happy path, err.cause unwrapping when both tiers fail). Updated 2 existing tests to pass `proxyAuth: null` so they stay focused on the direct-only failure mode the new fallback would otherwise mask. * fix(fatf-listing): bound per-URL fetch budget to fit FATF-Listing section timeoutMs Reviewer-found gap on PR #3415. The proxy-fallback addition created a budget overflow: direct(30s) + proxy(30s) + wayback-CDX-direct(45s) + wayback-CDX-proxy(45s) + wayback-snapshot-direct(45s) + wayback- snapshot-proxy(45s) = 240s per URL. FATF fetches 3 URLs (entry sequential, black/grey parallel) → ≤480s end-to-end worst case while seed-bundle-macro.mjs's FATF-Listing section was capped at 120_000ms. That meant bundle-runner would SIGTERM mid-fetch instead of letting runSeed reach its graceful "Failed gracefully" path — turning a recoverable fetch failure into a timeout-driven section failure. Three-part fix: 1. Tighten per-tier timeouts so worst-case-per-URL fits in 125s: - Direct fetchHtml: 30s → 10s (Cloudflare 403s in <1s when blocking) - Proxy fetchHtml: 30s → 15s (Decodo CONNECT overhead is ~1s) - Wayback per-tier (CDX + snapshot, direct + proxy): 45s → 25s (still 25% margin over Railway's observed 20s+ rate-limit window) 2. Bump seed-bundle-macro.mjs FATF-Listing section timeoutMs: 120_000 → 300_000. Matches peer sections (BIS-Data, Eurostat, etc.) and accommodates worst-case ≤250s end-to-end with ~50s margin. 3. Static-shape regression test that pins all four numeric values (10_000 / 15_000 / WAYBACK_TIMEOUT_MS = 25_000 / section 300_000) so a future cleanup can't silently regress the budget. Math after fix: Per URL: 10 + 15 + 25 + 25 + 25 + 25 = 125s Sequential entry + parallel black/grey: ≤ 250s Section budget: 300s (50s margin) * refactor(seed-utils): hoist describeErr + clarify proxy-path UA Greptile P2 findings on PR #3415: - describeErr was duplicated byte-for-byte in seed-fatf-listing.mjs and seed-unrest-events.mjs. Future improvements (AggregateError, multi- level cause chains) would need to be applied in both. Hoisted to scripts/_seed-utils.mjs alongside CHROME_UA, resolveProxyForConnect, httpsProxyFetchRaw — both seeders now import the canonical helper. - Proxy CDX/snapshot path in fetchViaWayback didn't visibly pass a User-Agent header. httpsProxyFetchRaw DOES inject CHROME_UA internally (scripts/_seed-utils.mjs line 174), so the AGENTS.md UA convention is already satisfied — but the call site doesn't make that obvious. Added inline comments at both proxy-fetcher call sites pointing at the implementation, so a future reader doesn't think the proxy path is header-less and 'fix' it by passing redundant headers. Other Greptile P2s on PR #3415 are stale (last-reviewed commit was ba863d3, before the budget commit 69e944b addressed them): - 'Worst-case 150s > 120s budget' → fixed in 69e944b (10/15/25*4 = 125s/URL; section bumped 120k→300k). - 'Placeholder PR #34xx in comment' → that comment was rewritten in 69e944b and no longer references a placeholder.
Problem
PR #3407's FATF seeder writes nothing on every Railway tick —
economic:fatf-listing:v1showsstatus=EMPTYin/api/healthbecause the canonical key has never been published.Root cause:
www.fatf-gafi.orgenforces a Cloudflare "Just a moment…" JS challenge that returns HTTP 403 to any client that doesn't execute JavaScript. Both direct Nodefetchand the existinghttpsProxyFetchRawCONNECT-proxy (Decodo) leg fail this check. From production logs (2026-04-25T19:01 UTC, run id1777143709725-d2wtr1):I confirmed today (2026-04-25) that even with full Chrome browser headers (Accept-Language, Referer, Sec-Fetch-Dest/Mode/Site, Upgrade-Insecure-Requests) the request still returns the Cloudflare challenge HTML (
<title>Just a moment...</title>). Header tweaks won't help.Fix
Add a third tier to
fetchHtml: direct → CONNECT proxy → Wayback Machine. The first two tiers are unchanged; Wayback runs only when both fail.Why Wayback works
statuscode:200snapshots in 2026 (most recent 2026-04-03 14:49 UTC, with current Feb-2026-plenary content — i.e. today's actual real-world FATF state).Why the
id_modifierThe implementation fetches
https://web.archive.org/web/<timestamp>id_/<url>. Theid_modifier returns the original captured HTML byte-for-byte without Wayback's banner injection or href/src rewriting — keeping the existing parser (findPublicationLink,extractListedCountries) unchanged. A regression test pins the modifier so a future "cleanup" can't silently switch to the bare/web/timestamp/urlform (which would prepend ~3KB of Wayback toolbar HTML and rewrite every link, breaking the parser).Self-healing direct path
The order is
direct → proxy → Wayback, not Wayback-first. The moment FATF rotates off Cloudflare or Decodo egress IPs get whitelisted, the seeder transparently goes back to live data with no code change.Tests
+7cases onfetchViaWaybackintests/seed-fatf-listing.test.mjs:id_filter=statuscode:200,from=YYYYMMDD,output=jsonid_modifier pin (regression guard against parser-breaking cleanup)Test plan
npm run typecheck+typecheck:apinpm run lint(no errors)npm run test:data(7194 pass)node --test tests/edge-functions.test.mjs(177 pass)npm run lint:mdnpm run version:check[FATF-Listing]log line that does NOT end inFETCH FAILED: HTTP 403. Wayback fallback kicks in after the proxy 403, parses the snapshot HTML, andrunSeedwriteseconomic:fatf-listing:v1./api/healthfatfListing.statusflips fromEMPTYtoOK(orSTALE_SEEDclearing once seed-meta is written).Out of scope
EMPTYstatus (separate root cause: lock contention / validate floor / bundle timeout). Will be a separate PR adding a SIGTERM lock-release handler defensively.