Skip to content

fix(fatf-listing): Wayback Machine fallback for Cloudflare-blocked fetches#3413

Merged
koala73 merged 3 commits into
mainfrom
fix/fatf-wayback-fallback
Apr 25, 2026
Merged

fix(fatf-listing): Wayback Machine fallback for Cloudflare-blocked fetches#3413
koala73 merged 3 commits into
mainfrom
fix/fatf-wayback-fallback

Conversation

@koala73

@koala73 koala73 commented Apr 25, 2026

Copy link
Copy Markdown
Owner

Problem

PR #3407's FATF seeder writes nothing on every Railway tick — economic:fatf-listing:v1 shows status=EMPTY in /api/health because the canonical key has never been published.

Root cause: www.fatf-gafi.org enforces a Cloudflare "Just a moment…" JS challenge that returns HTTP 403 to any client that doesn't execute JavaScript. Both direct Node fetch and the existing httpsProxyFetchRaw CONNECT-proxy (Decodo) leg fail this check. From production logs (2026-04-25T19:01 UTC, run id 1777143709725-d2wtr1):

[FATF-Listing] FATF https://...: direct failed (HTTP 403), retrying via proxy
[FATF-Listing] Retry 1/3 in 1000ms: HTTP 403   ← proxy ALSO 403
[FATF-Listing] Retry 2/3 in 2000ms: HTTP 403
[FATF-Listing] Retry 3/3 in 4000ms: HTTP 403
[FATF-Listing] FETCH FAILED: HTTP 403

I confirmed today (2026-04-25) that even with full Chrome browser headers (Accept-Language, Referer, Sec-Fetch-Dest/Mode/Site, Upgrade-Insecure-Requests) the request still returns the Cloudflare challenge HTML (<title>Just a moment...</title>). Header tweaks won't help.

Fix

Add a third tier to fetchHtml: direct → CONNECT proxy → Wayback Machine. The first two tiers are unchanged; Wayback runs only when both fail.

Why Wayback works

  • Different infrastructure: Wayback's crawler captures via paths Cloudflare doesn't gate the same way.
  • I confirmed the CDX API has multiple statuscode:200 snapshots in 2026 (most recent 2026-04-03 14:49 UTC, with current Feb-2026-plenary content — i.e. today's actual real-world FATF state).
  • Capture lag (1–3 days) is irrelevant given FATF publishes 3×/year and the seeder's bundle interval is 30 days, with a 42-day STALE_SEED threshold.

Why the id_ modifier

The implementation fetches https://web.archive.org/web/<timestamp>id_/<url>. The id_ modifier returns the original captured HTML byte-for-byte without Wayback's banner injection or href/src rewriting — keeping the existing parser (findPublicationLink, extractListedCountries) unchanged. A regression test pins the modifier so a future "cleanup" can't silently switch to the bare /web/timestamp/url form (which would prepend ~3KB of Wayback toolbar HTML and rewrite every link, breaking the parser).

Self-healing direct path

The order is direct → proxy → Wayback, not Wayback-first. The moment FATF rotates off Cloudflare or Decodo egress IPs get whitelisted, the seeder transparently goes back to live data with no code change.

Tests

+7 cases on fetchViaWayback in tests/seed-fatf-listing.test.mjs:

  • Happy path: queries CDX, fetches latest 200 snapshot via id_
  • CDX URL shape: filter=statuscode:200, from=YYYYMMDD, output=json
  • No-200-snapshots window → throws clear error
  • CDX HTTP 5xx → throws
  • Snapshot 4xx (Wayback re-fetched a Cloudflare challenge) → throws
  • Malformed CDX timestamp → throws (defends against schema drift)
  • id_ modifier pin (regression guard against parser-breaking cleanup)

Test plan

  • npm run typecheck + typecheck:api
  • CJS syntax check
  • npm run lint (no errors)
  • npm run test:data (7194 pass)
  • Edge bundle check
  • node --test tests/edge-functions.test.mjs (177 pass)
  • npm run lint:md
  • npm run version:check
  • After merge + Railway redeploy, watch for [FATF-Listing] log line that does NOT end in FETCH FAILED: HTTP 403. Wayback fallback kicks in after the proxy 403, parses the snapshot HTML, and runSeed writes economic:fatf-listing:v1.
  • Confirm /api/health fatfListing.status flips from EMPTY to OK (or STALE_SEED clearing once seed-meta is written).

Out of scope

  • BIS-LBS EMPTY status (separate root cause: lock contention / validate floor / bundle timeout). Will be a separate PR adding a SIGTERM lock-release handler defensively.

…tches

FATF's www.fatf-gafi.org enforces a Cloudflare "Just a moment…" JS
challenge that blocks both direct Node fetches and CONNECT-proxy
(Decodo) requests with HTTP 403. PR #3407's seeder consequently writes
nothing on every Railway tick — economic:fatf-listing:v1 shows
status=EMPTY in /api/health because the key is never published.

Direct probe today (2026-04-25): direct fetch with full Chrome browser
headers (Accept-Language, Referer, Sec-Fetch-*) still 403s from a
residential IP. The challenge requires JS execution, which neither
Node fetch nor httpsProxyFetchRaw provides. Header tweaks won't help.

Wayback Machine bypasses Cloudflare entirely (its crawler captures
through different infrastructure). CDX API has multiple status:200
snapshots in 2026 (most recent 2026-04-03 with current Feb-2026-plenary
content); FATF's 3x/year publishing cadence vs the seeder's 30-day
bundle interval and 42-day STALE_SEED threshold means Wayback's 1-3
day capture lag is irrelevant.

Three-tier fetch: direct -> CONNECT proxy -> Wayback. The first two
are preserved unchanged so this seeder transparently recovers the
moment FATF rotates off Cloudflare or whitelists Decodo egress.

Wayback fetch uses the id_ URL modifier (/web/<timestamp>id_/<url>),
which returns the original captured HTML byte-for-byte WITHOUT
Wayback's banner injection or href/src rewriting. The seeder's
existing parser (findPublicationLink, extractListedCountries) works
unchanged. A regression test pins the modifier to defend against
accidental cleanup that switches to the bare /web/timestamp/url form.

Tests: +7 cases on fetchViaWayback covering happy path, CDX/snapshot
HTTP errors, no-200-snapshots window, malformed timestamps, the id_
modifier pin, and the from-date floor in the CDX query.

Bundle interval, validate, and seed-meta wiring unchanged.
@vercel

vercel Bot commented Apr 25, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated (UTC)
worldmonitor Ignored Ignored Preview Apr 25, 2026 8:05pm

Request Review

@greptile-apps

greptile-apps Bot commented Apr 25, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR adds a Wayback Machine third-tier fallback to fetchHtml so the FATF seeder can bypass Cloudflare's JS challenge, along with 7 targeted tests for the new fetchViaWayback function. The approach (direct → proxy → Wayback CDX id_ snapshot) is well-reasoned and the id_ modifier choice is correctly justified.

  • P1 — stale snapshot selection: fetchViaWayback queries CDX with limit=20 and ascending order, then takes rows[rows.length - 1]. CDX with a positive limit returns the N oldest results, so if Wayback has captured the FATF page more than 20 times in the 180-day window (common for major sites), the code silently uses the 20th-oldest snapshot rather than the most recent one — potentially missing the April 2026 capture cited in the PR description. Fix: change limit=20 to limit=-1 (CDX shorthand for "most recent matching snapshot").

Confidence Score: 3/5

Not safe to merge as-is — the Wayback fallback may silently serve a months-old snapshot instead of the most recent one.

One P1 bug: limit=20 with ascending CDX order causes the code to pick the 20th-oldest snapshot rather than the most recent when more than 20 captures exist in the window, directly undermining the freshness goal of the fallback. The fix is a one-character change but until applied the seeder could seed months-stale data while logging no error.

scripts/seed-fatf-listing.mjs — specifically the cdxUrl construction on line 84.

Important Files Changed

Filename Overview
scripts/seed-fatf-listing.mjs Adds fetchViaWayback as a third-tier fallback in fetchHtml; P1 bug: limit=20 with ascending CDX order returns 20 oldest snapshots, not the most recent, so the seeder may use a stale snapshot when >20 captures exist in the 180-day window.
tests/seed-fatf-listing.test.mjs Adds 7 well-structured test cases for fetchViaWayback covering happy path, CDX URL shape, empty window, CDX 5xx, snapshot 4xx, malformed timestamp, and id_ modifier pin; mock data uses ≤3 timestamps so the limit=20 overflow bug is not caught.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[fetchHtml called] --> B[Tier 1: Direct fetch\nwith Chrome UA]
    B -->|ok| Z[Return HTML]
    B -->|fail| C{_proxyAuth\nconfigured?}
    C -->|yes| D[Tier 2: CONNECT proxy\nhttpsProxyFetchRaw]
    C -->|no| E[Log: no proxy,\nfall back to Wayback]
    D -->|ok| Z
    D -->|fail| E
    E --> F[Tier 3: fetchViaWayback]
    F --> G[CDX API query\nlimit=20 ascending ⚠️\nfilter=statuscode:200\nfrom=180d ago]
    G -->|ok 200| H[Pick rows at last index\n⚠️ may be 20th-oldest\nnot most recent]
    G -->|non-200| ERR[Throw CDX error]
    H --> I[Fetch snapshot via id_ modifier\nno Wayback toolbar injection]
    I -->|ok| Z
    I -->|non-200| ERR2[Throw snapshot error]
    ERR --> FINAL[Throw combined error\ndirect + proxy + wayback msgs]
    ERR2 --> FINAL
Loading

Reviews (1): Last reviewed commit: "fix(fatf-listing): Wayback Machine fallb..." | Re-trigger Greptile

Comment thread scripts/seed-fatf-listing.mjs Outdated
.toISOString()
.slice(0, 10)
.replace(/-/g, '');
const cdxUrl = `${WAYBACK_CDX_URL}?url=${encodeURIComponent(url)}&filter=statuscode:200&output=json&from=${fromDate}&limit=20`;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 limit=20 returns 20 oldest snapshots, not the 20 most recent

The CDX API returns results in ascending timestamp order by default, so limit=20 gives the 20 oldest snapshots since fromDate. rows[rows.length - 1] is therefore the 20th-oldest, not the most recent. The code comment says "the LAST row is most recent" — this is only true when there are ≤ 20 total captures in the window. The Wayback Machine crawls major sites like FATF's multiple times per week; over 180 days that easily exceeds 20 captures, meaning the code would silently use a months-old snapshot and miss the 2026-04-03 capture cited in the PR description.

Use limit=-1 (CDX shorthand for "return the single most recent match") to guarantee the latest snapshot is selected:

Suggested change
const cdxUrl = `${WAYBACK_CDX_URL}?url=${encodeURIComponent(url)}&filter=statuscode:200&output=json&from=${fromDate}&limit=20`;
const cdxUrl = `${WAYBACK_CDX_URL}?url=${encodeURIComponent(url)}&filter=statuscode:200&output=json&from=${fromDate}&limit=-1`;

The rest of the function (rows.length < 2 guard, rows[rows.length - 1]) then works correctly with 1 returned data row.

Suggested change
const cdxUrl = `${WAYBACK_CDX_URL}?url=${encodeURIComponent(url)}&filter=statuscode:200&output=json&from=${fromDate}&limit=20`;
const cdxUrl = `${WAYBACK_CDX_URL}?url=${encodeURIComponent(url)}&filter=statuscode:200&output=json&from=${fromDate}&limit=-1`;

Comment thread scripts/seed-fatf-listing.mjs Outdated
Comment on lines +85 to +100
const cdxResp = await fetchFn(cdxUrl, { signal: AbortSignal.timeout(20_000) });
if (!cdxResp.ok) throw new Error(`Wayback CDX HTTP ${cdxResp.status} for ${url}`);
const rows = await cdxResp.json();
// CDX returns: [headerRow, ...snapshotRows]. Each snapshot row =
// [urlkey, timestamp, original, mimetype, statuscode, digest, length].
// Snapshots are timestamp-ascending, so the LAST row is most recent.
if (!Array.isArray(rows) || rows.length < 2) {
throw new Error(`Wayback has no status-200 snapshots for ${url} since ${fromDate}`);
}
const latest = rows[rows.length - 1];
const timestamp = latest[1];
if (!/^\d{14}$/.test(timestamp)) {
throw new Error(`Wayback CDX returned malformed timestamp "${timestamp}" for ${url}`);
}
const snapshotUrl = `https://web.archive.org/web/${timestamp}id_/${url}`;
const snapResp = await fetchFn(snapshotUrl, { signal: AbortSignal.timeout(30_000) });

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Missing User-Agent on Wayback fetches

AGENTS.md lists "Always include User-Agent header in server-side fetch calls" as a critical convention. Both the CDX request and the snapshot request omit the User-Agent header. While archive.org is unlikely to block headless requests, this is inconsistent with the rest of the seeder (which passes CHROME_UA to the FATF direct fetch) and with the repo convention.

Suggested change
const cdxResp = await fetchFn(cdxUrl, { signal: AbortSignal.timeout(20_000) });
if (!cdxResp.ok) throw new Error(`Wayback CDX HTTP ${cdxResp.status} for ${url}`);
const rows = await cdxResp.json();
// CDX returns: [headerRow, ...snapshotRows]. Each snapshot row =
// [urlkey, timestamp, original, mimetype, statuscode, digest, length].
// Snapshots are timestamp-ascending, so the LAST row is most recent.
if (!Array.isArray(rows) || rows.length < 2) {
throw new Error(`Wayback has no status-200 snapshots for ${url} since ${fromDate}`);
}
const latest = rows[rows.length - 1];
const timestamp = latest[1];
if (!/^\d{14}$/.test(timestamp)) {
throw new Error(`Wayback CDX returned malformed timestamp "${timestamp}" for ${url}`);
}
const snapshotUrl = `https://web.archive.org/web/${timestamp}id_/${url}`;
const snapResp = await fetchFn(snapshotUrl, { signal: AbortSignal.timeout(30_000) });
const cdxResp = await fetchFn(cdxUrl, { headers: { 'User-Agent': CHROME_UA }, signal: AbortSignal.timeout(20_000) });

And similarly for the snapshot fetch on line 100.

PR #3413 review: with limit=20 the CDX query returned the OLDEST 20
captures within the from-date window (CDX default sort is timestamp-
ascending, positive `limit=N` = first N rows = oldest N). Picking
rows[length-1] then gave us the 20th-oldest capture, not the truly
most-recent. FATF accumulates well over 20 captures per 180-day
window so this would silently serve a stale archived list while a
newer snapshot existed — breaking the whole "latest FATF state via
Wayback" guarantee.

CDX accepts negative `limit` to mean "last N captures by timestamp".
`limit=-1` returns just the single most-recent capture, which is
exactly what we need and also avoids fetching ~20× more rows than we
use. Picking logic (rows[length-1]) is unchanged — with limit=-1 the
response is exactly [headerRow, mostRecentSnapshot] so length-1 still
points at the right row.

Test additions:
- Pin `limit=-1` in the URL-shape test (regression guard).
- Negative-assertion that no positive `limit=N` is present (defense
  against a future cleanup that "tidies up" the syntax and silently
  reverts to the bug).
@koala73

koala73 commented Apr 25, 2026

Copy link
Copy Markdown
Owner Author

Reviewer was right — verified the bug against the live CDX API. With limit=20 (or any positive limit) CDX returns the OLDEST N captures within the from-date window (default sort is timestamp-ASCENDING). Picking rows[length-1] then gave the 20th-oldest, not the truly most-recent. FATF accumulates well over 20 captures per 180-day window so this would silently serve a stale archived list while a newer snapshot existed.

Fix in ab4797adc: limit=20limit=-1. CDX's negative-limit semantics return the single most-recent capture (last N rows by timestamp). Picking logic (rows[length-1]) is unchanged — with limit=-1 the response is exactly [headerRow, mostRecentSnapshot] so length-1 still points at the right row.

Tests:

  • Existing URL-shape assertion now also pins limit=-1 (regression guard against re-introducing the positive form).
  • Added negative-assertion: assert.doesNotMatch(cdxUrl, /[?&]limit=(?!-)\d+/) — explicit guard so a future "tidy-up" can't silently revert.

Full /newpr verification re-run, all 8 checks green; test:data 7194 passing.

Greptile P2 review finding on PR #3413. AGENTS.md:185 mandates
"Always include User-Agent header in server-side fetch calls".
The direct FATF fetch passes CHROME_UA but the Wayback fallback's
two fetches (CDX query + snapshot) omitted it, breaking convention.

archive.org doesn't currently block UA-less requests, but: (a)
house-style consistency is the point, (b) a future Wayback rate
limiter could reasonably enforce UA, (c) the direct fetch the
seeder makes to fatf-gafi.org sets CHROME_UA, so a Wayback
fallback that swaps out the UA is unexpected behavior.

Test: +1 case asserts both calls receive a User-Agent header
matching the canonical CHROME_UA pattern (Mozilla/5.0...). Pinned
strongly enough to catch a regression that drops the header OR
substitutes a placeholder/empty value.

Also addresses Greptile P1 (limit=20 oldest-not-newest), already
fixed in commit ab4797a -- Greptile's last-reviewed commit was
the pre-fix a4f254e.
@koala73

koala73 commented Apr 25, 2026

Copy link
Copy Markdown
Owner Author

Greptile review responses:

P1 — limit=20 returns 20 oldest snapshots — already fixed in commit ab4797a (pushed yesterday). Greptile's last-reviewed commit was the pre-fix a4f254e; a re-trigger should pick up the fix.

P2 — Missing User-Agent on Wayback fetches — fixed in commit f74be5e (just pushed). AGENTS.md:185 mandates 'Always include User-Agent header in server-side fetch calls'. Both the CDX query and the snapshot fetch now pass CHROME_UA (matches the canonical UA the direct FATF fetch already uses).

Test added: 'sends a User-Agent header on BOTH the CDX query and the snapshot fetch (AGENTS.md convention)'. Asserts both calls receive a non-empty UA matching /Mozilla/5.0/ — strong enough to catch a regression that drops the header OR substitutes a placeholder.

Full /newpr verification re-run on f74be5e: 7195 tests passing (up from 7194), all 8 checks green.

@koala73
koala73 merged commit 6586dce into main Apr 25, 2026
10 checks passed
@koala73
koala73 deleted the fix/fatf-wayback-fallback branch April 25, 2026 20:27
koala73 added a commit that referenced this pull request Apr 25, 2026
…from Railway (#3415)

* fix(fatf-listing): proxy-fallback + describeErr for Wayback failures from Railway

Production observation 2026-04-25T20:35 (post PR #3413 merge): the
Wayback fallback path itself fails on every Railway tick. Direct CDX
query times out at 20s, subsequent retries surface as "fetch failed"
with no cause unwrapped. Local desktop probes complete in <2s, so the
issue is Railway-egress-specific — the IP pool gets soft-rate-limited
or routed slowly to archive.org.

Pre-fix log:

  wayback=The operation was aborted due to timeout
  wayback=fetch failed
  wayback=fetch failed

The bare "fetch failed" comes from undici's default error message and
gives operators no actionable signal.

Three changes:

1. CDX timeout 20s -> 45s. Generous ceiling that still keeps the
   seeder under bundle-runner's 120s timeoutMs and accommodates the
   added proxy-fallback hops.

2. Two-tier per-call inside fetchViaWayback: direct fetch first
   (fast when egress isn't rate-limited), CONNECT proxy fallback
   (Decodo's residential pool isn't bucketed alongside Railway IPs at
   archive.org). Same shape as the seeder's outer fetchHtml three-tier.
   Applied to BOTH the CDX query AND the snapshot fetch.

3. describeErr helper (mirrors seed-unrest-events.mjs:162) unwraps
   err.cause so production logs surface the actual failure mode
   (DNS / TCP reset / TLS abort) instead of undici's bare "fetch
   failed".

Tests: +2 cases on fetchViaWayback (proxy-fallback happy path,
err.cause unwrapping when both tiers fail). Updated 2 existing tests
to pass `proxyAuth: null` so they stay focused on the direct-only
failure mode the new fallback would otherwise mask.

* fix(fatf-listing): bound per-URL fetch budget to fit FATF-Listing section timeoutMs

Reviewer-found gap on PR #3415. The proxy-fallback addition created a
budget overflow: direct(30s) + proxy(30s) + wayback-CDX-direct(45s) +
wayback-CDX-proxy(45s) + wayback-snapshot-direct(45s) + wayback-
snapshot-proxy(45s) = 240s per URL. FATF fetches 3 URLs (entry
sequential, black/grey parallel) → ≤480s end-to-end worst case while
seed-bundle-macro.mjs's FATF-Listing section was capped at 120_000ms.
That meant bundle-runner would SIGTERM mid-fetch instead of letting
runSeed reach its graceful "Failed gracefully" path — turning a
recoverable fetch failure into a timeout-driven section failure.

Three-part fix:

1. Tighten per-tier timeouts so worst-case-per-URL fits in 125s:
   - Direct fetchHtml: 30s → 10s (Cloudflare 403s in <1s when blocking)
   - Proxy fetchHtml: 30s → 15s (Decodo CONNECT overhead is ~1s)
   - Wayback per-tier (CDX + snapshot, direct + proxy): 45s → 25s
     (still 25% margin over Railway's observed 20s+ rate-limit window)

2. Bump seed-bundle-macro.mjs FATF-Listing section timeoutMs:
   120_000 → 300_000. Matches peer sections (BIS-Data, Eurostat, etc.)
   and accommodates worst-case ≤250s end-to-end with ~50s margin.

3. Static-shape regression test that pins all four numeric values
   (10_000 / 15_000 / WAYBACK_TIMEOUT_MS = 25_000 / section 300_000)
   so a future cleanup can't silently regress the budget.

Math after fix:
  Per URL: 10 + 15 + 25 + 25 + 25 + 25 = 125s
  Sequential entry + parallel black/grey: ≤ 250s
  Section budget: 300s (50s margin)

* refactor(seed-utils): hoist describeErr + clarify proxy-path UA

Greptile P2 findings on PR #3415:

- describeErr was duplicated byte-for-byte in seed-fatf-listing.mjs and
  seed-unrest-events.mjs. Future improvements (AggregateError, multi-
  level cause chains) would need to be applied in both. Hoisted to
  scripts/_seed-utils.mjs alongside CHROME_UA, resolveProxyForConnect,
  httpsProxyFetchRaw — both seeders now import the canonical helper.

- Proxy CDX/snapshot path in fetchViaWayback didn't visibly pass a
  User-Agent header. httpsProxyFetchRaw DOES inject CHROME_UA internally
  (scripts/_seed-utils.mjs line 174), so the AGENTS.md UA convention is
  already satisfied — but the call site doesn't make that obvious.
  Added inline comments at both proxy-fetcher call sites pointing at
  the implementation, so a future reader doesn't think the proxy path
  is header-less and 'fix' it by passing redundant headers.

Other Greptile P2s on PR #3415 are stale (last-reviewed commit was
ba863d3, before the budget commit 69e944b addressed them):
- 'Worst-case 150s > 120s budget' → fixed in 69e944b (10/15/25*4 =
  125s/URL; section bumped 120k→300k).
- 'Placeholder PR #34xx in comment' → that comment was rewritten in
  69e944b and no longer references a placeholder.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant