fix(seed-portwatch): rebalance direct/proxy budgets — proxy gets the real workload#3711
Conversation
…real workload WM 2026-05-15 incident — actual root cause finally pinned: ArcGIS Daily_Ports_Data is currently responding to Decodo's proxy (gate.decodo.com) in 51-56s per query — sometimes with real data, sometimes with a 400 "Invalid query parameters" body (the degradation pattern PR #3701 captured). Direct fetch from Railway container IPs is fully throttled and never returns within 60s. Pre-fix budgets (FETCH_TIMEOUT=45s + PROXY_FETCH_TIMEOUT=35s) couldn't catch either response: direct timed out (Railway throttled), proxy timed out (under-budgeted for the 51-56s response class). Every cold fetch emerged as `proxy fetch timeout`, the COVERAGE GATE failed at 2 < 25 each run, and seed-meta froze. extendExistingTtl correctly preserved prior data — but the seeder produced zero fresh writes for 3+ days. Diagnostic that pinned this (today): - Laptop → ArcGIS direct: 6-17s success (residential IP, no throttle) - Laptop → Decodo gate.decodo.com → ArcGIS: 51-56s response, mixed data + 400 bodies (returns the slow-400 pattern PR #3701 anticipated) - Laptop → Decodo us.decodo.com → ArcGIS: HTTP/2 framing errors, empty replies — that pool doesn't work for ArcGIS data queries - Railway direct: 100% timeout at 45s (throttled tier) - Railway proxy: 100% timeout at 35s (under-budgeted for the 51-56s response that Decodo IS successfully tunneling) Rebalance the per-country 90s budget to where the work actually happens: FETCH_TIMEOUT 45_000 → 15_000 (fail fast; direct is useless from Railway right now) PROXY_FETCH_TIMEOUT 35_000 → 70_000 (catch ArcGIS's 51-56s response via Decodo gate pool) Total: 15 + 70 = 85s, within 90s PER_COUNTRY_TIMEOUT_MS with 5s slack Once proxy budget can actually capture the body, PR #3701's existing machinery does the rest: - arcgisProxyRetry parses body.error.message and throws an `ArcGIS error (via proxy after ...)` Error - fetchWithRetryOnInvalidParams's `/Invalid query parameters/i` regex matches that, increments the counter, surfaces the clean "ArcGIS degraded — N errors" message after 5 occurrences - Real-data countries (e.g. PAK 69 features in 52s observed) actually publish, freshFetchedCount ≥ 5 trips the cap-mode bypass earned path, seed-meta advances Disabled the diagnostic body-capture path (MAX_BODY_CAPTURE_ATTEMPTS: 3 → 0). It's redundant when proxy gets the body directly, and at FETCH_TIMEOUT=15s the +40s capture overhead would crowd out the proxy budget. Code path stays intact (gate code preserved) for future scenarios where direct-fetch DOES land close to a body. Tests: - 2 existing tests updated to assert the new FETCH_TIMEOUT=15s, PROXY_FETCH_TIMEOUT=70s, MAX_BODY_CAPTURE_ATTEMPTS=0 - Budget invariant test (direct + proxy < per-country - 5s slack) still passes: 15 + 70 = 85 < 90 - 5 = 85 ✓ - 97/97 portwatch-port-activity-seed tests pass - 8842/8842 npm run test:data pass
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
Greptile SummaryRebalances the seeder's fetch-budget constants after diagnosing that Railway-direct access to ArcGIS is fully throttled while the Decodo proxy path works but takes 51-56s — longer than the previous 35s proxy budget. The body-capture diagnostic path is also disabled (attempts set to 0) since the proxy now has enough headroom to receive the error body directly.
Confidence Score: 4/5Safe to merge; the budget arithmetic is sound and all test assertions are consistent with the new constants. The logic changes are correct and well-tested — budget arithmetic holds at 15+70=85s within the 90s per-country wrap. The only finding is a stale call-site comment block that still cites old values (3, 35s, 45+20+35>90) which no longer match the updated constants. The call-site comment block in scripts/seed-portwatch-port-activity.mjs around the body-capture gate (lines 244–254) has stale figures that should be updated. Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[fetchWithTimeout URL] -->|response within 15s| B{resp.ok?}
A -->|timeout / error after 15s| C{body capture gate - always false}
B -->|429| D[arcgisProxyRetry HTTP 429]
B -->|not ok| E[throw ArcGIS HTTP N]
B -->|ok + body.error| F[throw ArcGIS error]
B -->|ok| G[return body]
C -->|skipped| H[arcgisProxyRetry direct timeout PROXY_FETCH_TIMEOUT=70s]
H -->|proxy response within 70s| I{proxied.error?}
I -->|yes| J[throw ArcGIS error via proxy]
I -->|no| K[return proxied body]
J --> L[fetchWithRetryOnInvalidParams counter threshold circuit-breaker]
|
Greptile PR #3711 P2: the inline comment at the body-capture call site still cited the pre-rebalance figures (3 attempts, 35s proxy budget, 45+20+35>90 scenario) that no longer match the new constants. Updated the block to: - Note the path is currently disabled via MAX_BODY_CAPTURE_ATTEMPTS=0 - Explain why (today's 15s direct + 70s proxy budget makes the proxy leg sufficient to receive the 51-56s body directly, so capture is redundant) - Point readers to the constant-level comment for the full rationale - Drop the specific 3 / 35s / 45+20+35>90 figures so the block stays accurate as the constants evolve Comment-only change. 97/97 portwatch tests pass.
…esh (P3)
P1 — Preflight could blow the container budget under ArcGIS degradation.
fetchMaxDate runs once per eligible country (174) at
PREFLIGHT_CONCURRENCY=24 BEFORE the cold-fetch cap. With the new
budgets a preflight that fell back to proxy would cost 15s direct +
70s proxy = 85s per country; with 8 waves that's 680s — over the
570s container budget BEFORE any activity-phase work.
Empirically preflight outStatistics is fast today (2.8s for 174
countries) because Railway IPs aren't yet throttled on small responses.
But the protection is unprotected against the day ArcGIS shifts that
behavior.
Fix: parameterize fetchWithTimeout with `{ timeoutMs, noProxyFallback }`
options. fetchMaxDate calls it directly (skipping the
fetchWithRetryOnInvalidParams retry layer, since preflight is
best-effort) with:
- timeoutMs: PREFLIGHT_FETCH_TIMEOUT = 5_000 — tight direct budget
- noProxyFallback: true — soft fail, no proxy
Worst-case preflight wall-clock now ceil(174/24)=8 waves × 5s = 40s,
well under the 570s container budget. Preflight failures fall through
to the expensive per-country activity path which has the full
direct+proxy budget. Companion change: 429s in preflight also throw
fast instead of triggering proxy.
P3 — Stale comments.
Refreshed the arcgisProxyRetry header comment that still said
"PROXY_FETCH_TIMEOUT (35s, shorter than 45s direct FETCH_TIMEOUT)" and
the fetchWithTimeout catch-block header that still said "Only the
internal 45s FETCH_TIMEOUT" so a top-down reader sees the current
constants (15 direct + 70 proxy = 85 < 90 wrap) instead of the
pre-PR-#3711 figures. The body-capture call-site comment was already
refreshed in 014eb64.
Tests:
- 2 new structural pattern-match tests for PREFLIGHT_FETCH_TIMEOUT,
parameterized fetchWithTimeout, noProxyFallback gates on timeout +
429, fetchMaxDate using the new options
- 1 documentation-grade test enforcing preflight worst-case stays
under 25% of the container budget
- 1 existing test updated to assert `AbortSignal.timeout(timeoutMs)`
(parameterized) instead of the hardcoded FETCH_TIMEOUT
- 99/99 portwatch tests pass
- 8844/8844 npm run test:data pass
Greptile PR #3711 round 2 P3: ERROR_BODY_CAPTURE_EXTRA_MS comment still cited "Our 45s FETCH_TIMEOUT fires first" and "direct (45s timeout) + capture (40s budget) = 85s" — both stale figures from before the budget rebalance to FETCH_TIMEOUT=15s. Updated to: - Frame 40s as the budget SIZED FOR the original 45s direct timeout - Clarify the path is currently DISABLED (pointer to the MAX_BODY_CAPTURE_ATTEMPTS comment for rationale) - Explain that today's 15s+70s split makes capture redundant (proxy receives the body directly + arcgisProxyRetry throws a message the circuit-breaker matches) - Add a runbook note: any future re-enable should re-derive the budget against the FETCH_TIMEOUT in effect at that time, not assume 45+40 Comment-only change. 99/99 portwatch tests pass.
…ulates (#3714) * fix(seed-portwatch): lower temp gate 25→20 so cap-mode rotation accumulates WM 2026-05-16 observation (post-PR-#3711 first real-data run): - Cap-mode cold-fetched 30 countries, 22 succeeded (~73%), 8 errored with the real "Invalid query parameters" upstream signal - COVERAGE GATE failed at 22 < 25 → extendExistingTtl-only → 22 fresh successes were NOT written to Redis - Net effect: each run restarts from 0 cache-fresh, rotation never accumulates, full-coverage convergence never happens Root cause: the temp gate at 25 was set on 2026-05-14 when we had zero visibility into the actual success rate. PR #3711's budget rebalance now produces a stable ~22/30 success rate, just shy of the gate. Fix: lower the temp gate 25 → 20. A 22-success run now persists, the 22 fresh countries become cache-fresh (asof = upstream maxDate), next run skips them via the cacheHits path, and cap-mode picks a different 30 from the remaining 152. Projected convergence: Run 1: 22 fresh persisted Run 2: 22 + 22 = 44 cumulative ... Run 8: ~174 ≈ full coverage (~4 days at 12h cadence) Caveats noted in the comment: - If upstream maxDate advances between runs, all cached entries become stale (`Cache: 0 hits, 174 misses`) and rotation restarts from 0 - Revert path: bump to 30 when single-run success rate exceeds 90% (≥27/30), back to 50 once upstream stable enough for non-cap-mode runs. Target review: 2026-05-22 Tests: - 1 existing test updated: assert MIN_VALID_COUNTRIES=20, "TEMPORARILY lowered" + "Revert path" comment markers (slightly relaxed from the pre-existing specific "from 50" and "to 50" so future bumps don't require fresh test edits) - 99/99 portwatch tests pass - 8853/8853 npm run test:data pass * review: preserve 'was 50' baseline anchor in temp-gate canary test (P2) Greptile PR #3714 P2: my relaxation of the MIN_VALID_COUNTRIES canary test from /TEMPORARILY lowered from 50/ → /TEMPORARILY lowered/ removed the explicit anchor to the original permanent baseline (50). A future editor writing a vague 'TEMPORARILY lowered for now' could silently pass CI and lose the 'don't forget this is temporary' property. Re-tighten with one anchor to the historical baseline (per Greptile's suggested patch). The existing comment already says 'was 50 → 25 on 2026-05-14', so the test passes without further source changes. Test-only change. 99/99 portwatch tests pass.
…p proxy short-circuit (#3760) * fix(seed-portwatch): bump direct timeout 15→30s, lower gate 20→5, drop proxy short-circuit WM 2026-05-18 — three composable changes after re-measuring ArcGIS behavior and analyzing the post-#3714 run logs: ═══════════════════════════════════════════════════════════════════ 1. FETCH_TIMEOUT 15s → 30s (PROXY_FETCH_TIMEOUT 70s → 50s) ═══════════════════════════════════════════════════════════════════ PR #3711's 15s direct timeout was sized for "Railway-direct never returns" (pre-fix evidence: 100% timeout at 45s). Today's live residential laptop probe on today's failing-country set shows ArcGIS actually responds in 1.8-28.4s: JPN 1.8s ✓ USA 8.0s ✓ KWT 17.3s ← killed by 15s CHN 3.9s ✓ POL 9.2s ✓ PAK 18.0s ← killed by 15s SLB 4.8s ✓ VIR 16.6s ← killed SEN 7.0s ✓ CMR 18.1s ← killed BHS 26.1s ← killed PRT 28.4s ← killed 6 of 12 fetches (50%) exceed 15s — each gets killed and routed through the slower degraded proxy, where many fail with Decodo CONNECT 522s. 30s covers ~95% of the observed range with PRT (28.4s) as the only edge case. Rebalanced PROXY_FETCH_TIMEOUT 70→50s to keep per-country budget under wrap: 30 + 50 = 80s ≤ 90s with 10s slack. ═══════════════════════════════════════════════════════════════════ 2. MIN_VALID_COUNTRIES 20 → 5 ═══════════════════════════════════════════════════════════════════ Three consecutive post-#3714 runs landed 6/8/10 fresh successes — all failed validation at gate=20, all triggered extendExistingTtl-only. Net effect: fresh successes were NEVER written to Redis, so the cap-mode rotation never accumulated and every run restarted from 0 cache-fresh (\`Cache: 0 hits, 174 misses\` every single run). Floor at 5 matches MIN_FRESH_FETCH_FOR_CAP_BYPASS (the load-bearing silent-loss safety from PR #3711). Cap-mode bypass still requires 5 fresh upstream contacts before publishing, so the "everything-stale" scenario still trips the 80% guard. The validateFn lower bound is the only thing changing; the silent-loss surface stays the same. ═══════════════════════════════════════════════════════════════════ 3. Remove "via proxy after" short-circuit in fetchWithRetryOnInvalidParams ═══════════════════════════════════════════════════════════════════ PR #3701 P2 added \`if (/via proxy after/i.test(msg)) throw err;\` assuming proxy responses were authoritative — same query via proxy twice would give same result. Live laptop probe disproved this: SLB attempt 1: OK 189 features (10.2s) SLB attempt 2: Proxy CONNECT: 522 (3.1s) SLB attempt 3: OK 189 features (8.6s) Non-deterministic upstream + Decodo gate flakiness means proxy retries DO recover. Today's logs show ~6 errors per run with "Invalid query parameters via proxy" on countries that succeed when I probe them right after — those would now recover via the retry. INVALID_PARAMS_RETRY_THRESHOLD=5 still caps the total per-run budget impact during full-degradation episodes. ═══════════════════════════════════════════════════════════════════ Tests: - 98/98 portwatch-port-activity-seed tests pass - typecheck clean - 3 existing tests updated for new constants (FETCH_TIMEOUT=30_000, PROXY_FETCH_TIMEOUT=50_000, MIN_VALID_COUNTRIES=5) - "proxy-confirmed Invalid query parameters short-circuits the retry" test INVERTED to assert the short-circuit is now ABSENT - "threshold check fires BEFORE proxy short-circuit" test REMOVED (the short-circuit it tested ordering against is gone) - Test:data shows 134 pre-existing failures in unrelated files (mcp-jmespath, brief-carousel) — confirmed against origin/main, not caused by this change. * review: decouple per-country writes from canonical advance (P1) + restore proxy short-circuit (P2) Greptile PR #3760 round 2 — both findings addressed. ═══════════════════════════════════════════════════════════════════ P1 — gate=5 alone could publish a 5-country canonical as healthy ═══════════════════════════════════════════════════════════════════ Pre-fix: lowering MIN_VALID_COUNTRIES to 5 let a 5-country fresh-fetch run pass validateFn, earn the cap-mode bypass, advance the canonical list, and clear the operator WARNING — exposing consumers to a 3% coverage canonical published as fresh. The reviewer also noted that at the observed 6-10/run success rate, the rotation may not reach full coverage before MAX_CACHE_AGE_MS (7d) ages out the early entries. Fix: decouple per-country writes from canonical/meta advance. Per-country writes (KEY_PREFIX${iso2}): Gate at MIN_VALID_COUNTRIES = 5 → cache-fresh rotation can accumulate even during slow recovery. CANONICAL_KEY + META_KEY: Gate at MIN_CANONICAL_PUBLISH = 50 (new constant) → operator-facing healthy signal only flips when coverage genuinely recovers. Below the canonical floor: extendExistingTtl preserves prior canonical + meta, and a new "PARTIAL PERSIST: N/50 below canonical-publish floor" log line surfaces what happened. Consumers reading the canonical list still see the prior 174 countries with their existing data (mostly stale for not-yet-rotated entries, fresh for the ones we just wrote). ═══════════════════════════════════════════════════════════════════ P2 — Proxy retry burns budget the per-country wrap won't honour ═══════════════════════════════════════════════════════════════════ Pre-fix: removed the `if (/via proxy after/i) throw err` short-circuit based on a standalone laptop probe showing proxy retries DO recover. But inside the seeder's 90s per-country wrap, by the time we hit this catch we've already used direct(30s) + proxy(50s) ≈ 80s. The retry's 500ms backoff + new direct+proxy can't fit in the ~10s remaining before the wrap aborts. Restored: keep the short-circuit. INVALID_PARAMS_RETRY_THRESHOLD still ticks for global visibility. ═══════════════════════════════════════════════════════════════════ Tests: - 99/99 portwatch-port-activity-seed tests pass - typecheck clean - "proxy-confirmed Invalid query parameters short-circuits the retry" test re-inverted to assert short-circuit is BACK - New test: canonical/seed-meta advance gated on MIN_CANONICAL_PUBLISH - MIN_VALID_COUNTRIES canary test updated (no longer asserts the literal "TEMPORARILY lowered" phrase — the structural anchors (Revert path + was 50 + MIN_CANONICAL_PUBLISH reference) carry the canary intent) * review: extend prevCountryKeys in PARTIAL PERSIST path so prior payloads don't expire (P1) Greptile PR #3760 round 3 P1: the new partial-persist branch I added in round 2 only extended CANONICAL_KEY and META_KEY: await extendExistingTtl([CANONICAL_KEY, META_KEY], TTL).catch(...) That left the per-country payload keys for UNTOUCHED countries (i.e. the ones not refreshed this run, still referenced by the preserved canonical list) at risk of TTL expiry (TTL=3d) during a multi-day partial recovery. Consumers reading CANONICAL would see the 174-country list but find many backing per-country keys missing — contradicting the inline-comment claim that they'd see 'the prior 174-country list with their existing data.' Fix mirrors the COVERAGE GATE / DEGRADATION GUARD failure paths: await extendExistingTtl([CANONICAL_KEY, META_KEY, ...prevCountryKeys], TTL).catch(...) The keys we DO write in this run get fresh SET ... EX TTL (in the `commands` pipeline below), so this only affects un-refreshed entries from the prior list. Test updated to assert prevCountryKeys is in the extend list. 99/99 portwatch tests pass; typecheck clean.
Summary
ArcGIS isn't down, and Decodo isn't blocked. Direct fetch from Railway IPs is throttled (always times out), but the proxy via
gate.decodo.comreaches ArcGIS and returns the actual response in 51-56s. The seeder'sPROXY_FETCH_TIMEOUT=35swas killing the proxy retry before the response could land — so every cold fetch surfaced asproxy fetch timeouteven though the proxy was working.Rebalanced budgets so the proxy leg gets the budget the actual workload needs:
FETCH_TIMEOUT(direct)PROXY_FETCH_TIMEOUTPER_COUNTRY_TIMEOUT_MSMAX_BODY_CAPTURE_ATTEMPTSHow the diagnosis pinned it
Each layer disproved by direct probe (full session in the commit message):
us.decodo.comlike the Yahoo workaround" — that pool returns HTTP/2 framing errors / empty replies for any data-bearing querygate.decodo.com(the seeder's existing pool) WORKS — just takes 51-56s, longer than the 35s budgetWhat this unlocks
Once the proxy budget can actually capture the body, PR #3701's existing machinery does the rest:
arcgisProxyRetryparsesbody.error.messageand throwsArcGIS error (via proxy after ...)fetchWithRetryOnInvalidParams's/Invalid query parameters/iregex matches that → counter increments → after 5 errors the cleanArcGIS degraded — N errorslog fires (finally visible)freshFetchedCount ≥ 5trips the cap-mode bypass earned path → seed-meta advances → operator-facing WARNING clearsTest plan
npm run typecheck+npm run typecheck:apipassnpm run test:data— 8842/8842 pass (97/97 in the focused portwatch test file)degraded ArcGIS response captured(no — capture path disabled) ORarcgisProxyRetry's error throw ORArcGIS degraded — N 'Invalid query parameters' errors(yes, post-merge)MIN_VALID_COUNTRIESfrom 25 → 50 (target ~2026-05-20 per existing inline comment)