fix(seed-portwatch): surface degraded ArcGIS 400 body + cap degenerate retries by koala73 · Pull Request #3701 · koala73/worldmonitor

koala73 · 2026-05-15T05:12:06Z

Summary

ArcGIS Daily_Ports_Data, during degradation episodes, returns HTTP 200 with a 400 error body (Cannot perform query. Invalid query parameters.) after 30-56s of server processing. Our 45s FETCH_TIMEOUT fires before the body lands → caller sees AbortError → the existing circuit-breaker that pattern-matches the error message never fires → we treat every degraded country as a generic timeout → proxy retry against the same degraded upstream → also fails → fetchWithRetryOnInvalidParams burns another full 45s on a doomed direct retry.

Investigation that drove this (WM 2026-05-15): direct curl probes against services9.arcgis.com revealed (a) HTTP 200 with a 400 error body taking 30-56s to return, (b) non-deterministic results across attempts within minutes — same query, ERR → OK → ERR → 504, (c) WHERE-clause reshaping helps unevenly but never reliably. Right frame: upstream is degraded, surface it and protect the container budget, not "we're being rate-limited."

Two surgical fixes:

Diagnostic body-capture on timeout — when direct fetch hits its FETCH_TIMEOUT (not a caller-signal abort), make ONE extra fetch with +20s budget purely to read the response body. If it contains body.error, throw with the real ArcGIS message so the upstream circuit-breaker can act on the upstream-degradation signal. Gated to fire once per process — the proxy budget Greptile P2'd on fix(seed-portwatch): retry via Decodo proxy on TIMEOUT (not just HTTP 429) #3681 stays intact for subsequent timeouts.
Bail-don't-retry threshold on "Invalid query parameters" — preserves the legit one-shot retry for transient single-call flakes (2026-04-20 BRA/IDN/NGA pattern), but caps total retries-on-this-error per process at 5. Beyond that, throws a clear ArcGIS degraded — N errors message so the operator sees the regression class instead of N identical retries × 45s burning the 540s container budget.

The existing cap-mode + stale-serve + temp-gate-25 logic (PRs #3676/#3681/#3694) already rides through partial coverage — these changes just make sure we don't waste the container budget while ArcGIS recovers, and that the next diagnostic round has the real error message instead of generic AbortError.

Test plan

npm run test:data — 8803/8803 pass (96/96 in the focused portwatch test file, was 93)
npm run typecheck + npm run typecheck:api — pass
npm run lint — no new diagnostics (235 warnings pre-existing on origin/main)
node --test tests/edge-functions.test.mjs — 185/185 pass
npm run version:check — pass
3 new structural pattern-match tests covering: ERROR_BODY_CAPTURE_EXTRA_MS=20_000, _captureErrorBodyAfterTimeout helper, once-per-run gating (_bodyCapturedOnceThisRun), INVALID_PARAMS_RETRY_THRESHOLD=5, counter increment + threshold check + _resetInvalidParamsErrorCount / _resetBodyCapturedFlag reset helpers
Watch UptimeRobot post-merge: should see either (a) the real "Invalid query parameters" message in seed-portwatch-port-activity logs once per run (capture fired) AND/OR ArcGIS degraded — N 'Invalid query parameters' errors in this run after 5 such errors, instead of the current generic per-country timeout cascade
Once ArcGIS recovers, expect the degradation message to disappear; if it persists, that's the signal to investigate further upstream (publisher contact, alternate dataset, etc.)

…e retries ArcGIS Daily_Ports_Data, during degradation episodes, returns HTTP 200 with a 400 error body (`Cannot perform query. Invalid query parameters.`) after 30-56s of server processing. Our 45s FETCH_TIMEOUT fires before the body lands, callers see AbortError, the existing circuit-breaker that pattern-matches the error message never fires, and the seeder treats every degraded country as a generic timeout (then a doomed proxy retry, then another doomed first-direct-retry from the one-shot fetchWithRetryOnInvalidParams). Two surgical fixes: 1. **Diagnostic body-capture on timeout.** When the direct fetch hits its FETCH_TIMEOUT (not a caller-signal abort), make ONE extra fetch with +20s budget purely to read the response body. If it contains `body.error`, throw with the real ArcGIS message so the upstream circuit-breaker can act on the upstream-degradation signal. Gated to fire once per process — the proxy budget Greptile P2'd on PR #3681 stays intact for subsequent timeouts. 2. **Bail-don't-retry threshold on Invalid-query-parameters.** Preserve the legit one-shot retry for transient single-call flakes (2026-04-20 BRA/IDN/NGA pattern), but cap total retries-on-this-error per process at 5. Beyond that, throw a clear `ArcGIS degraded — N errors` message so the operator sees the regression class instead of N identical retries × 45s burning the 540s container budget. Investigation that drove this: direct curl probes against services9.arcgis.com revealed (a) HTTP 200 with a 400 error body, (b) non-deterministic results across attempts within minutes (same query, ERR → OK → ERR → 504), (c) WHERE-clause reshaping helps unevenly but never reliably. The right frame is "upstream is degraded, surface it and protect the container budget" not "we're being rate-limited." The existing cap-mode + stale-serve + temp-gate-25 logic (PRs #3676/#3681/#3694) already rides through partial coverage — these changes just make sure we don't waste the container budget while ArcGIS recovers, and that the next diagnostic round has the real error message instead of generic AbortError. Tests: - 3 new structural pattern-match tests covering ERROR_BODY_CAPTURE_EXTRA_MS, _captureErrorBodyAfterTimeout helper, once-per-run gating, INVALID_PARAMS_RETRY_THRESHOLD, counter increment + threshold check, reset helpers. - 96/96 tests pass (was 93 before).

vercel · 2026-05-15T05:12:11Z

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
worldmonitor	Ignored	Preview	May 15, 2026 8:27am

greptile-apps · 2026-05-15T05:20:57Z

Greptile Summary

This PR adds two defensive mechanisms to seed-portwatch-port-activity.mjs to handle the ArcGIS degradation pattern where the server returns HTTP 200 with a {\"error\":{\"code\":400}} body after 30-56s — a response that the existing 45s FETCH_TIMEOUT consistently misses, leaving the upstream circuit-breaker blind to the real failure class.

Diagnostic body-capture on timeout (_captureErrorBodyAfterTimeout, ERROR_BODY_CAPTURE_EXTRA_MS=20_000, _bodyCapturedOnceThisRun gate): when the first direct fetch in a run times out, one additional re-fetch with a 20s budget is attempted to surface the actual ArcGIS error body; the gate ensures this extra cost is paid only once per process run.
Bail-don't-retry threshold (_invalidParamsErrorCount, INVALID_PARAMS_RETRY_THRESHOLD=5): fetchWithRetryOnInvalidParams now counts per-process occurrences and stops retrying once the count exceeds 5, emitting a clear \"ArcGIS degraded — N errors\" message. Three structural pattern-match tests cover the new constants, gate ordering, and threshold check.

Confidence Score: 4/5

Safe to merge; the changes are additive and well-bounded by the existing PER_COUNTRY_TIMEOUT_MS cap and the once-per-run gate.

The two new mechanisms are conservative: the body capture is best-effort with an explicit gate, and the retry threshold preserves existing transient-flake behavior for the first 5 occurrences before bailing. The main open question is whether the 20s re-fetch budget is realistic against an endpoint that takes 30-56s per query from connection start.

scripts/seed-portwatch-port-activity.mjs — specifically the interaction between ERROR_BODY_CAPTURE_EXTRA_MS and the ArcGIS server observed 30-56s response latency during degradation.

Important Files Changed

Filename	Overview
scripts/seed-portwatch-port-activity.mjs	Two new module-level mechanisms added: once-per-run diagnostic body-capture re-fetch and a per-process retry counter with bail threshold. Main concern is the 20s capture budget for a fresh request to an endpoint that requires 30-56s to respond during degradation.
tests/portwatch-port-activity-seed.test.mjs	Three new structural pattern-match tests added covering the new capture and threshold logic. No runtime mock-based tests added, consistent with the existing test style in this file.

Sequence Diagram

sequenceDiagram
    participant C as Caller
    participant R as fetchWithRetryOnInvalidParams
    participant F as fetchWithTimeout
    participant A as ArcGIS direct
    participant Cap as _captureErrorBodyAfterTimeout
    participant P as arcgisProxyRetry

    C->>R: call(url)
    R->>F: fetchWithTimeout(url)
    F->>A: fetch [45s FETCH_TIMEOUT]
    A-->>F: TimeoutError

    alt First timeout this run
        F->>F: "_bodyCapturedOnceThisRun = true"
        F->>Cap: capture(url, signal) [20s budget]
        alt ArcGIS responds with error body within 20s
            Cap-->>F: "{error: {code:400}}"
            F-->>R: throw ArcGIS error: Invalid query parameters
            R->>R: "_invalidParamsErrorCount += 1"
            alt "count <= 5"
                R->>F: retry fetchWithTimeout
                F->>P: proxy retry
                P-->>C: result or error
            else "count > 5"
                R-->>C: throw ArcGIS degraded - N errors
            end
        else Capture times out or returns valid body
            Cap-->>F: "null or {body}"
            F->>P: arcgisProxyRetry
            P-->>C: result or error
        end
    else Subsequent timeouts
        F->>P: arcgisProxyRetry direct
        P-->>R: result or error
        R->>R: count++ if Invalid query params
        R-->>C: result or bail
    end

_{Reviews (1): Last reviewed commit: "fix(seed-portwatch): surface degraded Ar..." | Re-trigger Greptile}

greptile-apps · 2026-05-15T05:21:00Z

+async function _captureErrorBodyAfterTimeout(url, signal) {
+  if (signal?.aborted) return null;
+  const captureSignal = signal
+    ? AbortSignal.any([signal, AbortSignal.timeout(ERROR_BODY_CAPTURE_EXTRA_MS)])
+    : AbortSignal.timeout(ERROR_BODY_CAPTURE_EXTRA_MS);
+  try {
+    const resp = await fetch(url, {
+      headers: { 'User-Agent': CHROME_UA, Accept: 'application/json' },
+      signal: captureSignal,
+    });
+    if (!resp.ok) return null;
+    const body = await resp.json();
+    if (body?.error) {
+      console.warn(`  [port-activity] degraded ArcGIS response captured: ${JSON.stringify(body.error).slice(0, 160)} (${url.slice(0, 80)})`);
+      return { error: body.error };
+    }
+    return { body };
+  } catch {
+    return null;
+  }


Capture re-fetch opens a fresh connection, so 20s budget may rarely succeed during consistent degradation

_captureErrorBodyAfterTimeout issues a brand-new request to ArcGIS — it doesn't read from the already-open stalled connection (which was aborted when combined fired). During the degradation episode the PR describes, ArcGIS needs 30-56s of server processing per query from connection start. A fresh 20s budget will also expire before the body lands whenever the server is consistently slow in that window. The capture will succeed only on requests where the non-deterministic server happens to respond faster on the second attempt.

This is acknowledged as best-effort and the fallback (proxy) is correct, but it means the diagnostic capture may silently produce null through an entire degraded run — and once the gate fires (on any outcome, including null), no later timeout in that run can try again. A longer budget (closer to FETCH_TIMEOUT) would capture more reliably at the cost of more time, but the once-per-run gate already limits the blast radius to a single country per run.

greptile-apps · 2026-05-15T05:21:01Z

    if (!/Invalid query parameters/i.test(msg)) throw err;
+    _invalidParamsErrorCount += 1;


When the proxy itself returns an "Invalid query parameters" error body, arcgisProxyRetry throws a message containing that phrase, which fetchWithRetryOnInvalidParams catches and counts, triggering a one-shot retry. That retry will timeout → proxy → same error again, costing a second full direct+proxy cycle. The intent of the original retry (clearing a transient direct-fetch flake) does not apply when the proxy has already confirmed the upstream error. A short guard before the increment would skip the wasted cycle.

Suggested change

if (!/Invalid query parameters/i.test(msg)) throw err;

_invalidParamsErrorCount += 1;

if (!/Invalid query parameters/i.test(msg)) throw err;

// Don't retry if the proxy already confirmed the error — the proxy

// response is the definitive upstream signal and a second cycle would

// just timeout → proxy → same error, burning the per-country budget.

if (/via proxy after/i.test(msg)) throw err;

_invalidParamsErrorCount += 1;

…backoff sleep (P1+P2) P1 — Cap-mode bypass could publish a stale-only canonical list as healthy. Pre-fix the capTriggered bypass of the 80% degradation guard was unconditional. With the lowered MIN_VALID_COUNTRIES=25 + cap-mode + servedStale entries seeding countryData, a run with all-stale entries (freshFetched=0, cacheHits=0, servedStale=27, dropped=147) could: - Pass validateFn (countryData.size >= 25 — all from stale-cache) - Bypass the 80% guard (capTriggered=true) - Shrink the canonical list from ~174 → 27 stale-only entries - Advance seed-meta, clearing the operator-facing WARNING Fix: gate the bypass on freshFetchedCount + cacheHitCount ≥ MIN_FRESH_FETCH_FOR_CAP_BYPASS (5). Below that floor, cap-mode is NOT rotational steady-state — it's an ArcGIS-completely-down scenario, and the run falls through to the 80% guard (which extendExistingTtl-only, preserves canonical list, keeps WARNING visible). New log line "CAP-MODE BYPASS REFUSED: only N fresh upstream contacts" makes the refusal observable to operators. This composes with the INVALID_PARAMS_RETRY_THRESHOLD=5 fix earlier in this PR — that change increases the rate at which cold fetches return errors during upstream degradation, making the stale-only scenario this P1 guards against MORE likely, not less. P2 — Inter-batch backoff sleep is now abort-aware. The 5s BATCH_BACKOFF_MS wait used a plain setTimeout that ignored the caller signal mid-sleep, so a SIGTERM during the wait still forced the full 5s before observing the abort. Now races the setTimeout against signal's abort event so the loop exits immediately on a real cancellation (the signal?.aborted check at the top of the next iteration then `break`s the loop). Cosmetic change but matches the PR-body intent that the loop is SIGTERM-responsive. Tests: - 2 new structural pattern-match tests covering MIN_FRESH_FETCH_FOR_CAP_BYPASS, freshFetchedCount counter, upstream-contact gate, CAP-MODE BYPASS REFUSED branch, and the abort-aware sleep shape. - 4 existing tests updated to match the new destructure shape + new prevCount × 0.8 reference count (now 4: 2 conditions + 2 Math.ceil sites, one set per guard branch). - 95/95 portwatch tests pass; 8804/8805 npm run test:data (1 flake is the pre-existing scripts/_bundle-fixture-env-*.mjs race between bundle-runner.test.mjs and news-classify-cache-prefix-audit.test.mjs, unrelated to this change).

…y-confirmed errors (P2×2) Two Greptile P2 comments on PR #3701 review round 2. P2 #1 — Body-capture gate fired on first ATTEMPT, not first SUCCESS. During consistent degradation, the 20s ERROR_BODY_CAPTURE_EXTRA_MS window isn't long enough — ArcGIS needs 30-56s of server processing per query from connection start. A failed first capture (capture also timing out) flipped the once-per-run flag, locking out every subsequent timing-out country from ever capturing the body. The diagnostic value could be lost for an entire run, defeating the whole point of the capture path. Fix: track SUCCESS count (cap at 1 — we only need the body shape once) AND ATTEMPT count (cap at 3 to bound total cost). Null captures consume an attempt but not a success, so subsequent timing-out countries keep trying. Worst-case cost: 3 × 20s wall-clock per run, still within PER_COUNTRY_TIMEOUT_MS=90s caps. P2 #2 — Proxy-confirmed "Invalid query parameters" still triggered retry. When arcgisProxyRetry returns an error body, the thrown message has the `(via proxy after ${reason})` prefix and still matches `/Invalid query parameters/i`. fetchWithRetryOnInvalidParams counted it, then retried via fetchWithTimeout — which would timeout → proxy → hit the same authoritative error, burning a full direct+proxy cycle for nothing. The proxy response IS the definitive upstream signal; retrying is meaningless. Fix: counter still increments (proxy-confirmed errors are valid degradation signals for the threshold), but short-circuit the retry with `if (/via proxy after/i.test(msg)) throw err`. Tests: 2 new structural pattern-match tests + 1 existing test relaxed to accept the new `if (captured?.error) { successCount++; throw ... }` shape. 96/96 portwatch tests pass; 8806/8806 npm run test:data (unrelated bundle-fixture race cleared this run).

…+P2 round 3) P1 — Proxy-confirmed Invalid Params never reached the degradation threshold. Pre-fix the proxy-confirmed short-circuit (`/via proxy after/i` → throw) ran BEFORE the threshold check. So during the actual incident — where nearly every error arrives via the proxy fallback path — every error threw the per-country proxy message and the clean `ArcGIS degraded — N 'Invalid query parameters' errors` message was unreachable. Fix: reorder so threshold check fires first, then proxy short-circuit. Counter still increments for proxy-confirmed errors (they contribute to the threshold), and once the threshold is exceeded the degraded message surfaces regardless of whether the error came from direct or proxy. P2 — 20s capture budget didn't realistically catch 30-56s responses. The 3-attempts-× 20s fix was bounded but each attempt was a FRESH request starting from t=0 (the original 45s direct fetch was already aborted). For the observed degraded response class (30-56s server-side from request start), each 20s capture aborts before the body lands. Fix: bump ERROR_BODY_CAPTURE_EXTRA_MS 20s → 40s. Math under PER_COUNTRY_TIMEOUT_MS=90s: direct (45s timeout) + capture (40s budget) = 85s, leaving 5s before the per-country wrap fires. Proxy retry effectively dies for timing-out countries in this mode — accepted tradeoff because the proxy is itself degraded (Decodo throttled per #3694 history), so it wasn't helping anyway. Attempt cap stays at 3 (across the run, not per country) so the diagnostic value is captured by 3rd timing-out country; subsequent countries go straight to proxy. Tests: 1 new ordering test using src.search() index comparison to assert threshold-throw-before-proxy-short-circuit; 2 existing tests updated for the bumped constant and the longer span between increment and short-circuit. 97/97 portwatch tests pass; 8806/8807 npm run test:data (1 flake = pre-existing _bundle-fixture-* race between bundle-runner and news-classify-cache-prefix-audit).

…real workload (#3711) * fix(seed-portwatch): rebalance direct/proxy budgets — proxy gets the real workload WM 2026-05-15 incident — actual root cause finally pinned: ArcGIS Daily_Ports_Data is currently responding to Decodo's proxy (gate.decodo.com) in 51-56s per query — sometimes with real data, sometimes with a 400 "Invalid query parameters" body (the degradation pattern PR #3701 captured). Direct fetch from Railway container IPs is fully throttled and never returns within 60s. Pre-fix budgets (FETCH_TIMEOUT=45s + PROXY_FETCH_TIMEOUT=35s) couldn't catch either response: direct timed out (Railway throttled), proxy timed out (under-budgeted for the 51-56s response class). Every cold fetch emerged as `proxy fetch timeout`, the COVERAGE GATE failed at 2 < 25 each run, and seed-meta froze. extendExistingTtl correctly preserved prior data — but the seeder produced zero fresh writes for 3+ days. Diagnostic that pinned this (today): - Laptop → ArcGIS direct: 6-17s success (residential IP, no throttle) - Laptop → Decodo gate.decodo.com → ArcGIS: 51-56s response, mixed data + 400 bodies (returns the slow-400 pattern PR #3701 anticipated) - Laptop → Decodo us.decodo.com → ArcGIS: HTTP/2 framing errors, empty replies — that pool doesn't work for ArcGIS data queries - Railway direct: 100% timeout at 45s (throttled tier) - Railway proxy: 100% timeout at 35s (under-budgeted for the 51-56s response that Decodo IS successfully tunneling) Rebalance the per-country 90s budget to where the work actually happens: FETCH_TIMEOUT 45_000 → 15_000 (fail fast; direct is useless from Railway right now) PROXY_FETCH_TIMEOUT 35_000 → 70_000 (catch ArcGIS's 51-56s response via Decodo gate pool) Total: 15 + 70 = 85s, within 90s PER_COUNTRY_TIMEOUT_MS with 5s slack Once proxy budget can actually capture the body, PR #3701's existing machinery does the rest: - arcgisProxyRetry parses body.error.message and throws an `ArcGIS error (via proxy after ...)` Error - fetchWithRetryOnInvalidParams's `/Invalid query parameters/i` regex matches that, increments the counter, surfaces the clean "ArcGIS degraded — N errors" message after 5 occurrences - Real-data countries (e.g. PAK 69 features in 52s observed) actually publish, freshFetchedCount ≥ 5 trips the cap-mode bypass earned path, seed-meta advances Disabled the diagnostic body-capture path (MAX_BODY_CAPTURE_ATTEMPTS: 3 → 0). It's redundant when proxy gets the body directly, and at FETCH_TIMEOUT=15s the +40s capture overhead would crowd out the proxy budget. Code path stays intact (gate code preserved) for future scenarios where direct-fetch DOES land close to a body. Tests: - 2 existing tests updated to assert the new FETCH_TIMEOUT=15s, PROXY_FETCH_TIMEOUT=70s, MAX_BODY_CAPTURE_ATTEMPTS=0 - Budget invariant test (direct + proxy < per-country - 5s slack) still passes: 15 + 70 = 85 < 90 - 5 = 85 ✓ - 97/97 portwatch-port-activity-seed tests pass - 8842/8842 npm run test:data pass * review: refresh stale call-site comment block in fetchWithTimeout (P2) Greptile PR #3711 P2: the inline comment at the body-capture call site still cited the pre-rebalance figures (3 attempts, 35s proxy budget, 45+20+35>90 scenario) that no longer match the new constants. Updated the block to: - Note the path is currently disabled via MAX_BODY_CAPTURE_ATTEMPTS=0 - Explain why (today's 15s direct + 70s proxy budget makes the proxy leg sufficient to receive the 51-56s body directly, so capture is redundant) - Point readers to the constant-level comment for the full rationale - Drop the specific 3 / 35s / 45+20+35>90 figures so the block stays accurate as the constants evolve Comment-only change. 97/97 portwatch tests pass. * review: cap preflight at 5s + skip proxy fallback (P1) + comment refresh (P3) P1 — Preflight could blow the container budget under ArcGIS degradation. fetchMaxDate runs once per eligible country (174) at PREFLIGHT_CONCURRENCY=24 BEFORE the cold-fetch cap. With the new budgets a preflight that fell back to proxy would cost 15s direct + 70s proxy = 85s per country; with 8 waves that's 680s — over the 570s container budget BEFORE any activity-phase work. Empirically preflight outStatistics is fast today (2.8s for 174 countries) because Railway IPs aren't yet throttled on small responses. But the protection is unprotected against the day ArcGIS shifts that behavior. Fix: parameterize fetchWithTimeout with `{ timeoutMs, noProxyFallback }` options. fetchMaxDate calls it directly (skipping the fetchWithRetryOnInvalidParams retry layer, since preflight is best-effort) with: - timeoutMs: PREFLIGHT_FETCH_TIMEOUT = 5_000 — tight direct budget - noProxyFallback: true — soft fail, no proxy Worst-case preflight wall-clock now ceil(174/24)=8 waves × 5s = 40s, well under the 570s container budget. Preflight failures fall through to the expensive per-country activity path which has the full direct+proxy budget. Companion change: 429s in preflight also throw fast instead of triggering proxy. P3 — Stale comments. Refreshed the arcgisProxyRetry header comment that still said "PROXY_FETCH_TIMEOUT (35s, shorter than 45s direct FETCH_TIMEOUT)" and the fetchWithTimeout catch-block header that still said "Only the internal 45s FETCH_TIMEOUT" so a top-down reader sees the current constants (15 direct + 70 proxy = 85 < 90 wrap) instead of the pre-PR-#3711 figures. The body-capture call-site comment was already refreshed in 014eb64. Tests: - 2 new structural pattern-match tests for PREFLIGHT_FETCH_TIMEOUT, parameterized fetchWithTimeout, noProxyFallback gates on timeout + 429, fetchMaxDate using the new options - 1 documentation-grade test enforcing preflight worst-case stays under 25% of the container budget - 1 existing test updated to assert `AbortSignal.timeout(timeoutMs)` (parameterized) instead of the hardcoded FETCH_TIMEOUT - 99/99 portwatch tests pass - 8844/8844 npm run test:data pass * review: refresh body-capture constant comment (P3) Greptile PR #3711 round 2 P3: ERROR_BODY_CAPTURE_EXTRA_MS comment still cited "Our 45s FETCH_TIMEOUT fires first" and "direct (45s timeout) + capture (40s budget) = 85s" — both stale figures from before the budget rebalance to FETCH_TIMEOUT=15s. Updated to: - Frame 40s as the budget SIZED FOR the original 45s direct timeout - Clarify the path is currently DISABLED (pointer to the MAX_BODY_CAPTURE_ATTEMPTS comment for rationale) - Explain that today's 15s+70s split makes capture redundant (proxy receives the body directly + arcgisProxyRetry throws a message the circuit-breaker matches) - Add a runbook note: any future re-enable should re-derive the budget against the FETCH_TIMEOUT in effect at that time, not assume 45+40 Comment-only change. 99/99 portwatch tests pass.

…p proxy short-circuit (#3760) * fix(seed-portwatch): bump direct timeout 15→30s, lower gate 20→5, drop proxy short-circuit WM 2026-05-18 — three composable changes after re-measuring ArcGIS behavior and analyzing the post-#3714 run logs: ═══════════════════════════════════════════════════════════════════ 1. FETCH_TIMEOUT 15s → 30s (PROXY_FETCH_TIMEOUT 70s → 50s) ═══════════════════════════════════════════════════════════════════ PR #3711's 15s direct timeout was sized for "Railway-direct never returns" (pre-fix evidence: 100% timeout at 45s). Today's live residential laptop probe on today's failing-country set shows ArcGIS actually responds in 1.8-28.4s: JPN 1.8s ✓ USA 8.0s ✓ KWT 17.3s ← killed by 15s CHN 3.9s ✓ POL 9.2s ✓ PAK 18.0s ← killed by 15s SLB 4.8s ✓ VIR 16.6s ← killed SEN 7.0s ✓ CMR 18.1s ← killed BHS 26.1s ← killed PRT 28.4s ← killed 6 of 12 fetches (50%) exceed 15s — each gets killed and routed through the slower degraded proxy, where many fail with Decodo CONNECT 522s. 30s covers ~95% of the observed range with PRT (28.4s) as the only edge case. Rebalanced PROXY_FETCH_TIMEOUT 70→50s to keep per-country budget under wrap: 30 + 50 = 80s ≤ 90s with 10s slack. ═══════════════════════════════════════════════════════════════════ 2. MIN_VALID_COUNTRIES 20 → 5 ═══════════════════════════════════════════════════════════════════ Three consecutive post-#3714 runs landed 6/8/10 fresh successes — all failed validation at gate=20, all triggered extendExistingTtl-only. Net effect: fresh successes were NEVER written to Redis, so the cap-mode rotation never accumulated and every run restarted from 0 cache-fresh (\`Cache: 0 hits, 174 misses\` every single run). Floor at 5 matches MIN_FRESH_FETCH_FOR_CAP_BYPASS (the load-bearing silent-loss safety from PR #3711). Cap-mode bypass still requires 5 fresh upstream contacts before publishing, so the "everything-stale" scenario still trips the 80% guard. The validateFn lower bound is the only thing changing; the silent-loss surface stays the same. ═══════════════════════════════════════════════════════════════════ 3. Remove "via proxy after" short-circuit in fetchWithRetryOnInvalidParams ═══════════════════════════════════════════════════════════════════ PR #3701 P2 added \`if (/via proxy after/i.test(msg)) throw err;\` assuming proxy responses were authoritative — same query via proxy twice would give same result. Live laptop probe disproved this: SLB attempt 1: OK 189 features (10.2s) SLB attempt 2: Proxy CONNECT: 522 (3.1s) SLB attempt 3: OK 189 features (8.6s) Non-deterministic upstream + Decodo gate flakiness means proxy retries DO recover. Today's logs show ~6 errors per run with "Invalid query parameters via proxy" on countries that succeed when I probe them right after — those would now recover via the retry. INVALID_PARAMS_RETRY_THRESHOLD=5 still caps the total per-run budget impact during full-degradation episodes. ═══════════════════════════════════════════════════════════════════ Tests: - 98/98 portwatch-port-activity-seed tests pass - typecheck clean - 3 existing tests updated for new constants (FETCH_TIMEOUT=30_000, PROXY_FETCH_TIMEOUT=50_000, MIN_VALID_COUNTRIES=5) - "proxy-confirmed Invalid query parameters short-circuits the retry" test INVERTED to assert the short-circuit is now ABSENT - "threshold check fires BEFORE proxy short-circuit" test REMOVED (the short-circuit it tested ordering against is gone) - Test:data shows 134 pre-existing failures in unrelated files (mcp-jmespath, brief-carousel) — confirmed against origin/main, not caused by this change. * review: decouple per-country writes from canonical advance (P1) + restore proxy short-circuit (P2) Greptile PR #3760 round 2 — both findings addressed. ═══════════════════════════════════════════════════════════════════ P1 — gate=5 alone could publish a 5-country canonical as healthy ═══════════════════════════════════════════════════════════════════ Pre-fix: lowering MIN_VALID_COUNTRIES to 5 let a 5-country fresh-fetch run pass validateFn, earn the cap-mode bypass, advance the canonical list, and clear the operator WARNING — exposing consumers to a 3% coverage canonical published as fresh. The reviewer also noted that at the observed 6-10/run success rate, the rotation may not reach full coverage before MAX_CACHE_AGE_MS (7d) ages out the early entries. Fix: decouple per-country writes from canonical/meta advance. Per-country writes (KEY_PREFIX${iso2}): Gate at MIN_VALID_COUNTRIES = 5 → cache-fresh rotation can accumulate even during slow recovery. CANONICAL_KEY + META_KEY: Gate at MIN_CANONICAL_PUBLISH = 50 (new constant) → operator-facing healthy signal only flips when coverage genuinely recovers. Below the canonical floor: extendExistingTtl preserves prior canonical + meta, and a new "PARTIAL PERSIST: N/50 below canonical-publish floor" log line surfaces what happened. Consumers reading the canonical list still see the prior 174 countries with their existing data (mostly stale for not-yet-rotated entries, fresh for the ones we just wrote). ═══════════════════════════════════════════════════════════════════ P2 — Proxy retry burns budget the per-country wrap won't honour ═══════════════════════════════════════════════════════════════════ Pre-fix: removed the `if (/via proxy after/i) throw err` short-circuit based on a standalone laptop probe showing proxy retries DO recover. But inside the seeder's 90s per-country wrap, by the time we hit this catch we've already used direct(30s) + proxy(50s) ≈ 80s. The retry's 500ms backoff + new direct+proxy can't fit in the ~10s remaining before the wrap aborts. Restored: keep the short-circuit. INVALID_PARAMS_RETRY_THRESHOLD still ticks for global visibility. ═══════════════════════════════════════════════════════════════════ Tests: - 99/99 portwatch-port-activity-seed tests pass - typecheck clean - "proxy-confirmed Invalid query parameters short-circuits the retry" test re-inverted to assert short-circuit is BACK - New test: canonical/seed-meta advance gated on MIN_CANONICAL_PUBLISH - MIN_VALID_COUNTRIES canary test updated (no longer asserts the literal "TEMPORARILY lowered" phrase — the structural anchors (Revert path + was 50 + MIN_CANONICAL_PUBLISH reference) carry the canary intent) * review: extend prevCountryKeys in PARTIAL PERSIST path so prior payloads don't expire (P1) Greptile PR #3760 round 3 P1: the new partial-persist branch I added in round 2 only extended CANONICAL_KEY and META_KEY: await extendExistingTtl([CANONICAL_KEY, META_KEY], TTL).catch(...) That left the per-country payload keys for UNTOUCHED countries (i.e. the ones not refreshed this run, still referenced by the preserved canonical list) at risk of TTL expiry (TTL=3d) during a multi-day partial recovery. Consumers reading CANONICAL would see the 174-country list but find many backing per-country keys missing — contradicting the inline-comment claim that they'd see 'the prior 174-country list with their existing data.' Fix mirrors the COVERAGE GATE / DEGRADATION GUARD failure paths: await extendExistingTtl([CANONICAL_KEY, META_KEY, ...prevCountryKeys], TTL).catch(...) The keys we DO write in this run get fresh SET ... EX TTL (in the `commands` pipeline below), so this only affects un-refreshed entries from the prior list. Test updated to assert prevCountryKeys is in the extend list. 99/99 portwatch tests pass; typecheck clean.

vercel Bot deployed to Preview May 15, 2026 05:15 View deployment

greptile-apps Bot reviewed May 15, 2026

View reviewed changes

koala73 added 3 commits May 15, 2026 09:38

koala73 merged commit f23f825 into main May 15, 2026
11 checks passed

koala73 mentioned this pull request May 15, 2026

fix(seed-portwatch): rebalance direct/proxy budgets — proxy gets the real workload #3711

Merged

5 tasks

koala73 mentioned this pull request May 17, 2026

fix(seed-portwatch): bump direct timeout 15→30s, lower gate 20→5, drop proxy short-circuit #3760

Merged

5 tasks

koala73 deleted the fix/portwatch-capture-degraded-arcgis-error-body branch June 25, 2026 14:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(seed-portwatch): surface degraded ArcGIS 400 body + cap degenerate retries#3701

fix(seed-portwatch): surface degraded ArcGIS 400 body + cap degenerate retries#3701
koala73 merged 4 commits into
mainfrom
fix/portwatch-capture-degraded-arcgis-error-body

koala73 commented May 15, 2026

Uh oh!

vercel Bot commented May 15, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot commented May 15, 2026

Uh oh!

greptile-apps Bot May 15, 2026

Uh oh!

greptile-apps Bot May 15, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

		if (!/Invalid query parameters/i.test(msg)) throw err;
		_invalidParamsErrorCount += 1;

Conversation

koala73 commented May 15, 2026

Summary

Test plan

Uh oh!

vercel Bot commented May 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

greptile-apps Bot commented May 15, 2026

Greptile Summary

Confidence Score: 4/5

Important Files Changed

Sequence Diagram

Uh oh!

greptile-apps Bot May 15, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot May 15, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

vercel Bot commented May 15, 2026 •

edited

Loading