Skip to content

fix(seed-portwatch): rate-limit recovery — concurrency 12→6 + batch backoff + temp gate#3694

Merged
koala73 merged 3 commits into
mainfrom
fix/portwatch-rate-limit-recovery
May 14, 2026
Merged

fix(seed-portwatch): rate-limit recovery — concurrency 12→6 + batch backoff + temp gate#3694
koala73 merged 3 commits into
mainfrom
fix/portwatch-rate-limit-recovery

Conversation

@koala73

@koala73 koala73 commented May 14, 2026

Copy link
Copy Markdown
Owner

Summary

Option E from the 2026-05-14 incident triage: combined response to the ongoing ArcGIS degradation where both direct AND Decodo-proxy paths are now throttled. Three small, coordinated changes in one file.

Why all three together

Post-#3676 (cold-fetch cap) + post-#3681 (proxy fallback on timeout), the proxy retry IS firing but Decodo itself is now rate-limited by our usage:

Run #1 (19:55 UTC May 13):  Activity queue 30  →  24 proxy successes  →  gate fails 24<50
Run #2 (00:03 UTC May 14):  Activity queue 30  →   5 proxy successes  →  gate fails  5<50

Run-over-run degradation (24 → 5) means we're depleting Decodo's bucket faster than it refills. Three orthogonal mitigations needed:

Change Effect
B: CONCURRENCY 12 → 6 Halve in-flight pressure on both upstreams
C: BATCH_BACKOFF_MS = 5_000 (new) Inter-batch sleep gives both buckets time to refill
gate: MIN_VALID_COUNTRIES 50 → 25 (temporary) Partial-success runs advance seed-meta and clear alert

Math fits the budget

With CONCURRENCY=6 and cold-fetch cap 30:

  • 5 batches × ~60s realistic + 4 × 5s backoff = ~320s realistic
  • 5 batches × ~90s worst-case + 4 × 5s backoff = ~470s worst-case
  • Bundle budget: 570s
  • Headroom: 100s minimum

Why the temporary gate is OK

Pre-fix the seed-meta was frozen at 2026-05-12 00:03 UTC for 60+ hours, even though SOME successful fetches were landing each run. The gate-of-50 was sized for "normal ArcGIS"; with ArcGIS in this degraded state, no run can clear 50.

Lowering to 25 means a 5-success run STILL FAILS (correctly flags a real outage), but a 25+ partial-success run advances meta and clears the operator-facing WARNING. The remaining countries serve from cache (≤7d old per MAX_CACHE_AGE_MS).

I marked the change TEMPORARILY lowered from 50 → 25 … Revert to 50 once successive runs reliably exceed 50 again (target: 2026-05-20 review). Test asserts the comment shape so a future reviewer can't silently normalise 25 as the new permanent floor.

SIGTERM responsiveness preserved

The 5s backoff is gated on !signal?.aborted — if SIGTERM fires during a backoff sleep, the loop exits immediately without paying the full 5s.

Test plan

  • node --test tests/portwatch-port-activity-seed.test.mjs87/87 pass (was 85, +2 for new invariants, 1 modified for CONCURRENCY)
    • CONCURRENCY now pins at 6 (was 12)
    • BATCH_BACKOFF_MS pins at 5000 + gated on batchIdx < batches && !signal?.aborted
    • MIN_VALID_COUNTRIES pins at 25 + comment includes "TEMPORARILY lowered from 50" + "Revert to 50"
  • node --check scripts/seed-portwatch-port-activity.mjs — syntax OK
  • Post-merge: next cron run should produce 20-30 successful fetches (proxy + reduced concurrency), gate passes at 25, seed-meta advances, /api/health flips HEALTHY (or stays WARNING only if <25 succeed → real outage signal)

This week's portwatch series — three layers of resilience

PR Layer What it fixes
#3676 (merged) Bundle budget Cap cold-fetch at 30/run to prevent SIGTERM cliff
#3681 (merged) Network routing Retry timeouts via Decodo (not just 429s)
This PR Rate-limit gentleness Halve concurrency + space out batches + temporary partial-coverage acceptance

If after this PR + 1-2 runs the seeder reliably hits 25+ successes, we ride out the ArcGIS degradation gracefully. If it still can't hit 25, that's an even deeper upstream problem and we'd need to investigate ArcGIS-side directly (different access pattern, different proxy region, support ticket to IMF).

Followups (not in this PR)

  1. Revert MIN_VALID_COUNTRIES 25 → 50 when ArcGIS recovers. Tracking via the inline comment + the test-shape assertion.
  2. Investigate ArcGIS upstream — try a different proxy egress, check Decodo plan tier, contact IMF if needed.
  3. Telemetry on stale-served vs fresh-fetched in seed-meta so operators can distinguish "X countries fresh, Y on cache" from "Z countries fresh" — currently the seed-meta only counts total, not the freshness split.

…ackoff + temporary gate

Option E from the 2026-05-14 incident triage: combined response to the
ongoing ArcGIS degradation where both direct AND proxy paths are
throttled.

Background (post-#3676 + #3681 deployed):
  Run #1 (19:55 UTC May 13): 24/30 proxy successes, gate fails 24<50
  Run #2 (00:03 UTC May 14):  5/30 proxy successes, gate fails 5<50

Both #3676 (cold-fetch cap) and #3681 (proxy fallback on timeout) are
working as designed. The remaining problem: Decodo proxy is now also
being rate-limited — successive runs degrade as our usage spikes the
proxy's bucket. ArcGIS itself appears degraded for this dataset.

Three coordinated changes:

(B) CONCURRENCY 12 → 6

Halve in-flight fetches so neither ArcGIS-direct nor Decodo-proxy
sees a 12-concurrent burst. Math at cold-fetch cap 30:
  5 batches × ~60s realistic + 4×5s backoff ≈ 320s
  5 batches × ~90s worst case + 4×5s backoff ≈ 470s
Both fit the 570s bundle budget.

(C) BATCH_BACKOFF_MS = 5_000 (new)

Sleep 5s between batches (skip on last, skip on signal-abort to
preserve SIGTERM responsiveness). Spaces out per-batch bursts so
neither service hits its rate-limit window from our run alone.
20s total added — negligible against the 570s budget.

(Temporary) MIN_VALID_COUNTRIES 50 → 25

Coverage gate lowered so partial-success runs (5-25 successful
fetches + stale-served from cache) can advance seed-meta. Pre-fix,
seed-meta was frozen at 2026-05-12 00:03 UTC for 60+ hours because
no run reached 50. The 25 floor is sized so a 5-success run still
fails (real outage) but a 25+ partial-success run advances meta and
clears the operator-facing WARNING.

Marked TEMPORARY in the inline comment with a 2026-05-20 review
target — must revert to 50 once ArcGIS recovers and runs reliably
exceed 50 again. Test enforces the comment shape so a future
reviewer can't silently normalise 25 as the new permanent floor.

Tests: 87/87 pass (was 85, +2 for BATCH_BACKOFF + MIN_VALID_COUNTRIES
invariants; +1 modification for CONCURRENCY value).

This week's portwatch series:
  #3676  Structural cold-fetch cap (merged)
  #3681  Proxy retry on timeout (merged)
  This   Rate-limit recovery (concurrency + backoff + gate)
@vercel

vercel Bot commented May 14, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated (UTC)
worldmonitor Ignored Ignored Preview May 14, 2026 2:20pm

Request Review

@greptile-apps

greptile-apps Bot commented May 14, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR applies three coordinated operational mitigations to the seed-portwatch-port-activity script to survive an ongoing ArcGIS + Decodo rate-limit degradation where run-over-run successes dropped from 24/30 to 5/30. All changes are isolated to one script and its test file.

  • CONCURRENCY 12 \u2192 6 halves in-flight pressure on both the direct ArcGIS path and the Decodo proxy path, with budget math verified inline.
  • BATCH_BACKOFF_MS = 5_000 adds a 5 s inter-batch sleep (skipped on the final batch and when the abort signal is already set) to let both upstream buckets refill between bursts.
  • MIN_VALID_COUNTRIES 50 \u2192 25 (temporary) lets partial-success runs advance seed-meta; gated with a review-date comment and a test assertion enforcing the "TEMPORARILY" and "Revert to 50" language to prevent silent normalization.

Confidence Score: 4/5

Safe to merge as an emergency hotfix; the three operational knob changes are well-reasoned and budget math checks out.

All three core changes are straightforward constant adjustments with clear justification and matching test coverage. The signal-abort guard skips the sleep correctly but does not break the loop, leaving process.exit(1) in onSigterm as the real SIGTERM backstop. A wrong PR number in a test comment is the only other finding.

No files require special attention beyond the two minor findings noted above.

Important Files Changed

Filename Overview
scripts/seed-portwatch-port-activity.mjs Three targeted mitigations for ArcGIS rate-limit degradation: CONCURRENCY halved 12→6, new BATCH_BACKOFF_MS=5000 inter-batch sleep, MIN_VALID_COUNTRIES lowered 50→25 with inline documentation; signal-abort guard on the sleep has a correctness subtlety
tests/portwatch-port-activity-seed.test.mjs Two new invariant tests and one updated CONCURRENCY pin; correct structure, but the CONCURRENCY test comment cites the wrong PR number (#3683 vs #3694)

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[fetchAll called] --> B[Partition countries]
    B --> C{needsFetch > MAX_COLD_FETCH=30?}
    C -- yes --> D[Cap at 30, serve rest stale]
    C -- no --> E[Use all misses]
    D --> F[batches = ceil / CONCURRENCY=6]
    E --> F
    F --> G[for batchIdx 1..batches]
    G --> H[Promise.allSettled 6 fetches]
    H --> I[Write results to countryData]
    I --> J{batchIdx==1 AND >=80% errors?}
    J -- yes --> K[CIRCUIT-BREAKER break]
    J -- no --> L{batchIdx < batches AND !signal.aborted?}
    L -- yes --> M[sleep BATCH_BACKOFF_MS=5s]
    L -- no --> N{more batches?}
    M --> N
    N -- yes --> G
    N -- no --> O[validateFn: size >= MIN_VALID=25]
    K --> O
    O -- fail --> P[extendTtl + return early]
    O -- pass --> Q[Write Redis + advance seed-meta]
Loading

Reviews (1): Last reviewed commit: "fix(seed-portwatch): rate-limit recovery..." | Re-trigger Greptile

it('CONCURRENCY is 12 and PER_COUNTRY_TIMEOUT_MS is 90s', () => {
assert.match(src, /CONCURRENCY\s*=\s*12/);
it('CONCURRENCY is 6 and PER_COUNTRY_TIMEOUT_MS is 90s', () => {
// Halved from 12 → 6 on 2026-05-14 (PR #3683) to ease pressure on both

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 The comment inside this test cites PR #3683, but that is a previously merged PR in this series (not the current one). The current PR is #3694. Since this comment is copy-pasted into the test suite as a permanent record of why CONCURRENCY was halved, a wrong PR number will send a future reader to the wrong diff.

Suggested change
// Halved from 12 → 6 on 2026-05-14 (PR #3683) to ease pressure on both
// Halved from 12 → 6 on 2026-05-14 (PR #3694) to ease pressure on both

Comment on lines +818 to +820
if (batchIdx < batches && !signal?.aborted) {
await new Promise((r) => setTimeout(r, BATCH_BACKOFF_MS));
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Signal-abort check skips sleep but doesn't exit the loop

After the !signal?.aborted guard skips the 5 s sleep, the for loop immediately continues to the next iteration and starts a fresh batch of up to 6 concurrent fetches. Because withPerCountryTimeout creates its own independent AbortController (not linked to the parent signal), those fetches run completely oblivious to the abort. The PR description and inline comment claim "the loop exits immediately without paying the full 5 s," but that is only true for the sleep itself; the onSigtermprocess.exit(1) path is the actual backstop. Adding if (signal?.aborted) break; immediately after the skipped-sleep block would match the stated intent.

Greptile PR #3694 review:

P2 — Signal-abort skipped the sleep but kept the loop running.

Pre-fix: `if (batchIdx < batches && !signal?.aborted) await sleep(...)`
correctly skipped the 5s sleep on abort, but the for-loop continued
to the next iteration and started a fresh batch of up to 6 concurrent
fetches via withPerCountryTimeout (which creates its own AbortController
not linked to the parent signal). Net effect: the actual SIGTERM
backstop was onSigterm → process.exit(1), not the abort guard.

Fix: `if (signal?.aborted) break;` before the sleep — exits the loop
immediately so SIGTERM doesn't start additional in-flight work.

P2 — Test comment cited wrong PR number (#3683 vs #3694).

The "Halved from 12 → 6 on 2026-05-14 (PR #3683)" comment in the
CONCURRENCY test referenced a non-existent PR in this series. Fixed
to #3694 so a future reviewer following the comment lands on the
right diff.

Tests: 87/87 still pass. Updated the BATCH_BACKOFF_MS test to assert
both the new `break` shape and the simplified sleep condition; the
previous combined `batchIdx < batches && !signal?.aborted` pattern is
now split across the two checks.
@koala73

koala73 commented May 14, 2026

Copy link
Copy Markdown
Owner Author

Both P2s addressed in `2ec4b41c2`:

P2 — Signal-abort skipped sleep but didn't break the loop

Greptile's catch was correct. The pre-fix if (batchIdx < batches && !signal?.aborted) await sleep(...) correctly avoided paying the 5s on abort, but the for-loop then ran the next iteration and started up to 6 fresh concurrent fetches through `withPerCountryTimeout` (which uses its own AbortController, not linked to the parent signal). The actual SIGTERM responsiveness came entirely from the `onSigterm → process.exit(1)` handler, not from the abort guard — my comment overclaimed.

Fix: `if (signal?.aborted) break;` BEFORE the sleep check. Now the loop exits immediately on abort, no additional in-flight batch starts, and the comment matches behavior.

P2 — Test comment cited wrong PR number

The CONCURRENCY test had `"Halved from 12 → 6 on 2026-05-14 (PR #3683)"` — `#3683` was from a hypothetical PR-numbering reference in my drafting, not a real PR. Fixed to `#3694`.

87/87 tests still pass. Updated the BATCH_BACKOFF_MS test to assert both the new `break` shape and the simplified sleep condition.

Greptile PR #3694 round 3 P1: with the temp coverage gate lowered to 25,
a cap-mode partial-success run would CLEAR the coverage gate
(countryData.size ≥ 25) but STILL fail the unchanged degradation guard
(countryData.size < prevCount × 0.8 ≈ 139 in the incident state) —
seed-meta never advances, WARNING persists, PR's main recovery claim
broken.

Math at the current incident state:
  prevCount               174
  degradation threshold   0.8 × 174 = 139
  cap-mode realistic      30 cold-fetch + ~24 stale-served = ~54
  → 54 << 139 → DEGRADATION GUARD FAILS → extendTtl + return
  → seed-meta stays frozen, WARNING persists.

Fix shape: signal cap-mode from fetchAll() back to main() via a new
`capTriggered: bool` field plus `servedStaleCount` / `droppedTooOldCount`
/ `droppedNoCacheCount` counters. main() bypasses the 80% guard for
cap-mode runs (intentional partial coverage by design — see #3676's
rotation contract) and logs a `PARTIAL PUBLISH (cap-mode)` line with
the fresh/stale split.

The bypass is scoped: non-cap runs (where needsFetch ≤ 30 and we
fetched all misses normally) STILL apply the 80% guard so silent
data-loss scenarios (ArcGIS regression silently drops 100 → 50
countries) remain caught.

Logic shape:
  if (!validateFn) → extendTtl + return                    [coverage gate]
  if (capTriggered) → log PARTIAL PUBLISH + fall through   [bypass]
  else if (countryData.size < prevCount × 0.8) → extendTtl + return [silent-loss guard]
  → publish + advance seed-meta

Tests (3 new): fetchAll-shape (returns the 4 new fields), bypass
shape (else-if structure ensures cap-mode skips the guard), bypass
exclusivity (non-cap runs still enforce the guard with exactly 2
references to `prevCount × 0.8` — one in condition, one in error msg).
90/90 pass (was 87).
@koala73

koala73 commented May 14, 2026

Copy link
Copy Markdown
Owner Author

P1 addressed in `ab9dc5c38`. Critical catch — the temp gate lowering was structurally insufficient because the 80% degradation guard would still fire and block seed-meta refresh.

Math at the incident state (confirms the bug)

Quantity Value
prevCount (May 12 canonical) 174
Degradation threshold 0.8 × 174 = 139
Cap-mode realistic publish 30 cold-fetch + ~24 stale-served = ~54
54 vs 139 Fails by 85 countries

So even after the coverage gate change (gate=25, 54 ≥ 25 passes), the very next line — `if (countryData.size < prevCount * 0.8)` — would still trip, extendTtl, return, and seed-meta would never advance. WARNING would persist exactly as before. Your reading of the code is correct.

Fix shape

`fetchAll()` now returns four new fields:

```js
return {
countries, countryData, fetchedAt,
capTriggered, // true if needsFetch > MAX_COLD_FETCH_PER_RUN fired
servedStaleCount, // deferred countries served from prior payload
droppedTooOldCount, // deferred but cache > 7d (hard-dropped)
droppedNoCacheCount, // deferred with no prior payload (new countries)
};
```

`main()` reads them and wraps the degradation guard in a cap-mode bypass:

```js
if (capTriggered) {
console.warn(`PARTIAL PUBLISH (cap-mode): ${countryData.size}/${prevCount} countries — ${servedStaleCount} stale-served, ${droppedTooOldCount} dropped (cache > 7d), ${droppedNoCacheCount} dropped (no prior payload). Degradation guard bypassed; seed-meta will advance.`);
} else if (prevCount > 0 && countryData.size < prevCount * 0.8) {
// unchanged — silent-loss protection still active for non-cap runs
...
}
```

Why the bypass is scoped, not blanket

The 80% guard exists to catch silent data loss — an ArcGIS regression that drops 100 → 50 countries with no signal. That scenario still produces `capTriggered = false` because `needsFetch.length ≤ MAX_COLD_FETCH_PER_RUN` (in a regression, most countries' `asof` matches → cache hit → not added to needsFetch). So non-cap runs still go through the guard, silent-loss protection preserved.

Cap-mode is the LOUD, INTENTIONAL, ROTATIONAL path — partial coverage is by design per #3676's rotation contract (refresh 30/run, full coverage in ~6 runs).

Tests (3 new)

  1. fetchAll-shape: returns the 4 new fields
  2. bypass-shape: `if (capTriggered) { PARTIAL PUBLISH log } else if (...) { DEGRADATION GUARD }` structure
  3. bypass-exclusivity: exactly 2 references to `prevCount × 0.8` (the if-condition + the error message's `Math.ceil()` suggestion) — both inside the else-if scope. A 3rd reference would mean a duplicated check leaked outside the bypass.

90/90 tests pass (was 87).

@koala73
koala73 merged commit e245308 into main May 14, 2026
11 checks passed
@koala73
koala73 deleted the fix/portwatch-rate-limit-recovery branch May 14, 2026 14:45
koala73 added a commit that referenced this pull request May 15, 2026
…+P2 round 3)

P1 — Proxy-confirmed Invalid Params never reached the degradation threshold.

Pre-fix the proxy-confirmed short-circuit (`/via proxy after/i` → throw)
ran BEFORE the threshold check. So during the actual incident — where
nearly every error arrives via the proxy fallback path — every error
threw the per-country proxy message and the clean
`ArcGIS degraded — N 'Invalid query parameters' errors` message was
unreachable.

Fix: reorder so threshold check fires first, then proxy short-circuit.
Counter still increments for proxy-confirmed errors (they contribute to
the threshold), and once the threshold is exceeded the degraded
message surfaces regardless of whether the error came from direct or
proxy.

P2 — 20s capture budget didn't realistically catch 30-56s responses.

The 3-attempts-× 20s fix was bounded but each attempt was a FRESH
request starting from t=0 (the original 45s direct fetch was already
aborted). For the observed degraded response class (30-56s server-side
from request start), each 20s capture aborts before the body lands.

Fix: bump ERROR_BODY_CAPTURE_EXTRA_MS 20s → 40s. Math under
PER_COUNTRY_TIMEOUT_MS=90s:
  direct (45s timeout) + capture (40s budget) = 85s, leaving 5s before
  the per-country wrap fires. Proxy retry effectively dies for
  timing-out countries in this mode — accepted tradeoff because the
  proxy is itself degraded (Decodo throttled per #3694 history), so it
  wasn't helping anyway. Attempt cap stays at 3 (across the run, not
  per country) so the diagnostic value is captured by 3rd timing-out
  country; subsequent countries go straight to proxy.

Tests: 1 new ordering test using src.search() index comparison to
assert threshold-throw-before-proxy-short-circuit; 2 existing tests
updated for the bumped constant and the longer span between
increment and short-circuit. 97/97 portwatch tests pass; 8806/8807
npm run test:data (1 flake = pre-existing _bundle-fixture-* race
between bundle-runner and news-classify-cache-prefix-audit).
koala73 added a commit that referenced this pull request May 15, 2026
…e retries (#3701)

* fix(seed-portwatch): surface degraded ArcGIS 400 body + cap degenerate retries

ArcGIS Daily_Ports_Data, during degradation episodes, returns HTTP 200
with a 400 error body (`Cannot perform query. Invalid query parameters.`)
after 30-56s of server processing. Our 45s FETCH_TIMEOUT fires before
the body lands, callers see AbortError, the existing circuit-breaker
that pattern-matches the error message never fires, and the seeder
treats every degraded country as a generic timeout (then a doomed proxy
retry, then another doomed first-direct-retry from the one-shot
fetchWithRetryOnInvalidParams).

Two surgical fixes:

1. **Diagnostic body-capture on timeout.** When the direct fetch
   hits its FETCH_TIMEOUT (not a caller-signal abort), make ONE extra
   fetch with +20s budget purely to read the response body. If it
   contains `body.error`, throw with the real ArcGIS message so the
   upstream circuit-breaker can act on the upstream-degradation signal.
   Gated to fire once per process — the proxy budget Greptile P2'd on
   PR #3681 stays intact for subsequent timeouts.

2. **Bail-don't-retry threshold on Invalid-query-parameters.** Preserve
   the legit one-shot retry for transient single-call flakes (2026-04-20
   BRA/IDN/NGA pattern), but cap total retries-on-this-error per
   process at 5. Beyond that, throw a clear `ArcGIS degraded — N errors`
   message so the operator sees the regression class instead of N
   identical retries × 45s burning the 540s container budget.

Investigation that drove this: direct curl probes against
services9.arcgis.com revealed (a) HTTP 200 with a 400 error body, (b)
non-deterministic results across attempts within minutes (same query,
ERR → OK → ERR → 504), (c) WHERE-clause reshaping helps unevenly but
never reliably. The right frame is "upstream is degraded, surface it
and protect the container budget" not "we're being rate-limited."

The existing cap-mode + stale-serve + temp-gate-25 logic (PRs
#3676/#3681/#3694) already rides through partial coverage — these
changes just make sure we don't waste the container budget while
ArcGIS recovers, and that the next diagnostic round has the real error
message instead of generic AbortError.

Tests:
- 3 new structural pattern-match tests covering ERROR_BODY_CAPTURE_EXTRA_MS,
  _captureErrorBodyAfterTimeout helper, once-per-run gating,
  INVALID_PARAMS_RETRY_THRESHOLD, counter increment + threshold check,
  reset helpers.
- 96/96 tests pass (was 93 before).

* review: gate cap-mode bypass on fresh upstream contact + abort-aware backoff sleep (P1+P2)

P1 — Cap-mode bypass could publish a stale-only canonical list as healthy.

Pre-fix the capTriggered bypass of the 80% degradation guard was
unconditional. With the lowered MIN_VALID_COUNTRIES=25 + cap-mode +
servedStale entries seeding countryData, a run with all-stale entries
(freshFetched=0, cacheHits=0, servedStale=27, dropped=147) could:
  - Pass validateFn (countryData.size >= 25 — all from stale-cache)
  - Bypass the 80% guard (capTriggered=true)
  - Shrink the canonical list from ~174 → 27 stale-only entries
  - Advance seed-meta, clearing the operator-facing WARNING

Fix: gate the bypass on freshFetchedCount + cacheHitCount ≥
MIN_FRESH_FETCH_FOR_CAP_BYPASS (5). Below that floor, cap-mode is NOT
rotational steady-state — it's an ArcGIS-completely-down scenario, and
the run falls through to the 80% guard (which extendExistingTtl-only,
preserves canonical list, keeps WARNING visible). New log line
"CAP-MODE BYPASS REFUSED: only N fresh upstream contacts" makes the
refusal observable to operators.

This composes with the INVALID_PARAMS_RETRY_THRESHOLD=5 fix earlier in
this PR — that change increases the rate at which cold fetches return
errors during upstream degradation, making the stale-only scenario
this P1 guards against MORE likely, not less.

P2 — Inter-batch backoff sleep is now abort-aware.

The 5s BATCH_BACKOFF_MS wait used a plain setTimeout that ignored the
caller signal mid-sleep, so a SIGTERM during the wait still forced the
full 5s before observing the abort. Now races the setTimeout against
signal's abort event so the loop exits immediately on a real
cancellation (the signal?.aborted check at the top of the next
iteration then `break`s the loop). Cosmetic change but matches the
PR-body intent that the loop is SIGTERM-responsive.

Tests:
- 2 new structural pattern-match tests covering MIN_FRESH_FETCH_FOR_CAP_BYPASS,
  freshFetchedCount counter, upstream-contact gate, CAP-MODE BYPASS REFUSED
  branch, and the abort-aware sleep shape.
- 4 existing tests updated to match the new destructure shape + new prevCount
  × 0.8 reference count (now 4: 2 conditions + 2 Math.ceil sites, one set per
  guard branch).
- 95/95 portwatch tests pass; 8804/8805 npm run test:data (1 flake is the
  pre-existing scripts/_bundle-fixture-env-*.mjs race between
  bundle-runner.test.mjs and news-classify-cache-prefix-audit.test.mjs,
  unrelated to this change).

* review: gate body-capture on success not attempt + short-circuit proxy-confirmed errors (P2×2)

Two Greptile P2 comments on PR #3701 review round 2.

P2 #1 — Body-capture gate fired on first ATTEMPT, not first SUCCESS.

During consistent degradation, the 20s ERROR_BODY_CAPTURE_EXTRA_MS
window isn't long enough — ArcGIS needs 30-56s of server processing
per query from connection start. A failed first capture (capture also
timing out) flipped the once-per-run flag, locking out every
subsequent timing-out country from ever capturing the body. The
diagnostic value could be lost for an entire run, defeating the whole
point of the capture path.

Fix: track SUCCESS count (cap at 1 — we only need the body shape once)
AND ATTEMPT count (cap at 3 to bound total cost). Null captures consume
an attempt but not a success, so subsequent timing-out countries keep
trying. Worst-case cost: 3 × 20s wall-clock per run, still within
PER_COUNTRY_TIMEOUT_MS=90s caps.

P2 #2 — Proxy-confirmed "Invalid query parameters" still triggered retry.

When arcgisProxyRetry returns an error body, the thrown message has
the `(via proxy after ${reason})` prefix and still matches
`/Invalid query parameters/i`. fetchWithRetryOnInvalidParams counted
it, then retried via fetchWithTimeout — which would timeout → proxy →
hit the same authoritative error, burning a full direct+proxy cycle
for nothing. The proxy response IS the definitive upstream signal;
retrying is meaningless.

Fix: counter still increments (proxy-confirmed errors are valid
degradation signals for the threshold), but short-circuit the retry
with `if (/via proxy after/i.test(msg)) throw err`.

Tests: 2 new structural pattern-match tests + 1 existing test relaxed
to accept the new `if (captured?.error) { successCount++; throw ... }`
shape. 96/96 portwatch tests pass; 8806/8806 npm run test:data
(unrelated bundle-fixture race cleared this run).

* review: threshold-before-short-circuit + realistic capture budget (P1+P2 round 3)

P1 — Proxy-confirmed Invalid Params never reached the degradation threshold.

Pre-fix the proxy-confirmed short-circuit (`/via proxy after/i` → throw)
ran BEFORE the threshold check. So during the actual incident — where
nearly every error arrives via the proxy fallback path — every error
threw the per-country proxy message and the clean
`ArcGIS degraded — N 'Invalid query parameters' errors` message was
unreachable.

Fix: reorder so threshold check fires first, then proxy short-circuit.
Counter still increments for proxy-confirmed errors (they contribute to
the threshold), and once the threshold is exceeded the degraded
message surfaces regardless of whether the error came from direct or
proxy.

P2 — 20s capture budget didn't realistically catch 30-56s responses.

The 3-attempts-× 20s fix was bounded but each attempt was a FRESH
request starting from t=0 (the original 45s direct fetch was already
aborted). For the observed degraded response class (30-56s server-side
from request start), each 20s capture aborts before the body lands.

Fix: bump ERROR_BODY_CAPTURE_EXTRA_MS 20s → 40s. Math under
PER_COUNTRY_TIMEOUT_MS=90s:
  direct (45s timeout) + capture (40s budget) = 85s, leaving 5s before
  the per-country wrap fires. Proxy retry effectively dies for
  timing-out countries in this mode — accepted tradeoff because the
  proxy is itself degraded (Decodo throttled per #3694 history), so it
  wasn't helping anyway. Attempt cap stays at 3 (across the run, not
  per country) so the diagnostic value is captured by 3rd timing-out
  country; subsequent countries go straight to proxy.

Tests: 1 new ordering test using src.search() index comparison to
assert threshold-throw-before-proxy-short-circuit; 2 existing tests
updated for the bumped constant and the longer span between
increment and short-circuit. 97/97 portwatch tests pass; 8806/8807
npm run test:data (1 flake = pre-existing _bundle-fixture-* race
between bundle-runner and news-classify-cache-prefix-audit).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant