fix(portwatch): unblock port-activity seeder (global EP4 refs, conc 12, progress logs, SIGTERM) by koala73 · Pull Request #3128 · koala73/worldmonitor

koala73 · 2026-04-16T11:01:44Z

Summary

seed-portwatch-port-activity has been SIGKILL'd at the Railway 10-min container ceiling on every run since 2026-04-14 (recordCount=174, seedAgeMin=3096 = 51.6h), leaving portwatchPortActivity STALE_SEED and the 30-min lock leaking between cron ticks.

Root cause: ~240 ISO3s × 2 per-country ArcGIS queries at CONCURRENCY=4 with no per-batch logging — slow enough to miss the 420s timeoutMs and silent enough that the timeout line was the only log on failure (see earlier run logs.1776183642501.log line 73: failed after 600.1s: timeout).

Four fixes applied:

fetchAllPortRefs() — one paginated EP4 query (where=1=1, outFields=portid,ISO3,lat,lon) grouped by ISO3 locally. Collapses ~240 ref calls into ~5 pages.
CONCURRENCY 4 → 12 and only queue activity fetches for ISO3s that (a) have a port in refsByIso3 and (b) are in the iso3→iso2 map. Skips ~66 unmapped/empty iso3s.
Per-page ref logs + per-batch activity logs every 5 batches (BATCH_LOG_EVERY=5). The next failure will show exactly where it stalls instead of a silent 10-min hang.
SIGTERM/SIGINT handler releases the 30-min lock and extends prev-snapshot TTLs before exiting with code 1, so the next cron tick isn't blocked by a lingering lock and Redis data doesn't evaporate when the bundle-runner kills the child.

Tests updated to match (EP4 now uses where=1=1, concurrency 12 assertion, SIGTERM handler assertion).

Test plan

…GTERM cleanup seed-portwatch-port-activity has been SIGKILL'd at the Railway 10-min container ceiling on every run since 2026-04-14 (recordCount=174, seedAgeMin=3096 = 51.6h = 4+ failed cycles), leaving portwatchPortActivity STALE_SEED and the 30-min lock leaking between runs. Root cause: ~240 ISO3s x 2 per-country ArcGIS queries at CONCURRENCY=4 with zero per-batch logging — slow enough to miss the 420s timeoutMs and silent enough that the timeout line was the only log on failure. Fixes (all 4): 1. fetchAllPortRefs(): one paginated EP4 query (where=1=1), grouped by ISO3 locally — collapses ~240 ref calls into ~5 pages. 2. CONCURRENCY 4 -> 12 and only queue activity fetches for iso3s that appear in refsByIso3 and in the iso3->iso2 map. 3. Per-page ref logs + per-batch activity logs every 5 batches — next failure will show exactly where it stalls. 4. SIGTERM/SIGINT handler releases the lock and extends prev-snapshot TTLs before exit so the next cron tick isn't blocked and Redis data doesn't evaporate when the bundle-runner kills the child.

vercel · 2026-04-16T11:01:55Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
worldmonitor	Ready	Preview, Comment	Apr 16, 2026 11:42am

greptile-apps · 2026-04-16T11:06:02Z

Greptile Summary

This PR unblocks the seed-portwatch-port-activity seeder that was being SIGKILL'd at Railway's 10-min ceiling by collapsing ~240 per-country EP4 queries into ~5 paginated global pages, raising concurrency from 4 to 12, adding per-batch progress logs, and installing a SIGTERM/SIGINT handler that releases the lock and extends TTLs before exiting. Two P2 observations: the SIGTERM handler should be deregistered in the finally block to prevent process.exit(1) from overriding a successful exit code in the rare race where a signal lands during the final event-loop drain; and the global EP4 fetch is now all-or-nothing — a mid-pagination failure aborts the entire ref collection, trading the old per-country fault-isolation for the speed that motivated this PR.

Confidence Score: 4/5

Safe to merge — all remaining findings are P2; data integrity and lock semantics are correct.

Both findings are P2: the SIGTERM deregistration race is low-probability and data-safe (Redis writes and lock release have already happened), and the EP4 all-or-nothing fault-isolation tradeoff is intentional and documented in the PR. Neither blocks correctness of a normal run. Score is 4 rather than 5 only to flag these for author awareness before the next Railway run is observed.

scripts/seed-portwatch-port-activity.mjs — SIGTERM handler deregistration and EP4 pagination fault isolation.

Important Files Changed

Filename	Overview
scripts/seed-portwatch-port-activity.mjs	Core seeder refactored: global EP4 ref fetch, CONCURRENCY 12, batch progress logs, SIGTERM handler — two minor concerns around handler deregistration and EP4 atomicity.
tests/portwatch-port-activity-seed.test.mjs	Tests updated to match new EP4 global query, concurrency bump, and SIGTERM handler; coverage is accurate, no issues found.

Sequence Diagram

sequenceDiagram
    participant R as Railway Runner
    participant M as main()
    participant FA as fetchAll()
    participant EP4 as ArcGIS EP4 (global refs)
    participant EP3 as ArcGIS EP3 (activity)
    participant Redis

    R->>M: spawn process
    M->>M: acquireLock()
    M->>M: register SIGTERM/SIGINT handler
    M->>M: readSeedSnapshot() → prevCountryKeys
    M->>FA: fetchAll()
    FA->>EP4: GET where=1=1 page 1..~5
    EP4-->>FA: Map<iso3, Map<portId,{lat,lon}>>
    FA->>FA: filter eligibleIso3 (have port + in iso3→iso2 map)
    loop batches of 12
        FA->>EP3: fetchActivityRows(iso3) ×12 in parallel
        EP3-->>FA: rawRows per country
        FA->>FA: computeCountryPorts(rawRows, refMap)
        FA->>FA: log progress every 5 batches
    end
    FA-->>M: {countries, countryData}
    M->>Redis: pipeline SET per-country + canonical + meta
    Redis-->>M: OK
    M->>M: releaseLock() [finally]

    alt SIGTERM arrives mid-run
        R->>M: SIGTERM
        M->>M: onSigterm(): extendExistingTtl + releaseLock
        M->>M: process.exit(1)
    end

_{Reviews (1): Last reviewed commit: "fix(portwatch): global EP4 refs + concur..." | Re-trigger Greptile}

greptile-apps · 2026-04-16T11:06:06Z

+  };
+  process.on('SIGTERM', onSigterm);
+  process.on('SIGINT', onSigterm);


SIGTERM handler not deregistered on clean exit

If the seed completes successfully but SIGTERM arrives in the small window before Node.js drains the event loop and exits naturally (e.g., a slow releaseLock network round-trip inside finally keeps the loop alive long enough for the Railway timeout signal to land), onSigterm will call process.exit(1) — overriding the successful exit code. Monitoring would see a failure even though all Redis writes succeeded and the lock was released.

Deregistering both listeners in the finally block closes the window:

Suggested change

};

process.on('SIGTERM', onSigterm);

process.on('SIGINT', onSigterm);

process.on('SIGTERM', onSigterm);

process.on('SIGINT', onSigterm);

Then in the finally block, before or after releaseLock:

} finally { process.off('SIGTERM', onSigterm); process.off('SIGINT', onSigterm); await releaseLock(LOCK_DOMAIN, runId); }

greptile-apps · 2026-04-16T11:06:07Z

@@ -77,15 +83,20 @@ async function fetchPortRef(iso3) {
      f: 'json',
    });
    body = await fetchWithTimeout(`${EP4_BASE}?${params}`);
-    for (const f of body.features ?? []) {
+    const features = body.features ?? [];
+    for (const f of features) {
      const a = f.attributes;
-      if (a?.portid != null) {
-        refMap.set(String(a.portid), { lat: Number(a.lat ?? 0), lon: Number(a.lon ?? 0) });
-      }
+      if (a?.portid == null || !a?.ISO3) continue;
+      const iso3 = String(a.ISO3);
+      const portId = String(a.portid);
+      let ports = byIso3.get(iso3);
+      if (!ports) { ports = new Map(); byIso3.set(iso3, ports); }
+      ports.set(portId, { lat: Number(a.lat ?? 0), lon: Number(a.lon ?? 0) });
    }
+    console.log(`  [port-activity]   ref page ${page}: +${features.length} ports (${byIso3.size} countries so far)`);
    offset += PAGE_SIZE;
  } while (body.exceededTransferLimit);
-  return refMap;
+  return byIso3;


EP4 pagination is all-or-nothing — single page failure aborts entire ref fetch

fetchAllPortRefs() is called with no error-isolation: if page 3 of 5 throws (timeout, ArcGIS 500, transient 429 exhausting the proxy), the function propagates the error all the way up through fetchAll() and into main(), which re-throws after extending TTLs. No port coordinates are collected for any country — the whole seed fails even if 4 pages completed successfully.

In the old per-country approach, a single fetchPortRef(iso3) failure caused that country's ports to default to {lat:0, lon:0} while the rest of the seed continued. The new design trades that fault-isolation for speed (which is the right call here), but if you want a middle ground you could loop over pages and continue on individual-page errors while logging a warning, or at minimum add a retry on transient failures before giving up.

…E_SIZE Review finding on PR #3128: the new fetchAllPortRefs() global pager assumes the server honors resultRecordCount=2000, but ArcGIS's PortWatch_ports_database FeatureServer caps responses at 1000 rows. Incrementing offset by PAGE_SIZE (=2000) silently skipped rows 1000-1999 on EVERY page. Verified against the live endpoint: - returnCountOnly: 2065 - offset=0 size=2000: returned 1000 (ETL=true) - offset=1000 size=2000: returned 1000 (ETL=true) - offset=2000 size=2000: returned 65 (ETL=false) The buggy loop therefore loaded 1065 refs instead of 2065 — silently dropping 34 mapped countries with EP3 activity entirely and leaving 110 more countries with partial ref coverage. Partial coverage fell back to (0,0) lat/lon via `refMap.get(portId) || { lat: 0, lon: 0 }`. Not a regression from the old code (per-country EP4 fetches maxed at ~148 ports and never hit the cap), but a real bug introduced by the global pager. Fix: advance `offset` by `features.length` (the actual returned count) instead of by `PAGE_SIZE`. Applied to both fetchAllPortRefs (EP4) and fetchActivityRows (EP3) for consistency. Added break-on-empty guard so a server that returns exceededTransferLimit=true with 0 features can't infinite-loop. Regression test asserts `offset += features.length` appears for both paginators and `offset += PAGE_SIZE` appears nowhere in the file.

…3384) * feat(seed): BUNDLE_RUN_STARTED_AT_MS env + runSeed SIGTERM cleanup Prereq for the re-export-share Comtrade seeder (plan 2026-04-24-003), usable by any cohort seeder whose consumer needs bundle-level freshness. Two coupled changes: 1. `_bundle-runner.mjs` injects `BUNDLE_RUN_STARTED_AT_MS` into every spawned child. All siblings in a single bundle run share one value (captured at `runBundle` start, not spawn time). Consumers use this to detect stale peer keys — if a peer's seed-meta predates the current bundle run, fall back to a hard default rather than read a cohort-peer's last-week output. 2. `_seed-utils.mjs::runSeed` registers a `process.once('SIGTERM')` handler that releases the acquired lock and extends existing-data TTL before exiting 143. `_bundle-runner.mjs` sends SIGTERM on section timeout, then SIGKILL after KILL_GRACE_MS (5s). Without this handler the `finally` path never runs on SIGKILL, leaving the 30-min acquireLock reservation in place until its own TTL expires — the next cron tick silently skips the resource. Regression guard memory: `bundle-runner-sigkill-leaks-child-lock` (PR #3128 root cause). Tests added: - bundle-runner env injection (value within run bounds) - sibling sections share the same timestamp (critical for the consumer freshness guard) - runSeed SIGTERM path: exit 143 + cleanup log - process.once contract: second SIGTERM does not re-enter handler * fix(seed): address P1/P2 review findings on SIGTERM + bundle contracts Addresses PR #3384 review findings (todos 256, 257, 259, 260): #256 (P1) — SIGTERM handler narrowed to fetch phase only. Was installed at runSeed entry and armed through every `process.exit` path; could race `emptyDataIsFailure: true` strict-floor exits (IMF-External, WB-bulk) and extend seed-meta TTL when the contract forbids it — silently re-masking 30-day outages. Now the handler is attached immediately before `withRetry(fetchFn)` and removed in a try/finally that covers all fetch-phase exit branches. #257 (P1) — `BUNDLE_RUN_STARTED_AT_MS` now has a first-class helper. Exported `getBundleRunStartedAtMs()` from `_seed-utils.mjs` with JSDoc describing the bundle-freshness contract. Fleet-wide helper so the next consumer seeder imports instead of rediscovering the idiom. #259 (P2) — SIGTERM cleanup runs `Promise.allSettled` on disjoint-key ops (`releaseLock` + `extendExistingTtl`). Serialising compounded Upstash latency during the exact failure mode (Redis degraded) this handler exists to handle, risking breach of the 5s SIGKILL grace. #260 (P2) — `_bundle-runner.mjs` asserts topological order on optional `dependsOn` section field. Throws on unknown-label refs and on deps appearing at a later index. Fleet-wide contract replacing the previous prose-comment ordering guarantee. Tests added/updated: - New: SIGTERM handler removed after fetchFn completes (narrowed-scope contract — post-fetch SIGTERM must NOT trigger TTL extension) - New: dependsOn unknown-label + out-of-order + happy-path (3 tests) Full test suite: 6,866 tests pass (+4 net). * fix(seed): getBundleRunStartedAtMs returns null outside a bundle run Review follow-up: the earlier `Math.floor(Date.now()/1000)*1000` fallback regressed standalone (non-bundle) runs. A consumer seeder invoked manually just after its peer wrote `fetchedAt = (now - 5s)` would see `bundleStartMs = Date.now()`, reject the perfectly-fresh peer envelope as "stale", and fall back to defaults — defeating the point of the peer-read path outside the bundle. Returning null when `BUNDLE_RUN_STARTED_AT_MS` is unset/invalid keeps the freshness gate scoped to its real purpose (across-bundle-tick staleness) and lets standalone runs skip the gate entirely. Consumers check `bundleStartMs != null` before applying the comparison; see the companion `seed-sovereign-wealth.mjs` change on the stacked PR. * test(seed): SIGTERM cleanup test now verifies Redis DEL + EXPIRE calls Greptile review P2 on PR #3384: the existing test only asserted exit code + log line, not that the Redis ops were actually issued. The log claim was ahead of the test. Fixture now logs every Upstash fetch call's shape (EVAL / pipeline- EXPIRE / other) to stderr. Test asserts: - >=1 EVAL op was issued during SIGTERM cleanup (releaseLock Lua script on the lock key) - >=1 pipeline-EXPIRE op was issued (extendExistingTtl on canonical + seed-meta keys) - The EVAL body carries the runSeed-generated runId (proves it's THIS run's release, not a phantom op) - The EXPIRE pipeline touches both the canonicalKey AND the seed-meta key (proves the keys[] array was built correctly including the extraKeys merge path) Full test suite: 6,866 tests pass, typecheck clean.

…read (#3385) * feat(seed): BUNDLE_RUN_STARTED_AT_MS env + runSeed SIGTERM cleanup Prereq for the re-export-share Comtrade seeder (plan 2026-04-24-003), usable by any cohort seeder whose consumer needs bundle-level freshness. Two coupled changes: 1. `_bundle-runner.mjs` injects `BUNDLE_RUN_STARTED_AT_MS` into every spawned child. All siblings in a single bundle run share one value (captured at `runBundle` start, not spawn time). Consumers use this to detect stale peer keys — if a peer's seed-meta predates the current bundle run, fall back to a hard default rather than read a cohort-peer's last-week output. 2. `_seed-utils.mjs::runSeed` registers a `process.once('SIGTERM')` handler that releases the acquired lock and extends existing-data TTL before exiting 143. `_bundle-runner.mjs` sends SIGTERM on section timeout, then SIGKILL after KILL_GRACE_MS (5s). Without this handler the `finally` path never runs on SIGKILL, leaving the 30-min acquireLock reservation in place until its own TTL expires — the next cron tick silently skips the resource. Regression guard memory: `bundle-runner-sigkill-leaks-child-lock` (PR #3128 root cause). Tests added: - bundle-runner env injection (value within run bounds) - sibling sections share the same timestamp (critical for the consumer freshness guard) - runSeed SIGTERM path: exit 143 + cleanup log - process.once contract: second SIGTERM does not re-enter handler * fix(seed): address P1/P2 review findings on SIGTERM + bundle contracts Addresses PR #3384 review findings (todos 256, 257, 259, 260): #256 (P1) — SIGTERM handler narrowed to fetch phase only. Was installed at runSeed entry and armed through every `process.exit` path; could race `emptyDataIsFailure: true` strict-floor exits (IMF-External, WB-bulk) and extend seed-meta TTL when the contract forbids it — silently re-masking 30-day outages. Now the handler is attached immediately before `withRetry(fetchFn)` and removed in a try/finally that covers all fetch-phase exit branches. #257 (P1) — `BUNDLE_RUN_STARTED_AT_MS` now has a first-class helper. Exported `getBundleRunStartedAtMs()` from `_seed-utils.mjs` with JSDoc describing the bundle-freshness contract. Fleet-wide helper so the next consumer seeder imports instead of rediscovering the idiom. #259 (P2) — SIGTERM cleanup runs `Promise.allSettled` on disjoint-key ops (`releaseLock` + `extendExistingTtl`). Serialising compounded Upstash latency during the exact failure mode (Redis degraded) this handler exists to handle, risking breach of the 5s SIGKILL grace. #260 (P2) — `_bundle-runner.mjs` asserts topological order on optional `dependsOn` section field. Throws on unknown-label refs and on deps appearing at a later index. Fleet-wide contract replacing the previous prose-comment ordering guarantee. Tests added/updated: - New: SIGTERM handler removed after fetchFn completes (narrowed-scope contract — post-fetch SIGTERM must NOT trigger TTL extension) - New: dependsOn unknown-label + out-of-order + happy-path (3 tests) Full test suite: 6,866 tests pass (+4 net). * fix(seed): getBundleRunStartedAtMs returns null outside a bundle run Review follow-up: the earlier `Math.floor(Date.now()/1000)*1000` fallback regressed standalone (non-bundle) runs. A consumer seeder invoked manually just after its peer wrote `fetchedAt = (now - 5s)` would see `bundleStartMs = Date.now()`, reject the perfectly-fresh peer envelope as "stale", and fall back to defaults — defeating the point of the peer-read path outside the bundle. Returning null when `BUNDLE_RUN_STARTED_AT_MS` is unset/invalid keeps the freshness gate scoped to its real purpose (across-bundle-tick staleness) and lets standalone runs skip the gate entirely. Consumers check `bundleStartMs != null` before applying the comparison; see the companion `seed-sovereign-wealth.mjs` change on the stacked PR. * test(seed): SIGTERM cleanup test now verifies Redis DEL + EXPIRE calls Greptile review P2 on PR #3384: the existing test only asserted exit code + log line, not that the Redis ops were actually issued. The log claim was ahead of the test. Fixture now logs every Upstash fetch call's shape (EVAL / pipeline- EXPIRE / other) to stderr. Test asserts: - >=1 EVAL op was issued during SIGTERM cleanup (releaseLock Lua script on the lock key) - >=1 pipeline-EXPIRE op was issued (extendExistingTtl on canonical + seed-meta keys) - The EVAL body carries the runSeed-generated runId (proves it's THIS run's release, not a phantom op) - The EXPIRE pipeline touches both the canonicalKey AND the seed-meta key (proves the keys[] array was built correctly including the extraKeys merge path) Full test suite: 6,866 tests pass, typecheck clean. * feat(resilience): Comtrade-backed re-export-share seeder + SWF Redis read Plan ref: docs/plans/2026-04-24-003-feat-reexport-share-comtrade-seeder-plan.md Motivating case. Before this PR, the SWF `rawMonths` denominator for the `sovereignFiscalBuffer` dimension used GROSS annual imports for every country. For re-export hubs (goods transiting without domestic settlement), this structurally under-reports resilience: UAE's 2023 $941B of imports include $334B of transit flow that never represents domestic consumption. Net imports = gross × (1 − reexport_share). The previous (PR 3A) design flattened a hand-curated YAML into Redis; the YAML shipped empty and never populated, so the correction never applied and the cohort audit showed no movement. Gap #2 (this PR). Two coupled changes to make the correction actually apply: 1. Comtrade-backed seeder (`scripts/seed-recovery-reexport-share.mjs`). Rewritten to fetch UN Comtrade `flowCode=RX` (re-exports) and `flowCode=M` (imports) per cohort member, compute share = RX/M at the latest co-populated year, clamp to [0.05, 0.95], publish the envelope. Header auth (`Ocp-Apim-Subscription-Key`) — subscription key never reaches URL/logs/Redis. `maxRecords=250000` cap with truncation detection. Sequential + retry-on-429 with backoff. Hub cohort resolved by Phase 0 empirical probe (plan §Phase 0): ['AE', 'PA']. Six candidates (SG/HK/NL/BE/MY/LT) return HTTP 200 with zero RX rows — Comtrade doesn't expose RX for those reporters. 2. SWF seeder reads from Redis (`scripts/seed-sovereign-wealth.mjs`). Swaps `loadReexportShareByCountry()` (YAML) for `loadReexportShareFromRedis()` (Redis key written by #1). Guarded by bundle-run freshness: if the sibling Reexport-Share seeder's `seed-meta` predates `BUNDLE_RUN_STARTED_AT_MS` (set by the prereq PR's `_bundle-runner.mjs` env-injection), HARD fallback to gross imports rather than apply last-month's stale share. Health registries. Both new keys registered in BOTH `api/health.js` SEED_META (60-day alert threshold) and `api/seed-health.js` SEED_DOMAINS (43200min interval). feedback_two_health_endpoints_must_match. Bundle wiring. `seed-bundle-resilience-recovery` Reexport-Share timeout bumped 60s → 300s (Comtrade + retry can take 2-3 min worst-case). Ordering preserved: Reexport-Share before Sovereign- Wealth so the SWF seeder reads a freshly-written key in the same cron tick. Deletions. YAML + loader + 7 obsolete loader tests removed; single source of truth is now Comtrade → Redis. Prereq. Stacks on PR #3384 (feat/bundle-runner-env-sigterm) which adds BUNDLE_RUN_STARTED_AT_MS env injection + runSeed SIGTERM cleanup. This PR's bundle-freshness guard depends on that env variable. Tests (19 new, 7 deleted, +12 net): - Pure math: parseComtradeFlowResponse, computeShareFromFlows, clampShare, declareRecords + credential-leak source scan (15) - Integration (Gap #2 regression guards): SWF seeder loadReexport ShareFromRedis — fresh/absent/malformed/stale-meta/missing-meta (5) - Health registry dual-registry drift guard — scoped to this PR's keys, respecting pre-existing asymmetry (4) - Bundle-ordering + timeout assertions (2) Phase 0 cohort validation committed to plan. Full test suite passes: 6,881 tests. * fix(resilience): address P1/P2 review findings — adopt shared helpers, pin freshness boundary Addresses PR #3385 review findings: #257 (P1) consumer — `seed-sovereign-wealth.mjs` imports the shared `getBundleRunStartedAtMs` helper from `_seed-utils.mjs` (added in the prereq commit) instead of its own `getBundleStartMs`. Single source of truth for the bundle-freshness contract. #258 (P2) — `seed-recovery-reexport-share.mjs` isMain guard uses the canonical `pathToFileURL(process.argv[1]).href === import.meta.url` form instead of basename-suffix matching. Handles symlinks, case- different paths on macOS HFS+, and Windows path separators without string munging. #260 (P2) consumer — Sovereign-Wealth declares `dependsOn: ['Reexport-Share']` in the bundle spec. `_bundle-runner.mjs` (prereq commit) now enforces topological order on load and throws on violation — replaces the previous prose-comment ordering contract. #261 (P2) — added a test to `tests/seed-sovereign-wealth-reads-redis- reexport-share.test.mts` pinning the inclusive-boundary semantic: `fetchedAtMs === bundleStartMs` must be treated as FRESH. Guards against a future refactor to `<=` that would silently reject peers writing at the very first millisecond of the bundle run. Rebased onto updated prereq. Full test suite: 6,886 tests pass (+5 net). * fix(resilience): freshness gate skipped in standalone mode; meta still required Review catch: the previous `bundleStartMs = Date.now()` fallback made standalone/manual `seed-sovereign-wealth.mjs` runs ALWAYS reject any previously-seeded re-export-share meta as "stale" — even when the operator ran the Reexport seeder milliseconds beforehand. Defeated the point of the peer-read path outside the bundle. With `getBundleRunStartedAtMs()` now returning null outside a bundle (companion commit on the prereq branch), the consumer only applies the freshness gate when `bundleStartMs != null`. Standalone runs accept any `fetchedAt` — the operator is responsible for ordering. Two guards survive the change: - Meta MUST exist (absence = peer-outage fail-safe, both modes) - In-bundle: meta MUST be at or after `BUNDLE_RUN_STARTED_AT_MS` Two new tests pin both modes: - standalone: accepts meta written 10 min before this process started - standalone: still rejects missing meta (peer-outage fail-safe survives gate bypass) Rebased onto updated prereq. Full test suite: 6,888 tests (+2 net). * fix(resilience): filter world-aggregate Comtrade rows + skip final-retry sleep Greptile review of PR #3385 flagged two P2s in the Comtrade seeder. Finding #3 (parseComtradeFlowResponse double-count risk): `cmdCode=TOTAL` without a partner filter currently returns only world-aggregate rows in practice — but `parseComtradeFlowResponse` summed every row unconditionally. A future refactor adding per- partner querying would silently double-count (world-aggregate row + partner-level rows for the same year), cutting the derived share in half with no test signal. Fix: explicit `partnerCode ∈ {'0', 0, null/undefined}` filter. Matches current empirical behavior (aggregate-only responses) and makes the construct robust to a future partner-level query. Finding #4 (wasted backoff on final retry): 429 and 5xx branches slept `backoffMs` before `continue`, but on `attempt === RETRY_MAX_ATTEMPTS` the loop condition fails immediately after — the sleep was pure waste. Added early-return (parallel to the existing pattern in the network-error catch branch) so the final attempt exits the retry loop at the first non-success response without extra latency. Tests: - 3 new `parseComtradeFlowResponse` variants: world-only filter, numeric-0 partnerCode shape, rows without partnerCode field - Existing tests updated: the double-count assertion replaced with a "per-partner rows must NOT sum into the world-aggregate total" assertion that pins the new contract Rebased onto updated prereq. Full test suite: 6,890 tests (+2 net).

greptile-apps Bot reviewed Apr 16, 2026

View reviewed changes

vercel Bot deployed to Preview April 16, 2026 11:42 View deployment

koala73 merged commit d1a3fdf into main Apr 16, 2026
10 checks passed

koala73 deleted the fix/portwatch-port-activity-timeout branch April 16, 2026 11:51

This was referenced Apr 20, 2026

fix(portwatch): per-country timeout + SIGTERM progress flush #3222

Merged

feat(portwatch): globalise EP3 — one paginated fetch, in-memory groupBy #3225

Merged

feat(seed): BUNDLE_RUN_STARTED_AT_MS env + runSeed SIGTERM cleanup #3384

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(portwatch): unblock port-activity seeder (global EP4 refs, conc 12, progress logs, SIGTERM)#3128

fix(portwatch): unblock port-activity seeder (global EP4 refs, conc 12, progress logs, SIGTERM)#3128
koala73 merged 2 commits into
mainfrom
fix/portwatch-port-activity-timeout

koala73 commented Apr 16, 2026

Uh oh!

vercel Bot commented Apr 16, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot commented Apr 16, 2026

Uh oh!

greptile-apps Bot Apr 16, 2026

Uh oh!

greptile-apps Bot Apr 16, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

koala73 commented Apr 16, 2026

Summary

Test plan

Uh oh!

vercel Bot commented Apr 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

greptile-apps Bot commented Apr 16, 2026

Greptile Summary

Confidence Score: 4/5

Important Files Changed

Sequence Diagram

Uh oh!

greptile-apps Bot Apr 16, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Apr 16, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

vercel Bot commented Apr 16, 2026 •

edited

Loading