feat(seed): BUNDLE_RUN_STARTED_AT_MS env + runSeed SIGTERM cleanup by koala73 · Pull Request #3384 · koala73/worldmonitor

koala73 · 2026-04-24T19:08:22Z

Summary

Two coupled infrastructure changes that prereq the Comtrade re-export-share seeder (plan 2026-04-24-003). Both are broadly usable by any cohort seeder with a freshness-dependent consumer.

_bundle-runner.mjs — inject BUNDLE_RUN_STARTED_AT_MS env into every spawned child. All siblings in a single bundle run share one value (captured at runBundle start, not spawn time). Consumers can use this to detect stale peer keys — if a peer's seed-meta predates the current bundle run, fall back to a hard default instead of reading a cohort-peer's last-week output.
_seed-utils.mjs::runSeed — process.once('SIGTERM') cleanup handler. _bundle-runner.mjs sends SIGTERM on section timeout, then SIGKILL after KILL_GRACE_MS (5s). Without this handler the existing finally-based cleanup never runs on SIGKILL, leaking the 30-min acquireLock reservation — the next cron tick silently skips the resource (root cause captured in memory bundle-runner-sigkill-leaks-child-lock, PR fix(portwatch): unblock port-activity seeder (global EP4 refs, conc 12, progress logs, SIGTERM) #3128). Handler releases the lock and extends existing-data TTL before process.exit(143).

Test plan

tests/bundle-runner.test.mjs — two new tests:
- env injection (value within run bounds)
- sibling sections share the same timestamp (critical for the consumer freshness guard)
tests/seed-utils-sigterm-cleanup.test.mjs — two new tests:
- runSeed SIGTERM path exits 143, logs "SIGTERM received — releasing lock"
- process.once contract: second SIGTERM does not re-enter handler
Full npm run test:data passes (6,862 tests)
npm run typecheck clean
npm run lint — no new errors (207 pre-existing warnings unchanged)

Post-Deploy Monitoring & Validation

What to monitor/search
- Logs: search Railway for SIGTERM received — releasing lock on any bundle seeder. Each occurrence = a section that timed out and cleaned up (expected, desired behavior).
- Metrics: api/health.js — no resource should flip to STALE_SEED after a bundle run completes.
Validation checks
- curl https://worldmonitor.app/api/health | jq '.bundles.resilienceRecovery' — after one monthly run, recoverySovereignWealth fetchedAt should be < 24h.
Expected healthy behavior
- Zero SIGTERM cleanup log lines on timely sections (normal case). SIGTERM lines appear only when a section blows its timeout — same population as before this PR, but cleanup now actually releases locks.
Failure signal / rollback trigger
- SIGTERM cleanup error log line (caught exception in handler) — safe to roll back; existing TTL-based recovery still works.
Validation window & owner
- Window: next weekly resilience-recovery bundle run (~7 days). Owner: @koala73.

🤖 Generated with Claude Opus 4.7 (1M context, extended thinking) via Claude Code + Compound Engineering v2.49.0

Prereq for the re-export-share Comtrade seeder (plan 2026-04-24-003), usable by any cohort seeder whose consumer needs bundle-level freshness. Two coupled changes: 1. `_bundle-runner.mjs` injects `BUNDLE_RUN_STARTED_AT_MS` into every spawned child. All siblings in a single bundle run share one value (captured at `runBundle` start, not spawn time). Consumers use this to detect stale peer keys — if a peer's seed-meta predates the current bundle run, fall back to a hard default rather than read a cohort-peer's last-week output. 2. `_seed-utils.mjs::runSeed` registers a `process.once('SIGTERM')` handler that releases the acquired lock and extends existing-data TTL before exiting 143. `_bundle-runner.mjs` sends SIGTERM on section timeout, then SIGKILL after KILL_GRACE_MS (5s). Without this handler the `finally` path never runs on SIGKILL, leaving the 30-min acquireLock reservation in place until its own TTL expires — the next cron tick silently skips the resource. Regression guard memory: `bundle-runner-sigkill-leaks-child-lock` (PR #3128 root cause). Tests added: - bundle-runner env injection (value within run bounds) - sibling sections share the same timestamp (critical for the consumer freshness guard) - runSeed SIGTERM path: exit 143 + cleanup log - process.once contract: second SIGTERM does not re-enter handler

vercel · 2026-04-24T19:08:28Z

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
worldmonitor	Ignored	Preview	Apr 24, 2026 8:08pm

greptile-apps · 2026-04-24T19:12:50Z

Greptile Summary

This PR adds two complementary infrastructure features for bundle seeders: a shared BUNDLE_RUN_STARTED_AT_MS env var (captured once at runBundle start) for downstream freshness guards, and a process.once('SIGTERM', …) handler in runSeed that releases the lock and extends existing-data TTL before exiting 143 — fixing the lock-leak on SIGKILL described in PR #3128.

The SIGTERM handler is not deregistered on the error-propagation catch path (throw err, lines 1014-1016); a SIGTERM arriving in that narrow window will call extendExistingTtl on potentially stale keys and exit 143 instead of propagating the error (P2).
The seed-utils-sigterm-cleanup test only asserts exit code and log message; it does not verify that the Redis DEL and EXPIRE calls are actually emitted, despite the in-code comment claiming otherwise (P2).

Confidence Score: 4/5

Safe to merge; both findings are P2 — the SIGTERM race window is narrow and low-impact, and the missing test assertions don't affect runtime correctness.

No P0/P1 issues found. Two P2s: a missing removeListener on the error-propagation path and incomplete test assertions. The core lock-release and env-injection logic is correct and well-tested for the primary timeout scenario.

scripts/_seed-utils.mjs (SIGTERM handler teardown on catch path) and tests/seed-utils-sigterm-cleanup.test.mjs (Redis call assertions)

Important Files Changed

Filename	Overview
scripts/_bundle-runner.mjs	Captures bundle-start timestamp (t0) once and threads it through spawnSeed as BUNDLE_RUN_STARTED_AT_MS env var for all sibling sections; fallback ?? Date.now() is only reachable if spawnSeed were called directly (it's private), so the invariant holds in practice.
scripts/_seed-utils.mjs	Adds SIGTERM handler after lock acquisition; uses process.once and process.exit(143) in finally — correct for the timeout case, but handler is not deregistered on the error-propagation (throw err) path, leaving a narrow window where a late SIGTERM could extend TTL on stale data or exit 143 instead of propagating the error.
tests/bundle-runner.test.mjs	Two new tests for env injection; the sibling-sharing test correctly relies on sequential section execution (await spawnSeed) so identical t0 values are guaranteed; assertions are thorough.
tests/seed-utils-sigterm-cleanup.test.mjs	READY-signal coordination correctly ensures the SIGTERM handler is registered before the signal is sent; however assertions only verify exit code and log message, not that the Redis DEL/EXPIRE operations were actually issued.

Sequence Diagram

sequenceDiagram
    participant BR as _bundle-runner.mjs
    participant Child as seed child (runSeed)
    participant Redis as Upstash Redis

    BR->>BR: t0 = Date.now() (once, at runBundle start)
    loop each section
        BR->>Child: spawn(scriptPath, env={BUNDLE_RUN_STARTED_AT_MS: t0})
        Child->>Redis: acquireLock (SET NX PX lockTtlMs)
        Child->>Child: process.once('SIGTERM', sigTermHandler)
        Child->>Child: await fetchFn() [Phase 1]

        alt normal completion
            Child->>Redis: atomicPublish + writeFreshnessMetadata
            Child->>Redis: releaseLock (EVAL Lua)
            Child->>Child: process.exit(0)
        else section timeout
            BR->>Child: SIGTERM (after timeoutMs)
            Child->>Child: sigTermHandler fires
            Child->>Redis: releaseLock (EVAL Lua)
            Child->>Redis: extendExistingTtl (EXPIRE pipeline)
            Child->>Child: process.exit(143)
            BR->>Child: SIGKILL (after KILL_GRACE_MS=5s, if still alive)
        end
    end
    BR->>BR: emit bundle summary (ran/skipped/failed)

_{Reviews (1): Last reviewed commit: "feat(seed): BUNDLE_RUN_STARTED_AT_MS env..." | Re-trigger Greptile}

greptile-apps · 2026-04-24T19:12:58Z

+  const { code, signal, stderr } = await runFixture(body);
+  // process.exit(143) should produce code=143; on some platforms Node maps it
+  // back to a signal termination, so accept either code or signal.
+  assert.ok(code === 143 || signal === 'SIGTERM',
+    `expected exit 143 or SIGTERM; got code=${code} signal=${signal}\nstderr:\n${stderr}`);


Test assertions don't verify the Redis operations

The test file-header comment (and the PR description) both claim this test verifies that releaseLock (DEL) and extendExistingTtl (EXPIRE pipeline) are actually called before exit. The fixture logs every fetch call via console.log('FETCH …'), but the two assert statements only check the exit code and the SIGTERM log line — they never inspect stdout for the expected FETCH DEL or FETCH pipeline lines. If either Redis operation were silently skipped or swallowed by a bug, the test would still pass.

// After the existing assert.match on stderr, add: assert.match(stdout, /FETCH pipeline/, `expected extendExistingTtl EXPIRE pipeline call; stdout:\n${stdout}`); assert.match(stdout, /FETCH eval/, `expected releaseLock EVAL call; stdout:\n${stdout}`);

greptile-apps · 2026-04-24T19:13:00Z

+  };
+  process.once('SIGTERM', sigTermHandler);


SIGTERM handler not deregistered on error-propagation path

The process.once listener is never explicitly removed. All normal exit paths call process.exit(0) so this is harmless there, but the catch block at the bottom of runSeed (lines 1014-1016) calls releaseLock and then throw err — it does NOT call process.exit. If a SIGTERM arrives in that narrow window (lock already released, error about to propagate), the handler fires, calls extendExistingTtl on potentially stale / partially-written keys, and exits 143 instead of letting the rejection propagate with the correct exit code. Adding process.removeListener('SIGTERM', sigTermHandler) before the throw would eliminate the ambiguity.

} catch (err) { process.removeListener('SIGTERM', sigTermHandler); await releaseLock(`${domain}:${resource}`, runId); throw err; }

Addresses PR #3384 review findings (todos 256, 257, 259, 260): #256 (P1) — SIGTERM handler narrowed to fetch phase only. Was installed at runSeed entry and armed through every `process.exit` path; could race `emptyDataIsFailure: true` strict-floor exits (IMF-External, WB-bulk) and extend seed-meta TTL when the contract forbids it — silently re-masking 30-day outages. Now the handler is attached immediately before `withRetry(fetchFn)` and removed in a try/finally that covers all fetch-phase exit branches. #257 (P1) — `BUNDLE_RUN_STARTED_AT_MS` now has a first-class helper. Exported `getBundleRunStartedAtMs()` from `_seed-utils.mjs` with JSDoc describing the bundle-freshness contract. Fleet-wide helper so the next consumer seeder imports instead of rediscovering the idiom. #259 (P2) — SIGTERM cleanup runs `Promise.allSettled` on disjoint-key ops (`releaseLock` + `extendExistingTtl`). Serialising compounded Upstash latency during the exact failure mode (Redis degraded) this handler exists to handle, risking breach of the 5s SIGKILL grace. #260 (P2) — `_bundle-runner.mjs` asserts topological order on optional `dependsOn` section field. Throws on unknown-label refs and on deps appearing at a later index. Fleet-wide contract replacing the previous prose-comment ordering guarantee. Tests added/updated: - New: SIGTERM handler removed after fetchFn completes (narrowed-scope contract — post-fetch SIGTERM must NOT trigger TTL extension) - New: dependsOn unknown-label + out-of-order + happy-path (3 tests) Full test suite: 6,866 tests pass (+4 net).

…read Plan ref: docs/plans/2026-04-24-003-feat-reexport-share-comtrade-seeder-plan.md Motivating case. Before this PR, the SWF `rawMonths` denominator for the `sovereignFiscalBuffer` dimension used GROSS annual imports for every country. For re-export hubs (goods transiting without domestic settlement), this structurally under-reports resilience: UAE's 2023 $941B of imports include $334B of transit flow that never represents domestic consumption. Net imports = gross × (1 − reexport_share). The previous (PR 3A) design flattened a hand-curated YAML into Redis; the YAML shipped empty and never populated, so the correction never applied and the cohort audit showed no movement. Gap #2 (this PR). Two coupled changes to make the correction actually apply: 1. Comtrade-backed seeder (`scripts/seed-recovery-reexport-share.mjs`). Rewritten to fetch UN Comtrade `flowCode=RX` (re-exports) and `flowCode=M` (imports) per cohort member, compute share = RX/M at the latest co-populated year, clamp to [0.05, 0.95], publish the envelope. Header auth (`Ocp-Apim-Subscription-Key`) — subscription key never reaches URL/logs/Redis. `maxRecords=250000` cap with truncation detection. Sequential + retry-on-429 with backoff. Hub cohort resolved by Phase 0 empirical probe (plan §Phase 0): ['AE', 'PA']. Six candidates (SG/HK/NL/BE/MY/LT) return HTTP 200 with zero RX rows — Comtrade doesn't expose RX for those reporters. 2. SWF seeder reads from Redis (`scripts/seed-sovereign-wealth.mjs`). Swaps `loadReexportShareByCountry()` (YAML) for `loadReexportShareFromRedis()` (Redis key written by #1). Guarded by bundle-run freshness: if the sibling Reexport-Share seeder's `seed-meta` predates `BUNDLE_RUN_STARTED_AT_MS` (set by the prereq PR's `_bundle-runner.mjs` env-injection), HARD fallback to gross imports rather than apply last-month's stale share. Health registries. Both new keys registered in BOTH `api/health.js` SEED_META (60-day alert threshold) and `api/seed-health.js` SEED_DOMAINS (43200min interval). feedback_two_health_endpoints_must_match. Bundle wiring. `seed-bundle-resilience-recovery` Reexport-Share timeout bumped 60s → 300s (Comtrade + retry can take 2-3 min worst-case). Ordering preserved: Reexport-Share before Sovereign- Wealth so the SWF seeder reads a freshly-written key in the same cron tick. Deletions. YAML + loader + 7 obsolete loader tests removed; single source of truth is now Comtrade → Redis. Prereq. Stacks on PR #3384 (feat/bundle-runner-env-sigterm) which adds BUNDLE_RUN_STARTED_AT_MS env injection + runSeed SIGTERM cleanup. This PR's bundle-freshness guard depends on that env variable. Tests (19 new, 7 deleted, +12 net): - Pure math: parseComtradeFlowResponse, computeShareFromFlows, clampShare, declareRecords + credential-leak source scan (15) - Integration (Gap #2 regression guards): SWF seeder loadReexport ShareFromRedis — fresh/absent/malformed/stale-meta/missing-meta (5) - Health registry dual-registry drift guard — scoped to this PR's keys, respecting pre-existing asymmetry (4) - Bundle-ordering + timeout assertions (2) Phase 0 cohort validation committed to plan. Full test suite passes: 6,881 tests.

Review follow-up: the earlier `Math.floor(Date.now()/1000)*1000` fallback regressed standalone (non-bundle) runs. A consumer seeder invoked manually just after its peer wrote `fetchedAt = (now - 5s)` would see `bundleStartMs = Date.now()`, reject the perfectly-fresh peer envelope as "stale", and fall back to defaults — defeating the point of the peer-read path outside the bundle. Returning null when `BUNDLE_RUN_STARTED_AT_MS` is unset/invalid keeps the freshness gate scoped to its real purpose (across-bundle-tick staleness) and lets standalone runs skip the gate entirely. Consumers check `bundleStartMs != null` before applying the comparison; see the companion `seed-sovereign-wealth.mjs` change on the stacked PR.

…read Plan ref: docs/plans/2026-04-24-003-feat-reexport-share-comtrade-seeder-plan.md Motivating case. Before this PR, the SWF `rawMonths` denominator for the `sovereignFiscalBuffer` dimension used GROSS annual imports for every country. For re-export hubs (goods transiting without domestic settlement), this structurally under-reports resilience: UAE's 2023 $941B of imports include $334B of transit flow that never represents domestic consumption. Net imports = gross × (1 − reexport_share). The previous (PR 3A) design flattened a hand-curated YAML into Redis; the YAML shipped empty and never populated, so the correction never applied and the cohort audit showed no movement. Gap #2 (this PR). Two coupled changes to make the correction actually apply: 1. Comtrade-backed seeder (`scripts/seed-recovery-reexport-share.mjs`). Rewritten to fetch UN Comtrade `flowCode=RX` (re-exports) and `flowCode=M` (imports) per cohort member, compute share = RX/M at the latest co-populated year, clamp to [0.05, 0.95], publish the envelope. Header auth (`Ocp-Apim-Subscription-Key`) — subscription key never reaches URL/logs/Redis. `maxRecords=250000` cap with truncation detection. Sequential + retry-on-429 with backoff. Hub cohort resolved by Phase 0 empirical probe (plan §Phase 0): ['AE', 'PA']. Six candidates (SG/HK/NL/BE/MY/LT) return HTTP 200 with zero RX rows — Comtrade doesn't expose RX for those reporters. 2. SWF seeder reads from Redis (`scripts/seed-sovereign-wealth.mjs`). Swaps `loadReexportShareByCountry()` (YAML) for `loadReexportShareFromRedis()` (Redis key written by #1). Guarded by bundle-run freshness: if the sibling Reexport-Share seeder's `seed-meta` predates `BUNDLE_RUN_STARTED_AT_MS` (set by the prereq PR's `_bundle-runner.mjs` env-injection), HARD fallback to gross imports rather than apply last-month's stale share. Health registries. Both new keys registered in BOTH `api/health.js` SEED_META (60-day alert threshold) and `api/seed-health.js` SEED_DOMAINS (43200min interval). feedback_two_health_endpoints_must_match. Bundle wiring. `seed-bundle-resilience-recovery` Reexport-Share timeout bumped 60s → 300s (Comtrade + retry can take 2-3 min worst-case). Ordering preserved: Reexport-Share before Sovereign- Wealth so the SWF seeder reads a freshly-written key in the same cron tick. Deletions. YAML + loader + 7 obsolete loader tests removed; single source of truth is now Comtrade → Redis. Prereq. Stacks on PR #3384 (feat/bundle-runner-env-sigterm) which adds BUNDLE_RUN_STARTED_AT_MS env injection + runSeed SIGTERM cleanup. This PR's bundle-freshness guard depends on that env variable. Tests (19 new, 7 deleted, +12 net): - Pure math: parseComtradeFlowResponse, computeShareFromFlows, clampShare, declareRecords + credential-leak source scan (15) - Integration (Gap #2 regression guards): SWF seeder loadReexport ShareFromRedis — fresh/absent/malformed/stale-meta/missing-meta (5) - Health registry dual-registry drift guard — scoped to this PR's keys, respecting pre-existing asymmetry (4) - Bundle-ordering + timeout assertions (2) Phase 0 cohort validation committed to plan. Full test suite passes: 6,881 tests.

Greptile review P2 on PR #3384: the existing test only asserted exit code + log line, not that the Redis ops were actually issued. The log claim was ahead of the test. Fixture now logs every Upstash fetch call's shape (EVAL / pipeline- EXPIRE / other) to stderr. Test asserts: - >=1 EVAL op was issued during SIGTERM cleanup (releaseLock Lua script on the lock key) - >=1 pipeline-EXPIRE op was issued (extendExistingTtl on canonical + seed-meta keys) - The EVAL body carries the runSeed-generated runId (proves it's THIS run's release, not a phantom op) - The EXPIRE pipeline touches both the canonicalKey AND the seed-meta key (proves the keys[] array was built correctly including the extraKeys merge path) Full test suite: 6,866 tests pass, typecheck clean.

…read Plan ref: docs/plans/2026-04-24-003-feat-reexport-share-comtrade-seeder-plan.md Motivating case. Before this PR, the SWF `rawMonths` denominator for the `sovereignFiscalBuffer` dimension used GROSS annual imports for every country. For re-export hubs (goods transiting without domestic settlement), this structurally under-reports resilience: UAE's 2023 $941B of imports include $334B of transit flow that never represents domestic consumption. Net imports = gross × (1 − reexport_share). The previous (PR 3A) design flattened a hand-curated YAML into Redis; the YAML shipped empty and never populated, so the correction never applied and the cohort audit showed no movement. Gap #2 (this PR). Two coupled changes to make the correction actually apply: 1. Comtrade-backed seeder (`scripts/seed-recovery-reexport-share.mjs`). Rewritten to fetch UN Comtrade `flowCode=RX` (re-exports) and `flowCode=M` (imports) per cohort member, compute share = RX/M at the latest co-populated year, clamp to [0.05, 0.95], publish the envelope. Header auth (`Ocp-Apim-Subscription-Key`) — subscription key never reaches URL/logs/Redis. `maxRecords=250000` cap with truncation detection. Sequential + retry-on-429 with backoff. Hub cohort resolved by Phase 0 empirical probe (plan §Phase 0): ['AE', 'PA']. Six candidates (SG/HK/NL/BE/MY/LT) return HTTP 200 with zero RX rows — Comtrade doesn't expose RX for those reporters. 2. SWF seeder reads from Redis (`scripts/seed-sovereign-wealth.mjs`). Swaps `loadReexportShareByCountry()` (YAML) for `loadReexportShareFromRedis()` (Redis key written by #1). Guarded by bundle-run freshness: if the sibling Reexport-Share seeder's `seed-meta` predates `BUNDLE_RUN_STARTED_AT_MS` (set by the prereq PR's `_bundle-runner.mjs` env-injection), HARD fallback to gross imports rather than apply last-month's stale share. Health registries. Both new keys registered in BOTH `api/health.js` SEED_META (60-day alert threshold) and `api/seed-health.js` SEED_DOMAINS (43200min interval). feedback_two_health_endpoints_must_match. Bundle wiring. `seed-bundle-resilience-recovery` Reexport-Share timeout bumped 60s → 300s (Comtrade + retry can take 2-3 min worst-case). Ordering preserved: Reexport-Share before Sovereign- Wealth so the SWF seeder reads a freshly-written key in the same cron tick. Deletions. YAML + loader + 7 obsolete loader tests removed; single source of truth is now Comtrade → Redis. Prereq. Stacks on PR #3384 (feat/bundle-runner-env-sigterm) which adds BUNDLE_RUN_STARTED_AT_MS env injection + runSeed SIGTERM cleanup. This PR's bundle-freshness guard depends on that env variable. Tests (19 new, 7 deleted, +12 net): - Pure math: parseComtradeFlowResponse, computeShareFromFlows, clampShare, declareRecords + credential-leak source scan (15) - Integration (Gap #2 regression guards): SWF seeder loadReexport ShareFromRedis — fresh/absent/malformed/stale-meta/missing-meta (5) - Health registry dual-registry drift guard — scoped to this PR's keys, respecting pre-existing asymmetry (4) - Bundle-ordering + timeout assertions (2) Phase 0 cohort validation committed to plan. Full test suite passes: 6,881 tests.

koala73 · 2026-04-24T20:10:43Z

Addressed both P2s from the Greptile review:

SIGTERM handler not deregistered on error path → fixed in commit 7577e618a (then refined in 31a7c4b57 and fa208be7a). Handler is now scoped to the fetch phase only via try { process.once(...); await withRetry(fetchFn); } catch { process.off(...); ... } finally { process.off(...); } — removed on all exit paths including the bottom-of-runSeed rethrow. Also narrows the race window that could corrupt emptyDataIsFailure: true strict-floor semantics. New test SIGTERM handler is removed AFTER fetchFn completes pins the contract.

Test assertions don't verify Redis ops → fixed in commit fa208be7a. Fixture now logs every Upstash fetch-call shape (EVAL / pipeline-EXPIRE / other) to stderr; test asserts at least one EVAL (releaseLock Lua) and at least one pipeline-EXPIRE (extendExistingTtl) were issued, that the EVAL body carries the runSeed runId (proves it's THIS run's release), and that the EXPIRE pipeline touches both the canonical key and the seed-meta key.

Full suite: 6,866 tests pass, typecheck clean.

…read (#3385) * feat(seed): BUNDLE_RUN_STARTED_AT_MS env + runSeed SIGTERM cleanup Prereq for the re-export-share Comtrade seeder (plan 2026-04-24-003), usable by any cohort seeder whose consumer needs bundle-level freshness. Two coupled changes: 1. `_bundle-runner.mjs` injects `BUNDLE_RUN_STARTED_AT_MS` into every spawned child. All siblings in a single bundle run share one value (captured at `runBundle` start, not spawn time). Consumers use this to detect stale peer keys — if a peer's seed-meta predates the current bundle run, fall back to a hard default rather than read a cohort-peer's last-week output. 2. `_seed-utils.mjs::runSeed` registers a `process.once('SIGTERM')` handler that releases the acquired lock and extends existing-data TTL before exiting 143. `_bundle-runner.mjs` sends SIGTERM on section timeout, then SIGKILL after KILL_GRACE_MS (5s). Without this handler the `finally` path never runs on SIGKILL, leaving the 30-min acquireLock reservation in place until its own TTL expires — the next cron tick silently skips the resource. Regression guard memory: `bundle-runner-sigkill-leaks-child-lock` (PR #3128 root cause). Tests added: - bundle-runner env injection (value within run bounds) - sibling sections share the same timestamp (critical for the consumer freshness guard) - runSeed SIGTERM path: exit 143 + cleanup log - process.once contract: second SIGTERM does not re-enter handler * fix(seed): address P1/P2 review findings on SIGTERM + bundle contracts Addresses PR #3384 review findings (todos 256, 257, 259, 260): #256 (P1) — SIGTERM handler narrowed to fetch phase only. Was installed at runSeed entry and armed through every `process.exit` path; could race `emptyDataIsFailure: true` strict-floor exits (IMF-External, WB-bulk) and extend seed-meta TTL when the contract forbids it — silently re-masking 30-day outages. Now the handler is attached immediately before `withRetry(fetchFn)` and removed in a try/finally that covers all fetch-phase exit branches. #257 (P1) — `BUNDLE_RUN_STARTED_AT_MS` now has a first-class helper. Exported `getBundleRunStartedAtMs()` from `_seed-utils.mjs` with JSDoc describing the bundle-freshness contract. Fleet-wide helper so the next consumer seeder imports instead of rediscovering the idiom. #259 (P2) — SIGTERM cleanup runs `Promise.allSettled` on disjoint-key ops (`releaseLock` + `extendExistingTtl`). Serialising compounded Upstash latency during the exact failure mode (Redis degraded) this handler exists to handle, risking breach of the 5s SIGKILL grace. #260 (P2) — `_bundle-runner.mjs` asserts topological order on optional `dependsOn` section field. Throws on unknown-label refs and on deps appearing at a later index. Fleet-wide contract replacing the previous prose-comment ordering guarantee. Tests added/updated: - New: SIGTERM handler removed after fetchFn completes (narrowed-scope contract — post-fetch SIGTERM must NOT trigger TTL extension) - New: dependsOn unknown-label + out-of-order + happy-path (3 tests) Full test suite: 6,866 tests pass (+4 net). * fix(seed): getBundleRunStartedAtMs returns null outside a bundle run Review follow-up: the earlier `Math.floor(Date.now()/1000)*1000` fallback regressed standalone (non-bundle) runs. A consumer seeder invoked manually just after its peer wrote `fetchedAt = (now - 5s)` would see `bundleStartMs = Date.now()`, reject the perfectly-fresh peer envelope as "stale", and fall back to defaults — defeating the point of the peer-read path outside the bundle. Returning null when `BUNDLE_RUN_STARTED_AT_MS` is unset/invalid keeps the freshness gate scoped to its real purpose (across-bundle-tick staleness) and lets standalone runs skip the gate entirely. Consumers check `bundleStartMs != null` before applying the comparison; see the companion `seed-sovereign-wealth.mjs` change on the stacked PR. * test(seed): SIGTERM cleanup test now verifies Redis DEL + EXPIRE calls Greptile review P2 on PR #3384: the existing test only asserted exit code + log line, not that the Redis ops were actually issued. The log claim was ahead of the test. Fixture now logs every Upstash fetch call's shape (EVAL / pipeline- EXPIRE / other) to stderr. Test asserts: - >=1 EVAL op was issued during SIGTERM cleanup (releaseLock Lua script on the lock key) - >=1 pipeline-EXPIRE op was issued (extendExistingTtl on canonical + seed-meta keys) - The EVAL body carries the runSeed-generated runId (proves it's THIS run's release, not a phantom op) - The EXPIRE pipeline touches both the canonicalKey AND the seed-meta key (proves the keys[] array was built correctly including the extraKeys merge path) Full test suite: 6,866 tests pass, typecheck clean. * feat(resilience): Comtrade-backed re-export-share seeder + SWF Redis read Plan ref: docs/plans/2026-04-24-003-feat-reexport-share-comtrade-seeder-plan.md Motivating case. Before this PR, the SWF `rawMonths` denominator for the `sovereignFiscalBuffer` dimension used GROSS annual imports for every country. For re-export hubs (goods transiting without domestic settlement), this structurally under-reports resilience: UAE's 2023 $941B of imports include $334B of transit flow that never represents domestic consumption. Net imports = gross × (1 − reexport_share). The previous (PR 3A) design flattened a hand-curated YAML into Redis; the YAML shipped empty and never populated, so the correction never applied and the cohort audit showed no movement. Gap #2 (this PR). Two coupled changes to make the correction actually apply: 1. Comtrade-backed seeder (`scripts/seed-recovery-reexport-share.mjs`). Rewritten to fetch UN Comtrade `flowCode=RX` (re-exports) and `flowCode=M` (imports) per cohort member, compute share = RX/M at the latest co-populated year, clamp to [0.05, 0.95], publish the envelope. Header auth (`Ocp-Apim-Subscription-Key`) — subscription key never reaches URL/logs/Redis. `maxRecords=250000` cap with truncation detection. Sequential + retry-on-429 with backoff. Hub cohort resolved by Phase 0 empirical probe (plan §Phase 0): ['AE', 'PA']. Six candidates (SG/HK/NL/BE/MY/LT) return HTTP 200 with zero RX rows — Comtrade doesn't expose RX for those reporters. 2. SWF seeder reads from Redis (`scripts/seed-sovereign-wealth.mjs`). Swaps `loadReexportShareByCountry()` (YAML) for `loadReexportShareFromRedis()` (Redis key written by #1). Guarded by bundle-run freshness: if the sibling Reexport-Share seeder's `seed-meta` predates `BUNDLE_RUN_STARTED_AT_MS` (set by the prereq PR's `_bundle-runner.mjs` env-injection), HARD fallback to gross imports rather than apply last-month's stale share. Health registries. Both new keys registered in BOTH `api/health.js` SEED_META (60-day alert threshold) and `api/seed-health.js` SEED_DOMAINS (43200min interval). feedback_two_health_endpoints_must_match. Bundle wiring. `seed-bundle-resilience-recovery` Reexport-Share timeout bumped 60s → 300s (Comtrade + retry can take 2-3 min worst-case). Ordering preserved: Reexport-Share before Sovereign- Wealth so the SWF seeder reads a freshly-written key in the same cron tick. Deletions. YAML + loader + 7 obsolete loader tests removed; single source of truth is now Comtrade → Redis. Prereq. Stacks on PR #3384 (feat/bundle-runner-env-sigterm) which adds BUNDLE_RUN_STARTED_AT_MS env injection + runSeed SIGTERM cleanup. This PR's bundle-freshness guard depends on that env variable. Tests (19 new, 7 deleted, +12 net): - Pure math: parseComtradeFlowResponse, computeShareFromFlows, clampShare, declareRecords + credential-leak source scan (15) - Integration (Gap #2 regression guards): SWF seeder loadReexport ShareFromRedis — fresh/absent/malformed/stale-meta/missing-meta (5) - Health registry dual-registry drift guard — scoped to this PR's keys, respecting pre-existing asymmetry (4) - Bundle-ordering + timeout assertions (2) Phase 0 cohort validation committed to plan. Full test suite passes: 6,881 tests. * fix(resilience): address P1/P2 review findings — adopt shared helpers, pin freshness boundary Addresses PR #3385 review findings: #257 (P1) consumer — `seed-sovereign-wealth.mjs` imports the shared `getBundleRunStartedAtMs` helper from `_seed-utils.mjs` (added in the prereq commit) instead of its own `getBundleStartMs`. Single source of truth for the bundle-freshness contract. #258 (P2) — `seed-recovery-reexport-share.mjs` isMain guard uses the canonical `pathToFileURL(process.argv[1]).href === import.meta.url` form instead of basename-suffix matching. Handles symlinks, case- different paths on macOS HFS+, and Windows path separators without string munging. #260 (P2) consumer — Sovereign-Wealth declares `dependsOn: ['Reexport-Share']` in the bundle spec. `_bundle-runner.mjs` (prereq commit) now enforces topological order on load and throws on violation — replaces the previous prose-comment ordering contract. #261 (P2) — added a test to `tests/seed-sovereign-wealth-reads-redis- reexport-share.test.mts` pinning the inclusive-boundary semantic: `fetchedAtMs === bundleStartMs` must be treated as FRESH. Guards against a future refactor to `<=` that would silently reject peers writing at the very first millisecond of the bundle run. Rebased onto updated prereq. Full test suite: 6,886 tests pass (+5 net). * fix(resilience): freshness gate skipped in standalone mode; meta still required Review catch: the previous `bundleStartMs = Date.now()` fallback made standalone/manual `seed-sovereign-wealth.mjs` runs ALWAYS reject any previously-seeded re-export-share meta as "stale" — even when the operator ran the Reexport seeder milliseconds beforehand. Defeated the point of the peer-read path outside the bundle. With `getBundleRunStartedAtMs()` now returning null outside a bundle (companion commit on the prereq branch), the consumer only applies the freshness gate when `bundleStartMs != null`. Standalone runs accept any `fetchedAt` — the operator is responsible for ordering. Two guards survive the change: - Meta MUST exist (absence = peer-outage fail-safe, both modes) - In-bundle: meta MUST be at or after `BUNDLE_RUN_STARTED_AT_MS` Two new tests pin both modes: - standalone: accepts meta written 10 min before this process started - standalone: still rejects missing meta (peer-outage fail-safe survives gate bypass) Rebased onto updated prereq. Full test suite: 6,888 tests (+2 net). * fix(resilience): filter world-aggregate Comtrade rows + skip final-retry sleep Greptile review of PR #3385 flagged two P2s in the Comtrade seeder. Finding #3 (parseComtradeFlowResponse double-count risk): `cmdCode=TOTAL` without a partner filter currently returns only world-aggregate rows in practice — but `parseComtradeFlowResponse` summed every row unconditionally. A future refactor adding per- partner querying would silently double-count (world-aggregate row + partner-level rows for the same year), cutting the derived share in half with no test signal. Fix: explicit `partnerCode ∈ {'0', 0, null/undefined}` filter. Matches current empirical behavior (aggregate-only responses) and makes the construct robust to a future partner-level query. Finding #4 (wasted backoff on final retry): 429 and 5xx branches slept `backoffMs` before `continue`, but on `attempt === RETRY_MAX_ATTEMPTS` the loop condition fails immediately after — the sleep was pure waste. Added early-return (parallel to the existing pattern in the network-error catch branch) so the final attempt exits the retry loop at the first non-success response without extra latency. Tests: - 3 new `parseComtradeFlowResponse` variants: world-only filter, numeric-0 partnerCode shape, rows without partnerCode field - Existing tests updated: the double-count assertion replaced with a "per-partner rows must NOT sum into the world-aggregate total" assertion that pins the new contract Rebased onto updated prereq. Full test suite: 6,890 tests (+2 net).

greptile-apps Bot reviewed Apr 24, 2026

View reviewed changes

koala73 mentioned this pull request Apr 24, 2026

feat(resilience): Comtrade-backed re-export-share seeder + SWF Redis read #3385

Merged

8 tasks

vercel Bot deployed to Preview April 24, 2026 19:48 View deployment

koala73 merged commit 5f40f8a into main Apr 24, 2026
10 checks passed

koala73 deleted the feat/bundle-runner-env-sigterm branch April 24, 2026 20:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(seed): BUNDLE_RUN_STARTED_AT_MS env + runSeed SIGTERM cleanup#3384

feat(seed): BUNDLE_RUN_STARTED_AT_MS env + runSeed SIGTERM cleanup#3384
koala73 merged 4 commits into
mainfrom
feat/bundle-runner-env-sigterm

koala73 commented Apr 24, 2026

Uh oh!

vercel Bot commented Apr 24, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot commented Apr 24, 2026

Uh oh!

greptile-apps Bot Apr 24, 2026

Uh oh!

greptile-apps Bot Apr 24, 2026

Uh oh!

koala73 commented Apr 24, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

koala73 commented Apr 24, 2026

Summary

Test plan

Post-Deploy Monitoring & Validation

Uh oh!

vercel Bot commented Apr 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

greptile-apps Bot commented Apr 24, 2026

Greptile Summary

Confidence Score: 4/5

Important Files Changed

Sequence Diagram

Uh oh!

greptile-apps Bot Apr 24, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Apr 24, 2026

Choose a reason for hiding this comment

Uh oh!

koala73 commented Apr 24, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

vercel Bot commented Apr 24, 2026 •

edited

Loading