fix(health): bump gdeltIntel maxStaleMin 420→720 (deploy-jitter)#3671
Conversation
…jitter)
WM 2026-05-10 18:02 UTC: UptimeRobot flipped DOWN→UP within 1 minute,
problems list showed `resilienceIntervals:STALE_SEED(722min)`. Diagnosed
via /api/health failure-log + Railway cron history:
04:04 UTC ✓ (10s)
10:00 UTC ✓ (3s)
16:00 UTC ✗ NO ROW — Railway didn't fire the scheduled tick
22:03 UTC ✓ (9s) ← caught up here
Multiple deploys today around 12:00 UTC (the missed slot was 12:00 UTC =
16:00 local in the user's TZ). Railway preempts scheduled cron ticks when
a deploy is in flight — normal platform behavior, not a code bug.
The 720min budget = 12h = 2× the real 6h cron cadence = exact gold-
standard floor with ZERO jitter tolerance. Any single missed tick (12h
gap) hits the threshold by definition. UptimeRobot polls every 60s and
fires a DOWN page on any WARNING — even one that resolves within 1
minute.
Bump 720 → 840 (14h ≈ 2.33× cadence):
- tolerates 1 missed tick (12h gap) + ~2h deploy jitter
- still alerts at 2 missed ticks (18h gap = real outage)
- upper bound (1080) stays untouched per existing comment's note that
1080 = "over-permissive: masks a 12h outage"
Same pattern as PR #3640 (correlationCards 15→30 and hyperliquidFlow
15→30): pre-fix all three sat at the exact 2-3× safety floor with no
margin for cron jitter. Each was independently chosen for tight
detection at design time; each independently produced UptimeRobot
false-positives in practice.
Tests:
- Updated value-pin assertion (720 → 840) + comment with the 2026-05-10
incident as the new prior-value entry
- Existing bounds checks (>= 540, <= 1080) unchanged and still passing
- 13/13 resilience tests pass; 18/18 health-content-age tests pass
…P2×3) Greptile PR #3652 review surfaced three valid P2 issues in the test file, all stemming from the budget bump 720→840: P2 — Top-of-describe context comment (line 203-205) still read "720 = 12h staleness = 2 missed cron ticks". Updated to reflect the new 840 ≈ 14h with explicit jitter rationale. P2 — Bounds-check failure message (line 286-287) still said the project convention was "720 here, matching resilienceRanking". Updated to "840 (≈2.33× cadence, matches resilienceRanking)" so a future test failure reports the correct reference value. P2 — Missing value-pin assertion for resilienceRanking. api/health.js explicitly notes both keys MUST stay in sync ("written by the SAME Resilience-Scores section"). Without an assertion enforcing parity, a future one-off edit to resilienceRanking would silently diverge from resilienceIntervals and bypass the contract described in the comments. Added a fifth test in the describe block: it('resilienceRanking.maxStaleMin matches resilienceIntervals ...', () => { assert.equal(rankingBudget, intervalsBudget, ...); }); This catches divergence regardless of whether someone bumps just one of the two — the assertion enforces equality, not a hardcoded value, so the next legitimate bump only needs to touch both api/health.js values (the test passes as long as they match). 14/14 tests pass (was 13, +1 for the parity assertion).
… as #3652) 2026-05-12 15:20 UTC: UptimeRobot flipped DOWN→UP within ~1 minute, failure-log showed `gdeltIntel:STALE_SEED(421min)` → cleared, then hit again at 467min. Same Railway-deploy-preempted-tick pattern as resilienceIntervals on 2026-05-10 (PR #3652). Root cause: maxStaleMin=420 at 6h cron cadence = 1.16× ratio with virtually zero jitter margin. Any single missed cron tick (deploy preempts a scheduled fire) produces a >7h gap → flips STALE_SEED for the ~1 minute until the next scheduled run catches up. Bumping to 720 (12h = 2× cadence) matches the gold-standard project convention: - tolerates 1 missed tick (12h gap) at 6h cadence - alerts at 2 missed ticks (18h gap) — real outage signal preserved - upstream CACHE_TTL is 24h, so per-topic merge always has a prior snapshot even at the upper end of the new budget No tests pin this value, so the only change is the production constant + the audit-trail comment. Fifth instance of the same class today/this week (correlationCards + hyperliquidFlow in #3640, resilienceIntervals/Ranking in #3652, now gdeltIntel). Pattern: knobs sized at the exact safety floor have no room for routine Railway deploy jitter.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Greptile SummaryThis PR addresses a fifth recurrence of the Railway deploy-jitter / missed-cron-tick false-positive pattern by bumping
Confidence Score: 4/5Safe to merge; all three config changes are well-justified by observed incident data and the math checks out. The resilience key changes (720→840) appear in both this diff and companion PR #3652 — whoever lands second will hit a conflict or a no-op on those two lines. Worth confirming landing order before merging. api/health.js lines 380-381 (resilienceRanking / resilienceIntervals overlap with #3652). Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[Railway cron fires\nevery 6h] --> B{seed-gdelt-intel.mjs\nruns OK?}
B -- Yes --> C[Write seed-meta key\nfetch time = now]
B -- No / Deploy preempts tick --> D[Tick missed\nseedAge += 6h]
C --> E[health.js check:\nseedAgeMin vs maxStaleMin]
D --> E
E -- seedAge < 720min --> F[Status: OK]
E -- 720 <= seedAge < 1080min --> G[Status: STALE_SEED\n1 missed tick + jitter]
E -- seedAge >= 1080min --> H[Status: STALE_SEED\n2+ missed ticks = real outage]
Reviews (1): Last reviewed commit: "fix(health): bump gdeltIntel maxStaleMin..." | Re-trigger Greptile |
| resilienceRanking: { key: 'seed-meta:resilience:ranking', maxStaleMin: 840 }, // RPC cache (12h TTL, refreshed every 6h by seed-resilience-scores cron); 14h budget tolerates 1 missed tick (12h gap) + ~2h jitter for in-flight deploys that preempt a scheduled tick; alerts at 2 missed ticks (18h gap). Bumped from 720 — see comment below. | ||
| resilienceIntervals: { key: 'seed-meta:resilience:intervals', maxStaleMin: 840 }, // bundled into seed-bundle-resilience, written by the Resilience-Scores section. Real Railway cron is `0 */6 * * *` (every 6h on the hour, UTC) — empirically verified 2026-04-28 via Railway logs showing 6h gaps between successful runs (the prior `intervalMs=2h with hourly fires` claim did not match what's deployed; either the bundle interval gate or the Railway service schedule makes the effective cadence 6h). 840 = 14h staleness ≈ 2.33× cadence: tolerates 1 missed tick (12h gap) + ~2h jitter for in-flight deploys; alerts at 2 missed ticks (18h gap). Matches resilienceRanking above (same Resilience-Scores section writes both). Prior values: 20160 (14d, 168× — silent on real outage), 1080 (18h, 3× — over-permissive: masks a 12h outage), 720 (12h, 2× — exact floor; flipped UptimeRobot WARNING for ~1min on every Railway-deploy-preempted tick: 2026-05-10 incident at 18:02 UTC, seedAgeMin=722 vs maxStale=720 after the 12:00 UTC tick was skipped during an in-flight deploy), 360 (1× — false-positive on routine jitter, 2026-04-28: seedAgeMin=367 vs maxStale=360). Re-tighten ONLY if/when the actual Railway cron schedule is verified sub-6h. |
There was a problem hiding this comment.
Scope of this diff exceeds PR title
The PR title and description table attribute the resilienceRanking / resilienceIntervals 720→840 bumps to companion PR #3652, but those lines are also changed here (the base commit still shows 720 for both). If #3652 merges to main before this branch is rebased, these two lines will conflict; if this PR lands first, #3652 will arrive as a no-op (or conflict) for those same lines. The final state is correct either way, but the landing-order dependency isn't reflected in the PR title, which could surprise whoever is sequencing the merges.
Summary
Fifth occurrence of the same class of issue we've been ironing out this week: a tunable knob sized at the exact safety floor with no room for Railway deploy jitter. One-line config change.
Incident
2026-05-12 ~15:20 UTC: UptimeRobot flipped DOWN→UP within ~1 minute.
health:failure-logshowed:At time of investigation (16:06 UTC), seedAgeMin was 467 min (7.8h).
Root cause: same pattern as #3652 (resilienceIntervals)
seed-gdelt-intel.mjsis a standalone Railway cron (not in anyseed-bundle-*). Railway preempts scheduled cron ticks during in-flight deploys — normal platform behavior. With a 1.16× ratio, ANY single missed tick produces a >7h gap → STALE_SEED for the ~1 minute until the next scheduled run catches up.We've now seen this exact failure mode on five separate keys:
Fix
Math:
CACHE_TTLis 24h so the per-topic merge always has a prior snapshot even at the upper end of the new budgetWhy 720 and not 840 (matching the resilience pattern)
Both work. 720 is the cleaner "exactly 2× cadence" match for a 6h cron. 840 was chosen for resilienceIntervals because that key's incident showed 12h-gap + ~2h-deploy-jitter could happen back-to-back. For gdeltIntel I'm starting with 720 since the cron isn't bundled (so it doesn't share a deploy window with sibling seeders) and the observed gap was 7.8h (well within 12h). If we see another flip post-merge, the same shape of bump to 840 is a one-liner.
Test plan
node --check api/health.js— syntax OKnpx esbuild api/health.js --bundle --format=esm --platform=browser— bundles cleanly for the edge runtimenode --test tests/health-content-age.test.mjs— 18/18 pass (no test was pinning this value)/api/health?history=1for the next ~24h. Expectation: zero gdeltIntel-driven flips when a deploy preempts a single cron tick; flip CORRECTLY fires only if 2+ consecutive ticks miss.Companion
Five PRs from this week's series, in dependency-order for cleanest landing: