Skip to content

fix(health): bump gdeltIntel maxStaleMin 420→720 (deploy-jitter)#3671

Merged
koala73 merged 3 commits into
mainfrom
fix/gdelt-intel-stale-budget-420-720
May 13, 2026
Merged

fix(health): bump gdeltIntel maxStaleMin 420→720 (deploy-jitter)#3671
koala73 merged 3 commits into
mainfrom
fix/gdelt-intel-stale-budget-420-720

Conversation

@koala73

@koala73 koala73 commented May 12, 2026

Copy link
Copy Markdown
Owner

Summary

Fifth occurrence of the same class of issue we've been ironing out this week: a tunable knob sized at the exact safety floor with no room for Railway deploy jitter. One-line config change.

Incident

2026-05-12 ~15:20 UTC: UptimeRobot flipped DOWN→UP within ~1 minute. health:failure-log showed:

2026-05-12T15:20:34  WARNING  gdeltIntel:STALE_SEED(421min)

At time of investigation (16:06 UTC), seedAgeMin was 467 min (7.8h).

Root cause: same pattern as #3652 (resilienceIntervals)

gdeltIntel.maxStaleMin = 420 (7h)
cron interval         = 6h ("6h cron + 1h grace" per the prior inline comment)
ratio                 = 1.16× cadence ← virtually zero jitter margin

seed-gdelt-intel.mjs is a standalone Railway cron (not in any seed-bundle-*). Railway preempts scheduled cron ticks during in-flight deploys — normal platform behavior. With a 1.16× ratio, ANY single missed tick produces a >7h gap → STALE_SEED for the ~1 minute until the next scheduled run catches up.

We've now seen this exact failure mode on five separate keys:

PR Key Old budget Cron Old ratio New budget
#3640 correlationCards 15min 5min 3× (floor) 30min (6×)
#3640 hyperliquidFlow 15min 5min 3× (floor) 30min (6×)
#3652 resilienceIntervals 720min 6h 2× (floor) 840min (2.33×)
#3652 resilienceRanking 720min 6h 2× (floor) 840min (2.33×)
This PR gdeltIntel 420min 6h 1.16× (below floor!) 720min (2×)

Fix

- gdeltIntel: { ... maxStaleMin: 420 }, // 6h cron + 1h grace
+ gdeltIntel: { ... maxStaleMin: 720 }, // 6h cron; 12h = 2× cadence

Math:

  • Tolerates 1 missed tick (12h gap) at 6h cadence
  • Alerts at 2 missed ticks (18h gap) — real outage signal preserved
  • Upstream CACHE_TTL is 24h so the per-topic merge always has a prior snapshot even at the upper end of the new budget

Why 720 and not 840 (matching the resilience pattern)

Both work. 720 is the cleaner "exactly 2× cadence" match for a 6h cron. 840 was chosen for resilienceIntervals because that key's incident showed 12h-gap + ~2h-deploy-jitter could happen back-to-back. For gdeltIntel I'm starting with 720 since the cron isn't bundled (so it doesn't share a deploy window with sibling seeders) and the observed gap was 7.8h (well within 12h). If we see another flip post-merge, the same shape of bump to 840 is a one-liner.

Test plan

  • node --check api/health.js — syntax OK
  • npx esbuild api/health.js --bundle --format=esm --platform=browser — bundles cleanly for the edge runtime
  • node --test tests/health-content-age.test.mjs — 18/18 pass (no test was pinning this value)
  • Post-merge: watch UptimeRobot + /api/health?history=1 for the next ~24h. Expectation: zero gdeltIntel-driven flips when a deploy preempts a single cron tick; flip CORRECTLY fires only if 2+ consecutive ticks miss.

Companion

Five PRs from this week's series, in dependency-order for cleanest landing:

koala73 added 3 commits May 10, 2026 22:29
…jitter)

WM 2026-05-10 18:02 UTC: UptimeRobot flipped DOWN→UP within 1 minute,
problems list showed `resilienceIntervals:STALE_SEED(722min)`. Diagnosed
via /api/health failure-log + Railway cron history:

  04:04 UTC ✓  (10s)
  10:00 UTC ✓  (3s)
  16:00 UTC ✗  NO ROW — Railway didn't fire the scheduled tick
  22:03 UTC ✓  (9s) ← caught up here

Multiple deploys today around 12:00 UTC (the missed slot was 12:00 UTC =
16:00 local in the user's TZ). Railway preempts scheduled cron ticks when
a deploy is in flight — normal platform behavior, not a code bug.

The 720min budget = 12h = 2× the real 6h cron cadence = exact gold-
standard floor with ZERO jitter tolerance. Any single missed tick (12h
gap) hits the threshold by definition. UptimeRobot polls every 60s and
fires a DOWN page on any WARNING — even one that resolves within 1
minute.

Bump 720 → 840 (14h ≈ 2.33× cadence):
  - tolerates 1 missed tick (12h gap) + ~2h deploy jitter
  - still alerts at 2 missed ticks (18h gap = real outage)
  - upper bound (1080) stays untouched per existing comment's note that
    1080 = "over-permissive: masks a 12h outage"

Same pattern as PR #3640 (correlationCards 15→30 and hyperliquidFlow
15→30): pre-fix all three sat at the exact 2-3× safety floor with no
margin for cron jitter. Each was independently chosen for tight
detection at design time; each independently produced UptimeRobot
false-positives in practice.

Tests:
- Updated value-pin assertion (720 → 840) + comment with the 2026-05-10
  incident as the new prior-value entry
- Existing bounds checks (>= 540, <= 1080) unchanged and still passing
- 13/13 resilience tests pass; 18/18 health-content-age tests pass
…P2×3)

Greptile PR #3652 review surfaced three valid P2 issues in the test
file, all stemming from the budget bump 720→840:

P2 — Top-of-describe context comment (line 203-205) still read
"720 = 12h staleness = 2 missed cron ticks". Updated to reflect the
new 840 ≈ 14h with explicit jitter rationale.

P2 — Bounds-check failure message (line 286-287) still said the
project convention was "720 here, matching resilienceRanking". Updated
to "840 (≈2.33× cadence, matches resilienceRanking)" so a future test
failure reports the correct reference value.

P2 — Missing value-pin assertion for resilienceRanking. api/health.js
explicitly notes both keys MUST stay in sync ("written by the SAME
Resilience-Scores section"). Without an assertion enforcing parity, a
future one-off edit to resilienceRanking would silently diverge from
resilienceIntervals and bypass the contract described in the comments.
Added a fifth test in the describe block:

    it('resilienceRanking.maxStaleMin matches resilienceIntervals ...', () => {
      assert.equal(rankingBudget, intervalsBudget, ...);
    });

This catches divergence regardless of whether someone bumps just one
of the two — the assertion enforces equality, not a hardcoded value,
so the next legitimate bump only needs to touch both api/health.js
values (the test passes as long as they match).

14/14 tests pass (was 13, +1 for the parity assertion).
… as #3652)

2026-05-12 15:20 UTC: UptimeRobot flipped DOWN→UP within ~1 minute,
failure-log showed `gdeltIntel:STALE_SEED(421min)` → cleared, then
hit again at 467min. Same Railway-deploy-preempted-tick pattern as
resilienceIntervals on 2026-05-10 (PR #3652).

Root cause: maxStaleMin=420 at 6h cron cadence = 1.16× ratio with
virtually zero jitter margin. Any single missed cron tick (deploy
preempts a scheduled fire) produces a >7h gap → flips STALE_SEED for
the ~1 minute until the next scheduled run catches up.

Bumping to 720 (12h = 2× cadence) matches the gold-standard project
convention:
  - tolerates 1 missed tick (12h gap) at 6h cadence
  - alerts at 2 missed ticks (18h gap) — real outage signal preserved
  - upstream CACHE_TTL is 24h, so per-topic merge always has a prior
    snapshot even at the upper end of the new budget

No tests pin this value, so the only change is the production
constant + the audit-trail comment.

Fifth instance of the same class today/this week (correlationCards +
hyperliquidFlow in #3640, resilienceIntervals/Ranking in #3652, now
gdeltIntel). Pattern: knobs sized at the exact safety floor have no
room for routine Railway deploy jitter.
@vercel

vercel Bot commented May 12, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
worldmonitor Ready Ready Preview, Comment May 12, 2026 4:17pm

Request Review

@greptile-apps

greptile-apps Bot commented May 12, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR addresses a fifth recurrence of the Railway deploy-jitter / missed-cron-tick false-positive pattern by bumping gdeltIntel.maxStaleMin from 420 → 720 (1.16× → 2× cadence), giving the 6h standalone cron one full missed-tick of headroom before alerting.

  • gdeltIntel 420 → 720: raises the staleness budget to exactly 2× the 6h cron cadence; the 24h CACHE_TTL means a prior snapshot always exists within the new window.
  • resilienceRanking + resilienceIntervals 720 → 840: also present in this diff (attributed to companion PR fix(health): bump resilienceIntervals/Ranking budget 720→840 (deploy-jitter) #3652 in the description); adds ~2h deploy-window jitter tolerance on top of the 2× floor.
  • Test suite: existing 720-pin assertions updated to 840 for the resilience keys; a new parity test added to enforce that resilienceRanking and resilienceIntervals always stay in sync.

Confidence Score: 4/5

Safe to merge; all three config changes are well-justified by observed incident data and the math checks out.

The resilience key changes (720→840) appear in both this diff and companion PR #3652 — whoever lands second will hit a conflict or a no-op on those two lines. Worth confirming landing order before merging.

api/health.js lines 380-381 (resilienceRanking / resilienceIntervals overlap with #3652).

Important Files Changed

Filename Overview
api/health.js Three config values changed: gdeltIntel 420→720, resilienceRanking 720→840, resilienceIntervals 720→840. The gdeltIntel change is the stated purpose of the PR; the resilience key bumps appear in this diff but are attributed to companion PR #3652 in the description — scope mismatch is worth tracking for landing-order safety.
tests/resilience-cache-keys-health-sync.test.mts Test assertions updated from 720→840 for resilienceIntervals; a new parity test added to enforce resilienceRanking == resilienceIntervals at all times. Well-structured audit-trail comments and bounds tests (>= 540, <= 1080) remain intact.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Railway cron fires\nevery 6h] --> B{seed-gdelt-intel.mjs\nruns OK?}
    B -- Yes --> C[Write seed-meta key\nfetch time = now]
    B -- No / Deploy preempts tick --> D[Tick missed\nseedAge += 6h]
    C --> E[health.js check:\nseedAgeMin vs maxStaleMin]
    D --> E
    E -- seedAge < 720min --> F[Status: OK]
    E -- 720 <= seedAge < 1080min --> G[Status: STALE_SEED\n1 missed tick + jitter]
    E -- seedAge >= 1080min --> H[Status: STALE_SEED\n2+ missed ticks = real outage]
Loading

Reviews (1): Last reviewed commit: "fix(health): bump gdeltIntel maxStaleMin..." | Re-trigger Greptile

Comment thread api/health.js
Comment on lines +380 to +381
resilienceRanking: { key: 'seed-meta:resilience:ranking', maxStaleMin: 840 }, // RPC cache (12h TTL, refreshed every 6h by seed-resilience-scores cron); 14h budget tolerates 1 missed tick (12h gap) + ~2h jitter for in-flight deploys that preempt a scheduled tick; alerts at 2 missed ticks (18h gap). Bumped from 720 — see comment below.
resilienceIntervals: { key: 'seed-meta:resilience:intervals', maxStaleMin: 840 }, // bundled into seed-bundle-resilience, written by the Resilience-Scores section. Real Railway cron is `0 */6 * * *` (every 6h on the hour, UTC) — empirically verified 2026-04-28 via Railway logs showing 6h gaps between successful runs (the prior `intervalMs=2h with hourly fires` claim did not match what's deployed; either the bundle interval gate or the Railway service schedule makes the effective cadence 6h). 840 = 14h staleness ≈ 2.33× cadence: tolerates 1 missed tick (12h gap) + ~2h jitter for in-flight deploys; alerts at 2 missed ticks (18h gap). Matches resilienceRanking above (same Resilience-Scores section writes both). Prior values: 20160 (14d, 168× — silent on real outage), 1080 (18h, 3× — over-permissive: masks a 12h outage), 720 (12h, 2× — exact floor; flipped UptimeRobot WARNING for ~1min on every Railway-deploy-preempted tick: 2026-05-10 incident at 18:02 UTC, seedAgeMin=722 vs maxStale=720 after the 12:00 UTC tick was skipped during an in-flight deploy), 360 (1× — false-positive on routine jitter, 2026-04-28: seedAgeMin=367 vs maxStale=360). Re-tighten ONLY if/when the actual Railway cron schedule is verified sub-6h.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Scope of this diff exceeds PR title

The PR title and description table attribute the resilienceRanking / resilienceIntervals 720→840 bumps to companion PR #3652, but those lines are also changed here (the base commit still shows 720 for both). If #3652 merges to main before this branch is rebased, these two lines will conflict; if this PR lands first, #3652 will arrive as a no-op (or conflict) for those same lines. The final state is correct either way, but the landing-order dependency isn't reflected in the PR title, which could surprise whoever is sequencing the merges.

@koala73
koala73 merged commit ab23c61 into main May 13, 2026
12 checks passed
@koala73
koala73 deleted the fix/gdelt-intel-stale-budget-420-720 branch May 13, 2026 13:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant