fix(health): bump correlationCards maxStaleMin 15→30 to absorb cron jitter by koala73 · Pull Request #3640 · koala73/worldmonitor

koala73 · 2026-05-10T04:11:59Z

Summary

One-line tuning fix. Stop overnight UptimeRobot flips on correlationCards.

Evidence

Read health:last-failure + health:failure-log from Upstash this morning. Four WARNING incidents overnight, all the same root cause:

2026-05-10T04:04 UTC  WARNING  correlationCards:STALE_SEED(19min)
2026-05-10T04:02 UTC  WARNING  correlationCards:STALE_SEED(17min)
2026-05-10T02:21 UTC  WARNING  correlationCards:STALE_SEED(17min)
2026-05-10T00:56 UTC  WARNING  correlationCards:STALE_SEED(16min)

UptimeRobot pages exactly correlate (DOWN durations of 2-13min match the time between flip and the next successful seed catching up).

Math

Knob	Value	Notes
Seeder cron interval	5 min	`seed-bundle-derived-signals`
Bundle's 80% skip-gate floor	4 min	Two consecutive successful runs always ≥4min apart
Gold-standard rule	cron ≤ maxStaleMin/3	maxStaleMin=15 → cron must be ≤5min, ours IS 5min
Old `maxStaleMin`	15 min	Exactly at the floor — zero safety margin
New `maxStaleMin`	30 min	6× interval = 2× gold-standard floor; absorbs all observed jitter

When a bundle run lags by 4-5 min (cold start, upstream latency, transient network), the next successful seed lands at minute ~9-10 instead of ~5. If the next run lags too, the gap stretches to 15-19 min — over the budget by 1-4 min, just long enough for one /api/health probe to flip WARNING and UptimeRobot to fire DOWN.

The data itself doesn't go stale at 15 min — the cadence is fine. Only the alert threshold was too tight.

Changes

One field, one file:

- correlationCards: { key: 'seed-meta:correlation:cards', maxStaleMin: 15 },
+ correlationCards: { key: 'seed-meta:correlation:cards', maxStaleMin: 30 }, // 5min cron; 6× interval; was 15 → flapped overnight

No code-path change, no test change — purely a tunable threshold.

Test plan

Math verified against health:failure-log Redis data (the 16-19min gaps would not have tripped at maxStaleMin=30)
Post-merge: monitor UptimeRobot for 24h. Expectation: zero correlationCards-driven flips.
Post-merge: re-pull health:failure-log next morning — entries for correlationCards:STALE_SEED should be gone unless the gap exceeds 30min (which would be a real outage worth paging on).

Companion (separate PR)

Surfacing health:failure-log via a ?history=1 query param so this kind of diagnosis doesn't require Upstash credentials. Tracking in a separate PR to keep this one minimal.

…itter Overnight UptimeRobot flipped DOWN/UP four times (00:56, 02:21, 04:02, 04:04 UTC), each time correlationCards STALE_SEED for 16-19 min. From the health:failure-log: 04:04 UTC WARNING correlationCards:STALE_SEED(19min) 02:21 UTC WARNING correlationCards:STALE_SEED(17min) 00:56 UTC WARNING correlationCards:STALE_SEED(16min) Math: cron is 5min (seed-bundle-derived-signals), bundle's 80% skip gate floors successive runs at 4min, maxStaleMin was 15 — exactly the gold-standard ⅓-of-stale-budget rule. No safety margin. When bundle runs lag 4-5 min (cold start, upstream latency, transient network), the gap between successful seeds stretches to 9-10 min; if the next run also lags slightly, total gap hits 15-19 min and trips STALE_SEED. Bumping to 30 gives 6× interval = 2× gold-standard floor. Same data cadence, no change to the cron, no change to the seeder. Just stops alerting on the natural jitter. No code-path or test changes — purely a tunable threshold.

vercel · 2026-05-10T04:12:04Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
worldmonitor	Ready	Preview, Comment	May 10, 2026 4:42am

greptile-apps · 2026-05-10T04:13:59Z

Greptile Summary

Bumps correlationCards.maxStaleMin from 15 to 30 minutes in api/health.js to add a safety margin above the maximum observed seed gap (19 min) caused by bundle jitter in the 5-min seed-bundle-derived-signals cron.

The old threshold (15 min = 3× cron interval, the documented "gold-standard floor") had zero headroom; a cold-start or upstream latency event delaying two consecutive bundle runs produced 15-19 min gaps that tripped STALE_SEED and paged UptimeRobot. The new value (30 min = 6× interval) is consistent with other high-jitter entries in the file.
No logic changes — purely a tunable constant; failure-log evidence makes the root cause clear and the math is sound.

Confidence Score: 5/5

Safe to merge — single-constant change with clear evidence backing it, no code-path impact.

The change is one integer in a configuration object, well-supported by Redis failure-log data showing real overnight gaps of 16-19 min against a 15-min threshold. The new value (30 min) aligns with the 6× interval pattern used by comparable entries throughout the file. The only observation is that hyperliquidFlow carries the same 3× floor ratio on a 5-min cron and could be worth a proactive bump, but that is a separate, non-blocking concern.

No files require special attention; api/health.js line 355 (hyperliquidFlow) shares the same threshold ratio that triggered this fix and may be worth revisiting.

Important Files Changed

Filename	Overview
api/health.js	Single-line threshold bump for `correlationCards.maxStaleMin` from 15→30 to add safety margin above the bundle seeder's jitter-driven gap ceiling; well-justified by failure-log evidence and consistent with the 6× interval pattern used elsewhere in the file.

Sequence Diagram

sequenceDiagram
    participant Cron as seed-bundle-derived-signals (5min cron)
    participant Redis as Upstash Redis
    participant Health as /api/health
    participant UR as UptimeRobot

    Note over Cron: Normal run T=0 → seed-meta written
    Cron->>Redis: "write seed-meta:correlation:cards (fetchedAt=T+0)"
    Note over Cron: Cold-start / upstream lag
    Cron->>Redis: "write seed-meta:correlation:cards (fetchedAt=T+9)"
    Note over Health: Probe at T+10
    Health->>Redis: GET seed-meta:correlation:cards
    Redis-->>Health: "fetchedAt=T+9 (1min ago — OK at maxStaleMin=30)"
    Health-->>UR: "status=OK ✓"

    Note over Cron: OLD behaviour (maxStaleMin=15)
    Cron->>Redis: "write seed-meta:correlation:cards (fetchedAt=T+0)"
    Note over Cron: Two lagged runs → gap stretches to 19min
    Health->>Redis: GET seed-meta:correlation:cards
    Redis-->>Health: "fetchedAt=T+0 (19min ago — STALE at maxStaleMin=15!)"
    Health-->>UR: "status=WARNING ✗"
    UR-->>UR: fire DOWN alert

Comments Outside Diff (1)

api/health.js, line 355 (link)

hyperliquidFlow sits at the same 3× floor that triggered this fix

hyperliquidFlow has a 5-min Railway cron and maxStaleMin: 15 — exactly the "gold-standard floor" (3× interval, zero safety margin) that caused the overnight correlationCards flapping. Railway crons are lower-jitter than the bundle seeder's 80%-skip-gate, so it hasn't flapped yet, but any cold-start or upstream hiccup that delays two consecutive runs by a combined ~5 min would push the gap past 15 min. Given the same root cause was just diagnosed for correlationCards, bumping hyperliquidFlow to 30 min (6× interval) proactively would make the health check consistent.

_{Reviews (1): Last reviewed commit: "fix(health): bump correlationCards maxSt..." | Re-trigger Greptile}

Greptile review of PR #3640 flagged that `hyperliquidFlow` sits at the same 3× ratio (5min cron, 15min budget = floor) that just bit correlationCards overnight. Pre-fix comment incorrectly called it a "Railway cron"; it's actually bundle-driven (seed-bundle-market-backup), so it's subject to the same 80% skip-gate that produced the 9-19min jitter we just observed. Bumping proactively to the same 30min budget (6× interval) — no flap reports for hyperliquidFlow yet, but the underlying exposure is identical and the parity makes the rationale self-documenting. Also corrected the inline comment to reflect the bundle source.

koala73 · 2026-05-10T04:40:02Z

Addressed Greptile's outside-diff P2 in ab2a47b98. Verified the parity claim before bumping:

hyperliquidFlow (line 355) had maxStaleMin: 15 with comment claiming "Railway cron 5min; 15min = 3x interval"
Actual source: scripts/seed-bundle-market-backup.mjs:9 — intervalMs: 5 * MIN inside a bundle, not a direct Railway cron
That means it shares the same 80% skip-gate exposure as correlationCards (4-min minimum spacing, 9-19min observed jitter under load)
Comment was misleading — corrected to reflect the bundle source

Now both 5-min-cron entries are at 30min budget (6× interval). Sweep confirmed no other tight ratios in the file:

$ grep -nE "maxStaleMin:\s*1[0-9]\b" api/health.js
300: telegramFeed: ... maxStaleMin: 10  // 60s poll = 10× ratio, very generous
355: hyperliquidFlow: ... maxStaleMin: 15  ← was the only other floor entry

Updated PR scope: now bumps two entries (correlationCards + hyperliquidFlow) plus the inline-comment correction. Greptile's confidence score should still hold (5/5 — same shape change, no logic, both backed by the same diagnostic).

…jitter) (#3652) * fix(health): bump resilienceIntervals/Ranking budget 720→840 (deploy-jitter) WM 2026-05-10 18:02 UTC: UptimeRobot flipped DOWN→UP within 1 minute, problems list showed `resilienceIntervals:STALE_SEED(722min)`. Diagnosed via /api/health failure-log + Railway cron history: 04:04 UTC ✓ (10s) 10:00 UTC ✓ (3s) 16:00 UTC ✗ NO ROW — Railway didn't fire the scheduled tick 22:03 UTC ✓ (9s) ← caught up here Multiple deploys today around 12:00 UTC (the missed slot was 12:00 UTC = 16:00 local in the user's TZ). Railway preempts scheduled cron ticks when a deploy is in flight — normal platform behavior, not a code bug. The 720min budget = 12h = 2× the real 6h cron cadence = exact gold- standard floor with ZERO jitter tolerance. Any single missed tick (12h gap) hits the threshold by definition. UptimeRobot polls every 60s and fires a DOWN page on any WARNING — even one that resolves within 1 minute. Bump 720 → 840 (14h ≈ 2.33× cadence): - tolerates 1 missed tick (12h gap) + ~2h deploy jitter - still alerts at 2 missed ticks (18h gap = real outage) - upper bound (1080) stays untouched per existing comment's note that 1080 = "over-permissive: masks a 12h outage" Same pattern as PR #3640 (correlationCards 15→30 and hyperliquidFlow 15→30): pre-fix all three sat at the exact 2-3× safety floor with no margin for cron jitter. Each was independently chosen for tight detection at design time; each independently produced UptimeRobot false-positives in practice. Tests: - Updated value-pin assertion (720 → 840) + comment with the 2026-05-10 incident as the new prior-value entry - Existing bounds checks (>= 540, <= 1080) unchanged and still passing - 13/13 resilience tests pass; 18/18 health-content-age tests pass * review: fix stale 720 refs + add resilienceRanking value-pin parity (P2×3) Greptile PR #3652 review surfaced three valid P2 issues in the test file, all stemming from the budget bump 720→840: P2 — Top-of-describe context comment (line 203-205) still read "720 = 12h staleness = 2 missed cron ticks". Updated to reflect the new 840 ≈ 14h with explicit jitter rationale. P2 — Bounds-check failure message (line 286-287) still said the project convention was "720 here, matching resilienceRanking". Updated to "840 (≈2.33× cadence, matches resilienceRanking)" so a future test failure reports the correct reference value. P2 — Missing value-pin assertion for resilienceRanking. api/health.js explicitly notes both keys MUST stay in sync ("written by the SAME Resilience-Scores section"). Without an assertion enforcing parity, a future one-off edit to resilienceRanking would silently diverge from resilienceIntervals and bypass the contract described in the comments. Added a fifth test in the describe block: it('resilienceRanking.maxStaleMin matches resilienceIntervals ...', () => { assert.equal(rankingBudget, intervalsBudget, ...); }); This catches divergence regardless of whether someone bumps just one of the two — the assertion enforces equality, not a hardcoded value, so the next legitimate bump only needs to touch both api/health.js values (the test passes as long as they match). 14/14 tests pass (was 13, +1 for the parity assertion).

* fix(health): bump resilienceIntervals/Ranking budget 720→840 (deploy-jitter) WM 2026-05-10 18:02 UTC: UptimeRobot flipped DOWN→UP within 1 minute, problems list showed `resilienceIntervals:STALE_SEED(722min)`. Diagnosed via /api/health failure-log + Railway cron history: 04:04 UTC ✓ (10s) 10:00 UTC ✓ (3s) 16:00 UTC ✗ NO ROW — Railway didn't fire the scheduled tick 22:03 UTC ✓ (9s) ← caught up here Multiple deploys today around 12:00 UTC (the missed slot was 12:00 UTC = 16:00 local in the user's TZ). Railway preempts scheduled cron ticks when a deploy is in flight — normal platform behavior, not a code bug. The 720min budget = 12h = 2× the real 6h cron cadence = exact gold- standard floor with ZERO jitter tolerance. Any single missed tick (12h gap) hits the threshold by definition. UptimeRobot polls every 60s and fires a DOWN page on any WARNING — even one that resolves within 1 minute. Bump 720 → 840 (14h ≈ 2.33× cadence): - tolerates 1 missed tick (12h gap) + ~2h deploy jitter - still alerts at 2 missed ticks (18h gap = real outage) - upper bound (1080) stays untouched per existing comment's note that 1080 = "over-permissive: masks a 12h outage" Same pattern as PR #3640 (correlationCards 15→30 and hyperliquidFlow 15→30): pre-fix all three sat at the exact 2-3× safety floor with no margin for cron jitter. Each was independently chosen for tight detection at design time; each independently produced UptimeRobot false-positives in practice. Tests: - Updated value-pin assertion (720 → 840) + comment with the 2026-05-10 incident as the new prior-value entry - Existing bounds checks (>= 540, <= 1080) unchanged and still passing - 13/13 resilience tests pass; 18/18 health-content-age tests pass * review: fix stale 720 refs + add resilienceRanking value-pin parity (P2×3) Greptile PR #3652 review surfaced three valid P2 issues in the test file, all stemming from the budget bump 720→840: P2 — Top-of-describe context comment (line 203-205) still read "720 = 12h staleness = 2 missed cron ticks". Updated to reflect the new 840 ≈ 14h with explicit jitter rationale. P2 — Bounds-check failure message (line 286-287) still said the project convention was "720 here, matching resilienceRanking". Updated to "840 (≈2.33× cadence, matches resilienceRanking)" so a future test failure reports the correct reference value. P2 — Missing value-pin assertion for resilienceRanking. api/health.js explicitly notes both keys MUST stay in sync ("written by the SAME Resilience-Scores section"). Without an assertion enforcing parity, a future one-off edit to resilienceRanking would silently diverge from resilienceIntervals and bypass the contract described in the comments. Added a fifth test in the describe block: it('resilienceRanking.maxStaleMin matches resilienceIntervals ...', () => { assert.equal(rankingBudget, intervalsBudget, ...); }); This catches divergence regardless of whether someone bumps just one of the two — the assertion enforces equality, not a hardcoded value, so the next legitimate bump only needs to touch both api/health.js values (the test passes as long as they match). 14/14 tests pass (was 13, +1 for the parity assertion). * fix(health): bump gdeltIntel maxStaleMin 420→720 (deploy-jitter, same as #3652) 2026-05-12 15:20 UTC: UptimeRobot flipped DOWN→UP within ~1 minute, failure-log showed `gdeltIntel:STALE_SEED(421min)` → cleared, then hit again at 467min. Same Railway-deploy-preempted-tick pattern as resilienceIntervals on 2026-05-10 (PR #3652). Root cause: maxStaleMin=420 at 6h cron cadence = 1.16× ratio with virtually zero jitter margin. Any single missed cron tick (deploy preempts a scheduled fire) produces a >7h gap → flips STALE_SEED for the ~1 minute until the next scheduled run catches up. Bumping to 720 (12h = 2× cadence) matches the gold-standard project convention: - tolerates 1 missed tick (12h gap) at 6h cadence - alerts at 2 missed ticks (18h gap) — real outage signal preserved - upstream CACHE_TTL is 24h, so per-topic merge always has a prior snapshot even at the upper end of the new budget No tests pin this value, so the only change is the production constant + the audit-trail comment. Fifth instance of the same class today/this week (correlationCards + hyperliquidFlow in #3640, resilienceIntervals/Ranking in #3652, now gdeltIntel). Pattern: knobs sized at the exact safety floor have no room for routine Railway deploy jitter.

vercel Bot deployed to Preview May 10, 2026 04:15 View deployment

koala73 mentioned this pull request May 10, 2026

feat(health): expose failure-log via /api/health?history=1 #3641

Merged

5 tasks

vercel Bot deployed to Preview May 10, 2026 04:42 View deployment

koala73 merged commit 6654710 into main May 10, 2026
11 checks passed

koala73 deleted the fix/correlation-cards-stale-budget branch May 10, 2026 04:43

This was referenced May 10, 2026

fix(brief-dedup): cap replay-log lists to prevent Upstash 500MB record limit #3649

Merged

fix(health): bump resilienceIntervals/Ranking budget 720→840 (deploy-jitter) #3652

Merged

koala73 mentioned this pull request May 12, 2026

fix(health): bump gdeltIntel maxStaleMin 420→720 (deploy-jitter) #3671

Merged

4 tasks

koala73 mentioned this pull request May 13, 2026

fix(seed-portwatch-port-activity): cap cold-fetch at 30/run to prevent 174-country cliff #3676

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(health): bump correlationCards maxStaleMin 15→30 to absorb cron jitter#3640

fix(health): bump correlationCards maxStaleMin 15→30 to absorb cron jitter#3640
koala73 merged 2 commits into
mainfrom
fix/correlation-cards-stale-budget

koala73 commented May 10, 2026

Uh oh!

vercel Bot commented May 10, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot commented May 10, 2026 •

edited

Loading

Comments Outside Diff (1)

Uh oh!

koala73 commented May 10, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

koala73 commented May 10, 2026

Summary

Evidence

Math

Changes

Test plan

Companion (separate PR)

Uh oh!

vercel Bot commented May 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

greptile-apps Bot commented May 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Sequence Diagram

Comments Outside Diff (1)

Uh oh!

koala73 commented May 10, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

vercel Bot commented May 10, 2026 •

edited

Loading

greptile-apps Bot commented May 10, 2026 •

edited

Loading