fix(health): resilienceIntervals maxStaleMin matches real 6h cron cadence by koala73 · Pull Request #3484 · koala73/worldmonitor

koala73 · 2026-04-28T06:22:18Z

Summary

resilienceIntervals was emitting false-positive STALE_SEED because api/health.js:381 had the wrong cadence assumption.

Comment claimed: "intervalMs=2h with hourly Railway fires + 96min skip-window → effective ~2h cadence", and set maxStaleMin: 360 as "3× the real 2h cadence".
Real Railway cron: `0 */6 * * *` (every 6h on the hour, UTC). Confirmed by the user 2026-04-28 and corroborated by Railway logs showing 6h2min between two successful clean bundle runs (no skips, no errors, no SIGKILL — the cron is just running every 6h, not every 2h).
Effect: with cadence=6h and maxStaleMin=360min, ANY normal cron jitter (5–15min) tips the entry into STALE_SEED. Reproduced 2026-04-28: `seedAgeMin=367 vs maxStale=360` → false alarm despite the bundle running cleanly on schedule.

Fix: bump `maxStaleMin` 360 → 720 (12h staleness = 2 missed cron ticks). Matches the `resilienceRanking` entry directly above, which is written by the same Resilience-Scores cron section.

Why 720 specifically

Mirror-of-the-pattern with the existing `resilienceRanking` entry on line 380:
```
resilienceRanking: { ... maxStaleMin: 720 }, // 12h staleness threshold = 2 missed cron ticks
```
Both keys are written by the same cron (`refreshRankingAggregate` runs inside the same Resilience-Scores section that writes the interval keys), so they should share the same maxStaleMin.

Why not bigger

Bigger thresholds make the alarm useless. The prior values demonstrate this:

`20160` (14d, 168× cadence) — pre-2026-04 — silenced the alarm completely.
`1080` (18h, 3× cadence) — first-pass fix — over-permissive; would miss a real ~12h outage.
`360` (1× cadence) — current — false-positive on every routine cron jitter.
`720` (2× cadence) — fires on real 2-tick outages, ignores 1-tick jitter. Standard project convention per the `health-maxstalemin-write-cadence` skill.

Not related to PR #3482

PR #3482 (the v17 → v18 cache bump for §U8.1) touches `resilience:score:`, `resilience:ranking:`, `resilience:history:` namespaces. This fix concerns `resilience:intervals:v2:*` (separate namespace, untouched by #3482). Both PRs can land independently in any order.

Test plan

Diff is one line in `api/health.js` plus an updated comment that documents the empirically-verified cadence + history of prior values + the 2026-04-28 incident timestamp.
Match-of-pattern: same value (`720`) and same rationale (`12h staleness = 2 missed cron ticks`) as the `resilienceRanking` entry written by the same cron.
Post-merge: `/api/health` returns `OK` for `resilienceIntervals` once Vercel ships, even with the current `seedAgeMin=367`.

Follow-up worth tracking separately (NOT in this PR)

The deeper question — why is the bundle's effective cadence 6h instead of the 2h documented in `scripts/seed-bundle-resilience.mjs:5-12`? The bundle's interval gate or the Railway service config has the resilience bundle firing every 6h. If 2h was the design intent, that's a separate config drift to investigate. Two related project skills cover the diagnosis: `bundle-runner-interval-gate-blocks-manual-rerun`, `bundle-runner-canonical-key-skip-gate`. For now, this PR aligns the alarm with the deployed cron schedule rather than fight it.

…ence The prior comment claimed `intervalMs=2h with hourly Railway fires + 96min skip-window → effective ~2h cadence` and set `maxStaleMin: 360` (= 1× claimed cadence). The deployed Railway cron is actually `0 */6 * * *` (every 6h on the hour, UTC), confirmed by the user 2026-04-28 and corroborated by Railway logs: only 2 successful runs in the seed-bundle-resilience service log over 6h2min, both clean, both writing 196 interval keys. With true cadence=6h and maxStaleMin=360min, ANY normal cron jitter or a slightly-late tick tips resilienceIntervals into STALE_SEED. Reproduced 2026-04-28: seedAgeMin=367 vs maxStale=360 → false alarm despite the bundle running cleanly on schedule. Bumped maxStaleMin 360 → 720 (12h staleness = 2 missed cron ticks). Matches the resilienceRanking entry directly above, written by the SAME cron (refreshRankingAggregate runs in the same Resilience-Scores section). Same pattern as the project's `health-maxstalemin-write- cadence` skill: maxStaleMin must be sized to actual write cadence + grace, not the documented one. Updated the comment to: - Replace the incorrect "intervalMs=2h with hourly fires" claim with the empirically-verified `0 */6 * * *` schedule. - Document the prior values and what each was tuned for, so the next operator who hits this can see why 360 was wrong without re-doing the log-archaeology. - Note the 2026-04-28 incident timestamp so the audit trail is clear. Verification: prefix bump v17→v18 in PR #3482 is in a DIFFERENT namespace (resilience:score:, resilience:ranking:, resilience:history:); this fix is independent and addresses resilience:intervals:v2:* which was unaffected by that PR.

vercel · 2026-04-28T06:22:23Z

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
worldmonitor	Ignored	Preview	Apr 28, 2026 6:31am

greptile-apps · 2026-04-28T06:23:39Z

Greptile Summary

This PR fixes a false-positive STALE_SEED alarm for resilienceIntervals in api/health.js by correcting maxStaleMin from 360 (1× the actual 6h cadence, which trips on routine cron jitter) to 720 (2× the 6h cadence = 12h staleness threshold), mirroring the identical convention already used by the resilienceRanking entry written by the same Railway cron.

Confidence Score: 5/5

Safe to merge — single-line corrective fix with thorough empirical justification and consistent precedent in the surrounding codebase.

The change is a one-line numeric constant correction (360 → 720) with: an empirically verified root cause (Railway logs showing 6h gaps), consistency with the identical value used by the directly adjacent resilienceRanking entry (same cron, same cadence), and alignment with the 2× interval convention used by other 6h entries in the file (chokepointFlows, energyIntelligence). No logic, API contracts, or data pipelines are affected.

No files require special attention.

Important Files Changed

Filename	Overview
api/health.js	Single-line change bumping resilienceIntervals maxStaleMin from 360 → 720 to match the empirically verified 6h Railway cron cadence; comment updated with full incident history and rationale.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A["Railway cron\n0 */6 * * * (UTC)"] -->|"runs every 6h"| B["seed-bundle-resilience\nResilience-Scores section"]
    B -->|"writes seed-meta:resilience:intervals"| C["resilienceIntervals\nmaxStaleMin: 720"]
    B -->|"runs refreshRankingAggregate\nwrites seed-meta:resilience:ranking"| D["resilienceRanking\nmaxStaleMin: 720"]
    E["/api/health check"] -->|"reads seed-meta key\nchecks age vs maxStaleMin"| C
    E -->|"reads seed-meta key\nchecks age vs maxStaleMin"| D
    C -->|"seedAgeMin ≤ 720"| F["✅ OK"]
    C -->|"seedAgeMin > 720"| G["🔴 STALE_SEED"]
    D -->|"seedAgeMin ≤ 720"| F

_{Reviews (1): Last reviewed commit: "fix(health): resilienceIntervals maxStal..." | Re-trigger Greptile}

…ence Review-fix for PR #3484. The existing health-sync test asserted three things that all assumed the wrong (~2h) writer cadence and rejected the new 720 contract: 1. line 216 — exact value 360 → 720 2. line 220 — floor 180 (1.5× 2h) → 540 (1.5× 6h) 3. line 229 — ceiling 480 (4× 2h) → 1080 (3× 6h) Updated all three plus the describe-block label and the rationale comment block at the top. The block now: - Documents both the 2026-04-27 (false-OK at 14d) and 2026-04-28 (false-positive at 360) incidents with root causes for each. - Replaces the in-bundle `intervalMs=2h` cadence claim with the empirically-verified Railway cron `0 */6 * * *` schedule. - Records the full audit trail of prior values (20160 / 1080 / 360 / 720) and what each was tuned for. The `Resilience-Scores section gate is 2h` test stays — that assertion checks the IN-BUNDLE intervalMs in scripts/seed-bundle-resilience.mjs, which is still 2h in the source. Updated only its rationale comment to clarify that the section's in-bundle 2h gate is independent from the OUTER Railway-cron cadence (which is 6h). 13/13 tests in this file pass. Typecheck clean.

…ence (koala73#3484) * fix(health): resilienceIntervals maxStaleMin matches real 6h cron cadence The prior comment claimed `intervalMs=2h with hourly Railway fires + 96min skip-window → effective ~2h cadence` and set `maxStaleMin: 360` (= 1× claimed cadence). The deployed Railway cron is actually `0 */6 * * *` (every 6h on the hour, UTC), confirmed by the user 2026-04-28 and corroborated by Railway logs: only 2 successful runs in the seed-bundle-resilience service log over 6h2min, both clean, both writing 196 interval keys. With true cadence=6h and maxStaleMin=360min, ANY normal cron jitter or a slightly-late tick tips resilienceIntervals into STALE_SEED. Reproduced 2026-04-28: seedAgeMin=367 vs maxStale=360 → false alarm despite the bundle running cleanly on schedule. Bumped maxStaleMin 360 → 720 (12h staleness = 2 missed cron ticks). Matches the resilienceRanking entry directly above, written by the SAME cron (refreshRankingAggregate runs in the same Resilience-Scores section). Same pattern as the project's `health-maxstalemin-write- cadence` skill: maxStaleMin must be sized to actual write cadence + grace, not the documented one. Updated the comment to: - Replace the incorrect "intervalMs=2h with hourly fires" claim with the empirically-verified `0 */6 * * *` schedule. - Document the prior values and what each was tuned for, so the next operator who hits this can see why 360 was wrong without re-doing the log-archaeology. - Note the 2026-04-28 incident timestamp so the audit trail is clear. Verification: prefix bump v17→v18 in PR koala73#3482 is in a DIFFERENT namespace (resilience:score:, resilience:ranking:, resilience:history:); this fix is independent and addresses resilience:intervals:v2:* which was unaffected by that PR. * test(health): align cache-keys health-sync test with real 6h cron cadence Review-fix for PR koala73#3484. The existing health-sync test asserted three things that all assumed the wrong (~2h) writer cadence and rejected the new 720 contract: 1. line 216 — exact value 360 → 720 2. line 220 — floor 180 (1.5× 2h) → 540 (1.5× 6h) 3. line 229 — ceiling 480 (4× 2h) → 1080 (3× 6h) Updated all three plus the describe-block label and the rationale comment block at the top. The block now: - Documents both the 2026-04-27 (false-OK at 14d) and 2026-04-28 (false-positive at 360) incidents with root causes for each. - Replaces the in-bundle `intervalMs=2h` cadence claim with the empirically-verified Railway cron `0 */6 * * *` schedule. - Records the full audit trail of prior values (20160 / 1080 / 360 / 720) and what each was tuned for. The `Resilience-Scores section gate is 2h` test stays — that assertion checks the IN-BUNDLE intervalMs in scripts/seed-bundle-resilience.mjs, which is still 2h in the source. Updated only its rationale comment to clarify that the section's in-bundle 2h gate is independent from the OUTER Railway-cron cadence (which is 6h). 13/13 tests in this file pass. Typecheck clean.

vercel Bot deployed to Preview April 28, 2026 06:24 View deployment

koala73 merged commit d615a25 into main Apr 28, 2026
11 checks passed

koala73 deleted the fix/health-resilience-intervals-maxstale-real-cadence branch April 28, 2026 06:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(health): resilienceIntervals maxStaleMin matches real 6h cron cadence#3484

fix(health): resilienceIntervals maxStaleMin matches real 6h cron cadence#3484
koala73 merged 2 commits into
mainfrom
fix/health-resilience-intervals-maxstale-real-cadence

koala73 commented Apr 28, 2026

Uh oh!

vercel Bot commented Apr 28, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot commented Apr 28, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

koala73 commented Apr 28, 2026

Summary

Why 720 specifically

Why not bigger

Not related to PR #3482

Test plan

Follow-up worth tracking separately (NOT in this PR)

Uh oh!

vercel Bot commented Apr 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

greptile-apps Bot commented Apr 28, 2026

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Flowchart

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

vercel Bot commented Apr 28, 2026 •

edited

Loading