Summary

Found the source of today's Upstash "Max Record Size 500MB hit 6× in 5 minutes" alerts. The digest:replay-log:v1:full:en:all:<date> lists have been growing unbounded within each day — the largest at 421,716 entries × ~1.5KB each = ~630MB per key, well over the 500MB Fixed-plan record-size limit.

Likely connected to today's seed-forecasts crash

The seed-forecasts cron failed at 13:00 UTC with a Redis SET timeout on a 0.17MB payload (well under any limit). When Upstash's backend struggles with a 630MB-record write, it back-pressures adjacent operations. Trimming the oversized lists should reduce that pressure.

Evidence

Sweep of all replay-log keys:

421716 entries  digest:replay-log:v1:full:en:all:2026-05-07
419782 entries  digest:replay-log:v1:full:en:all:2026-05-08
419473 entries  digest:replay-log:v1:full:en:all:2026-05-06
411401 entries  digest:replay-log:v1:full:en:all:2026-05-05
397556 entries  digest:replay-log:v1:full:en:all:2026-04-30
... 10 more daily logs at 100K-360K each

Root cause

scripts/lib/brief-dedup-replay-log.mjs:323 issued only RPUSH + EXPIRE, no LTRIM. The 30d TTL bounded cross-day rotation, but within a single busy day the list grew without limit:

const rpushCmd = ['RPUSH', key, ...records.map((r) => JSON.stringify(r))];
const expireCmd = ['EXPIRE', key, String(TTL_SECONDS)];
const result = await pipelineImpl([rpushCmd, expireCmd]);
//                                       ↑ no LTRIM

Fix

Insert LTRIM key -100000 -1 between RPUSH and EXPIRE — tail-keep the most recent 100,000 entries:

const rpushCmd = ['RPUSH', key, ...records.map((r) => JSON.stringify(r))];
const ltrimCmd = ['LTRIM', key, `-${REPLAY_LOG_MAX_ENTRIES_PER_DAY}`, '-1'];
const expireCmd = ['EXPIRE', key, String(TTL_SECONDS)];

Cap chosen: 100,000 × ~1.5KB ≈ 150MB → 3× safety margin under 500MB. Cap exported as REPLAY_LOG_MAX_ENTRIES_PER_DAY so a future bump only needs one location change.

Why tail-keep is safe for the use-case

The U6 14-day replay harness aggregates ACROSS days using stable repHash cluster identity (per the existing comments in replay-digest-cooldown.mjs). Within-day older-entry eviction is acceptable for calibration analysis — operators get a representative sample of each day's traffic, not exhaustive coverage. The newest 100K still vastly exceeds any realistic sweep sample size.

Test plan

• node --test tests/brief-dedup-replay-log.test.mjs — 31/31 pass
  • Updated existing pipeline-shape assertion (2 commands → 3, with LTRIM in the middle)
  • Added regression guard reading the cap from the exported REPLAY_LOG_MAX_ENTRIES_PER_DAY constant (no brittle dual-update)
  • Sanity assertion that the cap stays in the 10K-200K band (so future bumps stay safely under 500MB at observed entry sizes)
• Post-merge: monitor Upstash for "Max Record Size" alerts → expectation: zero new alerts on digest:replay-log:* after the next writer cycle
• Post-merge: monitor seed-forecasts cron success rate → expectation: improved publish reliability

One-time cleanup of EXISTING oversized keys

The new code only caps NEW writes. Existing keys are already at 200K-420K entries. They'll naturally trim when the next writer pass adds even one entry (the LTRIM applies to the whole list, not just new entries). But to immediately reclaim the ~6GB of overage, run this after merge:

TOKEN=$UPSTASH_REDIS_REST_TOKEN
URL=$UPSTASH_REDIS_REST_URL
for d in 04-25 04-26 04-27 04-28 04-29 04-30 05-01 05-02 05-03 05-04 05-05 05-06 05-07 05-08 05-09 05-10; do
  for variant in full finance markets safety conflict;  do
    k="digest:replay-log:v1:${variant}:en:all:2026-${d}"
    curl -s "$URL/ltrim/$k/-100000/-1" -H "Authorization: Bearer $TOKEN"
    echo "  trimmed $k"
  done
done

(Or wait for next writer cycle — same end state, just slower.)

Companion to today's other resilience work

• fix(health): bump correlationCards maxStaleMin 15→30 to absorb cron jitter #3640 (correlationCards budget) — reduces alert flap noise
• feat(health): expose failure-log via /api/health?history=1 #3641 (history endpoint) — makes diagnosis self-service
• This PR — fixes the 500MB record-limit overruns that are likely back-pressuring other writes
• Upcoming: retry-on-transient for atomicPublish in _seed-utils.mjs so a single Upstash hiccup doesn't kill a 3-minute LLM-heavy seed run (separate PR — different surface area, separate review focus)