Summary

seed-research crashed (exit 1, ZERO_YIELD) on the ~15:41 UTC tick (deployment 40446125-54ad-48b2-9ea6-572a2574ef07) and left researchArxivHnTrending EMPTY in prod (records: 0). As of 78+ min later the domain is still EMPTY (seedAgeMin climbing monotonically → no successful re-seed since the ~15:00 tick). Root cause is a single flaky upstream (arXiv) zeroing the entire primary record set. /api/health reports the site DEGRADED on this.

Discovered via /diagnose-railway-seeders. Two prior ticks (14:04, 15:00) succeeded with recordCount=50, so arXiv is transiently timing out — not permanently dead — but the design makes a single blip crash the run + empty the domain.

Root cause (scripts/seed-research.mjs)

The primary record key that drives declareRecords / the EMPTY-vs-OK verdict is solely arXiv cs.AI:

• runSeed('research', 'arxiv-hn-trending', 'research:arxiv:v1:cs.AI::50', …) — scripts/seed-research.mjs:343
• declareRecords = data?.papers?.length ?? 0 — :339-341 (papers = arXiv cs.AI only)
• fetchAll returns primaryKey || { papers: [] } — :331-332

Three compounding flaws:

1. No per-category error isolation in fetchArxivPapers (:22-66). All 3 arXiv categories (cs.AI, cs.CL, cs.CR) are fetched sequentially in one loop, each AbortSignal.timeout(15_000) (:30). A timeout on any category throws out of the whole function → Promise.allSettled marks arxiv rejected (:313, log line arXiv failed: The operation was aborted due to timeout) → allData.arxiv = null (:307) → primary { papers: [] } → declareRecords = 0. Even a successfully-fetched cs.AI is discarded if cs.CL/cs.CR time out. Contrast fetchTrendingRepos (:287-289), which isolates each language in try/catch.

2. Record count depends only on the flakiest source. On the crashed run HN (30), trending (120), tech-events (125) and the other arXiv cats all fetched fine and wrote their extra keys — but none count toward declareRecords. 275 real records in hand, verdict still records: 0 → EMPTY.

3. ARXIV_TTL = 3600s (:15) is razor-thin vs the ~hourly cron (log timestamps 14:04→15:00 = 56 min; margin ~4 min). When arXiv blips, the runner's zero-yield RETRY path (scripts/_seed-utils.mjs:1711-1748) tries extendExistingTtl on the last-good primary — but it had already expired, so preserved=false → exit 1 (RETRY FAILED, :1744-1748) instead of the graceful exit-0 (:1750-1753). Hence the "Deploy Crashed!" alert + EMPTY prod.

Evidence (deployment 40446125-…)

  arXiv failed: The operation was aborted due to timeout
  Extra key research:hackernews:v1:top:30: written      ← HN succeeded
  Extra key research:trending:v1:python:daily:50: written ← trending succeeded
  Extended TTL on 1 key(s) (3600s)
=== Done (18082ms, RETRY FAILED) ===
  WARNING: 1 key(s) were expired/missing — EXPIRE was a no-op; manual seed required
  FAILURE: declareRecords returned 0 and last-good preservation failed

Prod: researchArxivHnTrending: {status: EMPTY, records: 0, seedAgeMin: 78, maxStaleMin: 150}.

Proposed fix (3 composable parts; follow Bug Fix Protocol — failing test first)

1. Per-category isolation in fetchArxivPapers: wrap each category fetch in try/catch (mirror fetchTrendingRepos :287-289) so one category timeout doesn't discard the others.
2. Retry the cs.AI primary fetch (a short bounded retry / withRetry) so a single transient timeout retries before the run yields 0.
3. Raise ARXIV_TTL above the cron cadence (e.g. 3600 → 7200) so a single missed tick can't expire last-good → the runner exits 0 (graceful RETRY, prod keeps serving last-good) instead of exit 1 + EMPTY.

Parts 1+2 stop the zero-yield; part 3 makes any residual blip graceful instead of a crash. Optionally: consider whether the domain's health/record count should reflect the other successful sources (HN/trending/events) rather than arXiv alone, so an arXiv-only outage degrades rather than empties.

Repro / manual recovery

• Manual reseed to clear the current EMPTY immediately: run scripts/seed-research.mjs against prod Redis (once arXiv responds).
• Failing test: stub the arXiv fetch to time out and assert the run still yields the HN/trending records and exits 0 (or, minimally, that a cs.CL timeout no longer discards a good cs.AI).

Source file: scripts/seed-research.mjs · Runner: scripts/_seed-utils.mjs:1711-1753