Summary

Wraps atomicPublish's 3-call publish unit (staging SET → canonical SET → staging DEL) in withRetry so a single transient Upstash REST timeout no longer kills a multi-minute seeder run. Also tags redisCommand errors with nonRetryable: true for permanent 4xx and retryAfterMs for 429 — so the existing withRetry helper makes the right backoff decision.

Why now

WM 2026-05-10 incident: seed-forecasts cron failed with a Redis SET timeout on a 0.17MB payload (well under any limit) after completing the entire 3-minute run — LLM calls, 42 forecasts filtered, full publish payload built. Stack trace:

DOMException [TimeoutError]: The operation was aborted due to timeout
  at async redisCommand (_seed-utils.mjs:140:16)
  at async atomicPublish (_seed-utils.mjs:254:5)
  at async runSeed (_seed-utils.mjs:1236:27)

Cron just waited an hour for the next tick. forecasts + marketImplications stayed STALE_SEED for ~3 hours.

R2 timeouts and OpenRouter LLM failures earlier in the same run were caught and degraded gracefully (fallback narratives applied). The ONLY fragility was the publish step itself.

Companion to PR #3649

PR #3649 caps digest:replay-log:v1:* lists at 100K entries to prevent Upstash 500MB overages that were back-pressuring these timeouts. Together:

• fix(brief-dedup): cap replay-log lists to prevent Upstash 500MB record limit #3649 removes the source of the back-pressure
• This PR removes the single point of failure even when other transients hit

Changes

1. redisCommand — error tagging (no retry on its own)

   if (!resp.ok) {
     const text = await resp.text().catch(() => '');
-    throw new Error(`Redis command failed: HTTP ${resp.status} — ${text.slice(0, 200)}`);
+    const err = new Error(`Redis command failed: HTTP ${resp.status} — ${text.slice(0, 200)}`);
+    if (PERMANENT_4XX_STATUSES.has(resp.status)) {
+      err.nonRetryable = true;
+    } else if (resp.status === 429) {
+      const retryAfterMs = parseRetryAfterMs(resp.headers.get('retry-after'));
+      if (retryAfterMs != null) err.retryAfterMs = retryAfterMs;
+    }
+    err.httpStatus = resp.status;
+    throw err;
   }

redisCommand itself stays single-attempt — auto-retrying every redisCommand caller (e.g. acquireLock with NX semantics) would change behavior subtly. Only callers that opt in via withRetry get the retry behavior.

2. atomicPublish — wrap the body in withRetry

-  const runId = String(Date.now());
-  const stagingKey = `${canonicalKey}:staging:${runId}`;
   ...
-  await redisSet(url, token, stagingKey, payloadValue, 300);
-  if (ttlSeconds) await redisCommand(url, token, ['SET', canonicalKey, payload, 'EX', ttlSeconds]);
-  else await redisCommand(url, token, ['SET', canonicalKey, payload]);
-  await redisDel(url, token, stagingKey).catch(() => {});
-  return { payloadBytes, recordCount: ... };
+  return await withRetry(
+    async () => {
+      const runId = `${Date.now()}-${Math.random().toString(36).slice(2, 8)}`;
+      const stagingKey = `${canonicalKey}:staging:${runId}`;
+      await redisSet(url, token, stagingKey, payloadValue, 300);
+      if (ttlSeconds) await redisCommand(url, token, ['SET', canonicalKey, payload, 'EX', ttlSeconds]);
+      else await redisCommand(url, token, ['SET', canonicalKey, payload]);
+      await redisDel(url, token, stagingKey).catch(() => {});
+      return { payloadBytes, recordCount: ... };
+    },
+    2,    // 2 retries (3 attempts total)
+    1000, // 1s base; exponential → 1s, 2s, 4s ≈ 7s worst case
+  );

Each attempt re-stages with a fresh runId so a previous attempt's staging key (if the SET landed server-side but the response was lost) doesn't shadow the retry. The 5-min staging TTL cleans up orphans automatically.

Retry decision matrix

408 and 429 are NOT in PERMANENT_4XX_STATUSES (per PR #3635 review) — they're explicit "try again" signals.

Test plan

• node --test tests/seed-utils-with-retry.test.mjs — 17/17 pass
  • 5 new tests for atomicPublish retry behavior:
    • happy path (no retry, 3 fetch calls)
    • transient 503 → retry → success (4 calls, ≥1s elapsed for 1s backoff)
    • permanent 401 → fast-fail, no retry (1 call, <500ms)
    • persistent 503 → exhausts 3 attempts then throws
    • 429 with Retry-After: 2 → wait ≥2s before retry
  • 12 existing tests for withRetry, parseRetryAfterMs, PERMANENT_4XX_STATUSES, isTransientRedisError all unchanged
• node --test tests/seed-utils*.test.mjs — 54/54 pass (full seed-utils surface)
• Post-merge: monitor seed-forecasts (and other long-running seeders) for crashes on Redis timeouts. Expectation: transient timeouts retry silently; only persistent failures (3+ attempts) surface

What this DOES NOT change

• Permanent failures (auth, payload-too-large) still abort fast — no useless backoff
• The 5MB MAX_PAYLOAD_BYTES check stays at the top of atomicPublish — payload validation runs once, not per attempt
• acquireLock's single-attempt behavior unchanged — wrap at call site (acquireLockSafely:201) handles retry semantics specifically for NX
• Other Redis read paths (redisGet) unchanged — their callers can opt in via withRetry as needed