Summary
One-shot (at) jobs correctly detect transient errors (network, timeout, rate-limit, 5xx) and retry with exponential backoff. Recurring (cron / every) jobs have no such logic — on any error they simply wait until the next natural schedule time, which can be hours or days away.
Root Cause
In src/cron/service/timer.ts, the two code paths are asymmetric:
One-shot jobs — transient detection present ✅ (lines 379–396)
const retryConfig = resolveRetryConfig(state.deps.cronConfig);
const transient = isTransientCronError(result.error, retryConfig.retryOn);
if (transient && consecutive <= retryConfig.maxAttempts) {
// Schedule retry with backoff
job.state.nextRunAtMs = result.endedAt + backoff;
}Recurring jobs — transient detection absent ❌ (lines 416–434)
const backoff = errorBackoffMs(job.state.consecutiveErrors ?? 1);
let normalNext: number | undefined;
// ...
const backoffNext = result.endedAt + backoff;
// Use whichever is later: the natural next run or the backoff delay.
job.state.nextRunAtMs =
normalNext !== undefined ? Math.max(normalNext, backoffNext) : backoffNext;
Math.max(normalNext, backoffNext) picks whichever is later. For long-period jobs (hourly, daily) normalNext is always larger, so backoffNext is always ignored.
Impact
Daily job fails at 10:00 due to network jitter
→ Current: next run at 10:00 tomorrow (24 hours later)
→ Expected: retry after 30s; if still failing, 1m → 5m → ... → back to schedule
Transient errors that would self-heal in seconds force a full schedule wait. The consecutiveErrors counter increments correctly but the retry shortcut never fires.
Suggested Fix
Apply the same transient-error check to recurring jobs:
} else if (result.status === "error" && job.enabled) {
const retryConfig = resolveRetryConfig(state.deps.cronConfig);
const transient = isTransientCronError(result.error, retryConfig.retryOn);
const backoff = errorBackoffMs(job.state.consecutiveErrors ?? 1);
let normalNext: number | undefined;
try {
normalNext = opts?.preserveSchedule && job.schedule.kind === "every"
? computeNextWithPreservedLastRun(result.endedAt)
: computeJobNextRunAtMs(job, result.endedAt);
} catch (err) {
recordScheduleComputeError({ state, job, err });
}
const backoffNext = result.endedAt + backoff;
if (transient) {
// Transient error: retry soon with backoff, don't wait for next schedule
job.state.nextRunAtMs = backoffNext;
} else {
// Permanent error: respect natural schedule (backoff guards short-interval jobs)
job.state.nextRunAtMs = normalNext !== undefined
? Math.max(normalNext, backoffNext)
: backoffNext;
}
}Transient patterns already defined (TRANSIENT_PATTERNS): rate_limit, network, timeout, server_error — no new infrastructure needed.
Summary
One-shot (
at) jobs correctly detect transient errors (network, timeout, rate-limit, 5xx) and retry with exponential backoff. Recurring (cron/every) jobs have no such logic — on any error they simply wait until the next natural schedule time, which can be hours or days away.Root Cause
In
src/cron/service/timer.ts, the two code paths are asymmetric:One-shot jobs — transient detection present ✅ (lines 379–396)
Recurring jobs — transient detection absent ❌ (lines 416–434)
Math.max(normalNext, backoffNext)picks whichever is later. For long-period jobs (hourly, daily)normalNextis always larger, sobackoffNextis always ignored.Impact
Transient errors that would self-heal in seconds force a full schedule wait. The
consecutiveErrorscounter increments correctly but the retry shortcut never fires.Suggested Fix
Apply the same transient-error check to recurring jobs:
Transient patterns already defined (
TRANSIENT_PATTERNS):rate_limit,network,timeout,server_error— no new infrastructure needed.