Skip to content

Commit 64697d2

Browse files
liaoandiclaude
andcommitted
fix: scope ghost-run detection to wakeMode=now one-shot jobs
next-heartbeat mode is fire-and-forget — every healthy invocation completes in <50 ms, so a duration check produces 100% false positives. Recurring now-mode jobs are also excluded because the busy-lane early return (#58833) is a legitimate fast path, not a ghost run. Addresses review feedback: - P2: exclude busy main-lane fast path - P2: avoid warning on healthy next-heartbeat jobs Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
1 parent 971215e commit 64697d2

1 file changed

Lines changed: 18 additions & 12 deletions

File tree

src/gateway/server-cron.ts

Lines changed: 18 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -505,26 +505,31 @@ export function buildGatewayCronService(params: {
505505
}
506506
}
507507

508-
// Ghost-run detection: main-session systemEvent jobs with
509-
// wakeMode "next-heartbeat" return almost instantly because
510-
// executeMainSessionCronJob only enqueues a system event and
511-
// returns without waiting for the agent to process it. If the
512-
// gateway or agent session is unhealthy the event is silently
513-
// dropped, yet the run is recorded as ok. Warn when durationMs
514-
// is suspiciously low so operators can identify silent failures.
508+
// Ghost-run detection for main-session systemEvent jobs.
515509
//
516-
// Note: recurring main-session jobs also return fast when the
517-
// main lane is busy (requests-in-flight). We accept this as a
518-
// low-frequency false positive — the warning is non-blocking and
519-
// the structured fields let operators triage.
510+
// wakeMode "now" waits for runHeartbeatOnce to complete before
511+
// returning. A sub-threshold ok result means the heartbeat ran
512+
// but found nothing to process — the enqueued system event was
513+
// likely dropped silently, which is the ghost-run scenario.
514+
//
515+
// wakeMode "next-heartbeat" is excluded: it fires
516+
// requestHeartbeatNow and returns immediately (fire-and-forget),
517+
// so every healthy invocation completes in < 50 ms and a
518+
// duration check would produce 100 % false positives.
519+
//
520+
// Recurring "now" jobs are excluded: when the main lane is busy
521+
// (requests-in-flight) the timer returns early with status "ok"
522+
// to avoid blocking the cron lane (#58833), which is a
523+
// legitimate fast return, not a ghost run.
520524
if (
521525
evt.status === "ok" &&
522526
typeof evt.durationMs === "number" &&
523527
evt.durationMs < GHOST_RUN_THRESHOLD_MS &&
524528
job &&
525529
job.sessionTarget === "main" &&
526530
job.payload.kind === "systemEvent" &&
527-
job.wakeMode === "next-heartbeat"
531+
job.wakeMode === "now" &&
532+
job.schedule.kind === "at"
528533
) {
529534
cronLogger.warn(
530535
{
@@ -533,6 +538,7 @@ export function buildGatewayCronService(params: {
533538
sessionTarget: job.sessionTarget,
534539
payloadKind: job.payload.kind,
535540
wakeMode: job.wakeMode,
541+
scheduleKind: job.schedule.kind,
536542
ghostRunThresholdMs: GHOST_RUN_THRESHOLD_MS,
537543
},
538544
"cron: possible ghost run detected — job completed suspiciously fast; gateway may be unhealthy",

0 commit comments

Comments
 (0)