kagent-dev/kagent#3399: crash during post-turn suspension
After every turn, kagent suspends the Session's Actor to a durable snapshot. If the worker is lost during that suspend, for example in a node drain, a rollout, a pool scale-down or spot loss (the routine events in #394), the result is:
- The Actor is
CRASHED, whileGetSessionstill reportsREADY/NONE. - The idle worker keeps the suspension's claim, because the suspend's outcome is unknown. Nothing takes the claim over on a timeout.
- Every later send waits out the 10-second busy period and gets
KAGENT_SEND_NOT_ACCEPTED. A CLI-style client gives up after 30 seconds. - Even with
sessions.automaticRecovery: true, nothing recovers the Session. Only an explicitRevertSessionor a Delete retires the claim.
This is the same kind of failure as #394's mid-turn crash, at a different moment. Without this change, automatic recovery would cover a crash during a turn but not one a moment after it.
- Reservation: a held suspension claim is reported as
ErrDispatchSuspending, a kind ofErrDispatchBusy. Other callers ofrequireSettledRuntimeare unchanged. - Send path: with the flag on, a send whose 10-second wait ends on that claim attempts the existing bounded recovery once. A healthy suspension finishes well within the wait. Admitting the recovery retires the claim, and the send reserves again without waiting.
- Unchanged: if the suspension is still running, Revert leaves it alone and the send answers retry as before. With the flag off, nothing changes.
- Cluster: kind with Substrate 1.0.0, gVisor workers, a mock model on the host, and
automaticRecovery: true. The client is a probe that retries like the CLI: only onKAGENT_SEND_NOT_ACCEPTED, 100 ms apart, for 30 seconds. - Widening the window: with the mock agent's tiny state, a suspend finishes within milliseconds. A first attempt deleted the worker 20 ms into
SUSPENDING, and the suspend still completed. In production the window is as long as the snapshot takes, which grows with the agent's state and storage speed. To recreate a slow suspend, Substrate's object store (rustfs) was scaled to 0 just before the turn completed and restored right after the crash. - Trigger: the probe polled the Actor and force-deleted its worker pod the moment it reported
SUSPENDING. - State shown: Actor and Session state come from
ListSubstrateActorsandGetSession. The claim comes straight from kagent's PostgreSQL (session_task_event.quiescence_pendingandquiescence_executor_id).
Log excerpts below are trimmed; the database table is shortened to one DB: line. 1-before-fix.log and 2-after-fix.log are the unedited output.
20:33:39.129Z actor -> ACTOR_STATE_SUSPENDING worker=kagent-default-64c864c885-n2kms
20:33:39.151Z deleted worker pod kagent-default-64c864c885-n2kms during suspension
20:34:09.120Z controller: ERROR runtime boundary outcome unknown | suspend Actor ...: DeadlineExceeded
20:34:20.301Z actor -> ACTOR_STATE_CRASHED
20:34:29.077Z actor state=ACTOR_STATE_CRASHED
20:34:29.091Z GetSession state=RUNTIME_STATE_READY operation=RUNTIME_OPERATION_NONE
DB: quiescence_pending=t claimed=t
20:34:39.223Z +10.0s attempt 1: FailedPrecondition "input was not accepted; ..." reason=KAGENT_SEND_NOT_ACCEPTED
20:34:49.328Z +20.1s attempt 2: FailedPrecondition ... KAGENT_SEND_NOT_ACCEPTED
20:34:59.432Z +30.2s attempt 3: FailedPrecondition ... KAGENT_SEND_NOT_ACCEPTED
20:34:59.432Z sendloop done after 3 attempts in 30.2s, accepted=false
-- six minutes later, nothing changed:
20:39:54.868Z actor state=ACTOR_STATE_CRASHED
20:39:54.884Z GetSession state=RUNTIME_STATE_READY operation=RUNTIME_OPERATION_NONE
20:40:04.900Z +10.0s attempt 1: FailedPrecondition ... KAGENT_SEND_NOT_ACCEPTED
A. The Session left stuck above, after the upgrade, recovers on its first send:
20:41:02.351Z actor state=ACTOR_STATE_CRASHED
20:41:02.368Z GetSession state=RUNTIME_STATE_READY operation=RUNTIME_OPERATION_NONE
20:41:12.609Z +10.2s attempt 1: OK task=TASK_STATE_COMPLETED
DB: no suspension claim left
B. A fresh reproduction with the fix:
20:41:39.326Z actor -> ACTOR_STATE_SUSPENDING worker=kagent-default-64c864c885-64d57
20:41:39.349Z deleted worker pod kagent-default-64c864c885-64d57 during suspension
20:41:40.849Z controller: ERROR runtime boundary outcome unknown | suspend Actor ...: workflow failed at step CallAteletSuspend
20:41:40.991Z actor -> ACTOR_STATE_CRASHED
20:41:52.671Z actor state=ACTOR_STATE_CRASHED
20:41:52.687Z GetSession state=RUNTIME_STATE_READY operation=RUNTIME_OPERATION_NONE
DB: quiescence_pending=t claimed=t
20:42:02.976Z +10.2s attempt 1: OK task=TASK_STATE_COMPLETED
20:42:03.003Z actor state=ACTOR_STATE_RUNNING (the new turn's runtime)
Controller RPC log for run B, from the controller's own rpc completed lines:
20:41:36.815Z SendMessage OK 145ms first turn
20:41:39.321Z SendMessage OK 1857ms second turn; the suspension that crashes starts after it
20:42:02.970Z SendMessage OK 10189ms next send: waits out the claim, recovers, is answered
TestDispatchReportsUnsettledSuspension(PostgreSQL): a held suspension claim is reported asErrDispatchSuspending, while an ordinary dispatch in flight is not. Admitting a crash recovery retires the claim, and reservation then succeeds.TestSendRecoversACrashDuringSuspension:- a crashed suspension is recovered and the send proceeds;
- a suspension that is still running answers retry, with no second wait;
- a revert that can't run, the flag being off, and an ordinary busy dispatch all answer retry without recovering.
There's no e2e test for this: hitting the window needs a slow suspend and a worker lost inside it. That's why the reproduction above controls object storage, which a shared e2e cluster can't do safely.