Skip to content

Instantly share code, notes, and snippets.

@adalundhe
Created November 11, 2026 20:43
Show Gist options
  • Select an option

  • Save adalundhe/426f7e31388b2a59df517572788e9e2c to your computer and use it in GitHub Desktop.

Select an option

Save adalundhe/426f7e31388b2a59df517572788e9e2c to your computer and use it in GitHub Desktop.
kagent-dev/kagent#3399: crash during post-turn suspension, before/after on kind

kagent-dev/kagent#3399: crash during post-turn suspension

Why this was added

After every turn, kagent suspends the Session's Actor to a durable snapshot. If the worker is lost during that suspend, for example in a node drain, a rollout, a pool scale-down or spot loss (the routine events in #394), the result is:

  • The Actor is CRASHED, while GetSession still reports READY/NONE.
  • The idle worker keeps the suspension's claim, because the suspend's outcome is unknown. Nothing takes the claim over on a timeout.
  • Every later send waits out the 10-second busy period and gets KAGENT_SEND_NOT_ACCEPTED. A CLI-style client gives up after 30 seconds.
  • Even with sessions.automaticRecovery: true, nothing recovers the Session. Only an explicit RevertSession or a Delete retires the claim.

This is the same kind of failure as #394's mid-turn crash, at a different moment. Without this change, automatic recovery would cover a crash during a turn but not one a moment after it.

The change (4db2c400)

  • Reservation: a held suspension claim is reported as ErrDispatchSuspending, a kind of ErrDispatchBusy. Other callers of requireSettledRuntime are unchanged.
  • Send path: with the flag on, a send whose 10-second wait ends on that claim attempts the existing bounded recovery once. A healthy suspension finishes well within the wait. Admitting the recovery retires the claim, and the send reserves again without waiting.
  • Unchanged: if the suspension is still running, Revert leaves it alone and the send answers retry as before. With the flag off, nothing changes.

How it was reproduced

  • Cluster: kind with Substrate 1.0.0, gVisor workers, a mock model on the host, and automaticRecovery: true. The client is a probe that retries like the CLI: only on KAGENT_SEND_NOT_ACCEPTED, 100 ms apart, for 30 seconds.
  • Widening the window: with the mock agent's tiny state, a suspend finishes within milliseconds. A first attempt deleted the worker 20 ms into SUSPENDING, and the suspend still completed. In production the window is as long as the snapshot takes, which grows with the agent's state and storage speed. To recreate a slow suspend, Substrate's object store (rustfs) was scaled to 0 just before the turn completed and restored right after the crash.
  • Trigger: the probe polled the Actor and force-deleted its worker pod the moment it reported SUSPENDING.
  • State shown: Actor and Session state come from ListSubstrateActors and GetSession. The claim comes straight from kagent's PostgreSQL (session_task_event.quiescence_pending and quiescence_executor_id).

Log excerpts below are trimmed; the database table is shortened to one DB: line. 1-before-fix.log and 2-after-fix.log are the unedited output.

Before the fix (controller at 9d9f5f98) — full log: 1-before-fix.log

20:33:39.129Z actor -> ACTOR_STATE_SUSPENDING worker=kagent-default-64c864c885-n2kms
20:33:39.151Z deleted worker pod kagent-default-64c864c885-n2kms during suspension
20:34:09.120Z controller: ERROR runtime boundary outcome unknown | suspend Actor ...: DeadlineExceeded
20:34:20.301Z actor -> ACTOR_STATE_CRASHED
20:34:29.077Z actor state=ACTOR_STATE_CRASHED
20:34:29.091Z GetSession state=RUNTIME_STATE_READY operation=RUNTIME_OPERATION_NONE
              DB: quiescence_pending=t claimed=t
20:34:39.223Z +10.0s attempt 1: FailedPrecondition "input was not accepted; ..." reason=KAGENT_SEND_NOT_ACCEPTED
20:34:49.328Z +20.1s attempt 2: FailedPrecondition ... KAGENT_SEND_NOT_ACCEPTED
20:34:59.432Z +30.2s attempt 3: FailedPrecondition ... KAGENT_SEND_NOT_ACCEPTED
20:34:59.432Z sendloop done after 3 attempts in 30.2s, accepted=false
-- six minutes later, nothing changed:
20:39:54.868Z actor state=ACTOR_STATE_CRASHED
20:39:54.884Z GetSession state=RUNTIME_STATE_READY operation=RUNTIME_OPERATION_NONE
20:40:04.900Z +10.0s attempt 1: FailedPrecondition ... KAGENT_SEND_NOT_ACCEPTED

After the fix (controller at 4db2c400, image sha256:1885a9ce…) — full log: 2-after-fix.log

A. The Session left stuck above, after the upgrade, recovers on its first send:

20:41:02.351Z actor state=ACTOR_STATE_CRASHED
20:41:02.368Z GetSession state=RUNTIME_STATE_READY operation=RUNTIME_OPERATION_NONE
20:41:12.609Z +10.2s attempt 1: OK task=TASK_STATE_COMPLETED
              DB: no suspension claim left

B. A fresh reproduction with the fix:

20:41:39.326Z actor -> ACTOR_STATE_SUSPENDING worker=kagent-default-64c864c885-64d57
20:41:39.349Z deleted worker pod kagent-default-64c864c885-64d57 during suspension
20:41:40.849Z controller: ERROR runtime boundary outcome unknown | suspend Actor ...: workflow failed at step CallAteletSuspend
20:41:40.991Z actor -> ACTOR_STATE_CRASHED
20:41:52.671Z actor state=ACTOR_STATE_CRASHED
20:41:52.687Z GetSession state=RUNTIME_STATE_READY operation=RUNTIME_OPERATION_NONE
              DB: quiescence_pending=t claimed=t
20:42:02.976Z +10.2s attempt 1: OK task=TASK_STATE_COMPLETED
20:42:03.003Z actor state=ACTOR_STATE_RUNNING   (the new turn's runtime)

Controller RPC log for run B, from the controller's own rpc completed lines:

20:41:36.815Z SendMessage OK   145ms    first turn
20:41:39.321Z SendMessage OK  1857ms    second turn; the suspension that crashes starts after it
20:42:02.970Z SendMessage OK 10189ms    next send: waits out the claim, recovers, is answered

Tests

  • TestDispatchReportsUnsettledSuspension (PostgreSQL): a held suspension claim is reported as ErrDispatchSuspending, while an ordinary dispatch in flight is not. Admitting a crash recovery retires the claim, and reservation then succeeds.
  • TestSendRecoversACrashDuringSuspension:
    • a crashed suspension is recovered and the send proceeds;
    • a suspension that is still running answers retry, with no second wait;
    • a revert that can't run, the flag being off, and an ordinary busy dispatch all answer retry without recovering.

Not covered

There's no e2e test for this: hitting the window needs a slow suspend and a worker lost inside it. That's why the reproduction above controls object storage, which a shared e2e cluster can't do safely.

session 01a1cc50-ead2-7f2e-96c2-3dadc2e0a237 (controller 9d9f5f98, automaticRecovery: true)
20:33:37.773Z +0.3s attempt 1: OK task=TASK_STATE_COMPLETED
20:33:37.825Z actor state=ACTOR_STATE_SUSPENDED worker=
--- second turn held at the model
20:33:38.000Z object storage (rustfs) scaled to 0 so the coming suspend cannot finish quickly
20:33:39.000Z model released; the turn completes and quiescence suspends the Actor
20:33:37.858Z actor -> ACTOR_STATE_SUSPENDED worker=
20:33:37.985Z actor -> ACTOR_STATE_RESUMING worker=kagent-default-64c864c885-n2kms
20:33:38.050Z actor -> ACTOR_STATE_RUNNING worker=kagent-default-64c864c885-n2kms
20:33:39.120Z +1.3s attempt 1: OK task=TASK_STATE_COMPLETED
20:33:39.129Z actor -> ACTOR_STATE_SUSPENDING worker=kagent-default-64c864c885-n2kms
20:33:39.151Z deleted worker pod kagent-default-64c864c885-n2kms during suspension
20:34:20.301Z actor -> ACTOR_STATE_CRASHED
crash-rc=0
20:34:20.000Z object storage restored
--- state after the crash
20:34:29.077Z actor state=ACTOR_STATE_CRASHED worker=
20:34:29.091Z GetSession state=RUNTIME_STATE_READY operation=RUNTIME_OPERATION_NONE failure=""
kagent DB: latest task event for the Session (claimed = a suspension holds its claim)
sequence | published | quiescence_pending | claimed
----------+-----------+--------------------+---------
210 | t | t | t
(1 row)
--- controller log for the failed suspension
20:34:09.120Z ERROR runtime boundary outcome unknown | suspend Actor kagent/session-01a1cc50-ead2-7f2e-96c2-3dadc2e0a237: rpc error: code = DeadlineExceeded desc = context deadline exceeded
--- next message, retried like the CLI (KAGENT_SEND_NOT_ACCEPTED, 100ms apart, 30s)
20:34:39.223Z +10.0s attempt 1: FailedPrecondition "input was not accepted; retry after the session becomes available" metadata=map[reason:KAGENT_SEND_NOT_ACCEPTED retryAfterMs:100]
20:34:49.328Z +20.1s attempt 2: FailedPrecondition "input was not accepted; retry after the session becomes available" metadata=map[reason:KAGENT_SEND_NOT_ACCEPTED retryAfterMs:100]
20:34:59.432Z +30.2s attempt 3: FailedPrecondition "input was not accepted; retry after the session becomes available" metadata=map[reason:KAGENT_SEND_NOT_ACCEPTED retryAfterMs:100]
20:34:59.432Z sendloop done after 3 attempts in 30.2s, accepted=false
--- still stuck later (no client action in between)
20:39:54.868Z actor state=ACTOR_STATE_CRASHED worker=
20:39:54.884Z GetSession state=RUNTIME_STATE_READY operation=RUNTIME_OPERATION_NONE failure=""
20:40:04.900Z +10.0s attempt 1: FailedPrecondition "input was not accepted; retry after the session becomes available" metadata=map[reason:KAGENT_SEND_NOT_ACCEPTED retryAfterMs:100]
20:40:04.904Z sendloop done after 1 attempts in 10.0s, accepted=false
controller rebuilt with the fix (automaticRecovery: true); image sha256:1885a9ce36f5
=== A. the Session left stuck by the earlier run: 01a1cc50-ead2-7f2e-96c2-3dadc2e0a237
20:41:02.351Z actor state=ACTOR_STATE_CRASHED worker=
20:41:02.368Z GetSession state=RUNTIME_STATE_READY operation=RUNTIME_OPERATION_NONE failure=""
--- one CLI-style send, no RevertSession call
20:41:12.609Z +10.2s attempt 1: OK task=TASK_STATE_COMPLETED
20:41:12.609Z sendloop done after 1 attempts in 10.2s, accepted=true
20:41:12.636Z actor state=ACTOR_STATE_SUSPENDING worker=kagent-default-64c864c885-m8t6v
20:41:12.656Z GetSession state=RUNTIME_STATE_READY operation=RUNTIME_OPERATION_NONE failure=""
sequence | published | quiescence_pending | claimed
----------+-----------+--------------------+---------
(0 rows)
=== B. fresh reproduction with the fix: 01a1cc58-3a75-7f41-afae-a5a8c14c8010
20:41:36.816Z +0.1s attempt 1: OK task=TASK_STATE_COMPLETED
20:41:37.410Z actor state=ACTOR_STATE_SUSPENDED worker=
--- second turn held at the model
20:41:37.000Z object storage (rustfs) scaled to 0 so the coming suspend cannot finish quickly
20:41:39.000Z model released; the turn completes and quiescence suspends the Actor
20:41:37.466Z actor -> ACTOR_STATE_SUSPENDED worker=
20:41:37.592Z actor -> ACTOR_STATE_RESUMING worker=kagent-default-64c864c885-64d57
20:41:37.664Z actor -> ACTOR_STATE_RUNNING worker=kagent-default-64c864c885-64d57
20:41:39.322Z +1.9s attempt 1: OK task=TASK_STATE_COMPLETED
20:41:39.326Z actor -> ACTOR_STATE_SUSPENDING worker=kagent-default-64c864c885-64d57
20:41:39.349Z deleted worker pod kagent-default-64c864c885-64d57 during suspension
20:41:40.991Z actor -> ACTOR_STATE_CRASHED
crash-rc=0
20:41:41.000Z object storage restored
--- state after the crash
20:41:52.671Z actor state=ACTOR_STATE_CRASHED worker=
20:41:52.687Z GetSession state=RUNTIME_STATE_READY operation=RUNTIME_OPERATION_NONE failure=""
kagent DB: latest task event for the Session
sequence | published | quiescence_pending | claimed
----------+-----------+--------------------+---------
240 | t | t | t
(1 row)
--- next message, retried like the CLI (KAGENT_SEND_NOT_ACCEPTED, 100ms apart, 30s)
20:42:02.976Z +10.2s attempt 1: OK task=TASK_STATE_COMPLETED
20:42:02.976Z sendloop done after 1 attempts in 10.2s, accepted=true
--- afterwards
20:42:03.003Z actor state=ACTOR_STATE_RUNNING worker=kagent-default-64c864c885-5mmtv
20:42:03.019Z GetSession state=RUNTIME_STATE_READY operation=RUNTIME_OPERATION_NONE failure=""
claims still held: 0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment