Skip to content

Commit a8e3b81

Browse files
adalundheclaude
andcommitted
docs: scope send recovery in the retry guide to automaticRecovery
By default, sends do not recover a crashed runtime: a crashed turn is refused with KAGENT_TURN_ACTIVE, and a send to a pending recovery is refused outright. Say to call RevertSession in those cases, describe recovery on send under the flag, and document state RECOVERING with its retry delay. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: Ada Lundhe <[email protected]>
1 parent fb9a31d commit a8e3b81

1 file changed

Lines changed: 25 additions & 4 deletions

File tree

‎docs/lifecycle-retries.md‎

Lines changed: 25 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -41,9 +41,30 @@ should likewise retain the `request_id` passed to `create_sandbox`.
4141

4242
## Recovering a crashed Session
4343

44-
A send recovers a crashed runtime. It returns `KAGENT_SEND_NOT_ACCEPTED` after reverting, so retry the send as usual.
45-
A send refused because a crashed turn is still active recovers the runtime first and then proceeds.
46-
A send while recovery is pending joins it. Recovering through a send needs Session update access.
44+
By default, sends do not recover a crashed runtime, and a crashed Session still
45+
reads `READY` until something reverts it. Call `RevertSession`, then send again:
46+
47+
- A turn interrupted by the crash leaves that turn open. Later sends are refused
48+
with `FAILED_PRECONDITION` and ErrorInfo reason `KAGENT_TURN_ACTIVE`, which is not
49+
retryable.
50+
- A crash while the runtime suspends after a turn makes every send wait and return
51+
`KAGENT_SEND_NOT_ACCEPTED`. Once your retry budget runs out, revert.
52+
- While a recovery is pending, sends are refused because a lifecycle operation is
53+
active. Call `RevertSession` again to finish it.
54+
55+
With `sessions.automaticRecovery: true` in controller config, a send recovers the
56+
runtime itself:
57+
58+
- Three kinds of send start a revert: one refused for an open turn, one blocked by
59+
a suspension that crashed, and one the crashed runtime refused. The send then
60+
proceeds, or returns `KAGENT_SEND_NOT_ACCEPTED` so you retry.
61+
- A send while recovery is pending joins it.
62+
- A send waits at most 10 seconds for recovery. If recovery is still running, it
63+
returns `KAGENT_SEND_NOT_ACCEPTED` with metadata `state: RECOVERING` and
64+
`retryAfterMs: 1000`. Keep retrying; the retry continues the recovery.
65+
- Recovering through a send needs Session update access. Without it, or when
66+
Substrate is unavailable, the send gets the answer it would get by default.
67+
4768
Call `SessionService.RevertSession` with the Session ID to restore the recorded runtime to its last durable snapshot without sending.
4869
The caller needs Session update access. The operation verifies Substrate's CRASHED state and the recorded Actor identity.
4970
Recovery fails interrupted turns and waiting tasks absent from the retained snapshot.
@@ -54,7 +75,7 @@ A runtime without a snapshot restarts from its initial state.
5475
Retry an interrupted revert against the same Session ID; nothing retries it in the background.
5576
Pending recovery retains its operation across API restarts, and a retry joins it.
5677
While recovery is pending, `GetSession` reports state `FAILED`, operation `REVERT`, and failure reason `RuntimeCrashed`.
57-
Retry the send, or call `RevertSession` again, to finish the recovery.
78+
Call `RevertSession` again to finish the recovery. With automatic recovery on, a retried send also continues it.
5879
A retry after a revert whose response was lost returns the recovered Session.
5980
Checkpoint creation returns `ABORTED` until it finishes.
6081
A revert that observed the crash before another lifecycle operation began also returns `ABORTED`.

0 commit comments

Comments
 (0)