You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 71cb2c1
Browse filesBrowse the repository at this point in the historyBrowse files
docs: Explain the eviction window and asynchronous upload timing
The eviction section said an actor must be suspended within 30 minutes
and that an asynchronous suspend leaves the snapshot on the node, which
read as if the 30 minutes were how long the snapshot stays there. Say
that the 30 minutes is the worker's fixed shutdown budget, and spell out
what atelet does after an asynchronous suspend: it schedules the upload
around --snapshot-upload-delay with no promise on when it starts or
finishes, and keeps the actor pinned to the node until
--snapshot-detach-delay, so the snapshot is lost if the node goes away
before the upload completes.
Copy file name to clipboardExpand all lines: docs/api-guide.md
+9-2Lines changed: 9 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -541,9 +541,16 @@ Once a template is `Ready`, creating an actor logically (via `kubectl ate create
541
541
### Eviction
542
542
A worker pod can go away while it hosts running actors: when its `WorkerPool` rolls to a new worker image or pod template, as in an [upgrade](upgrade.md), or scales down, when its node is drained, or when its Spot VM is reclaimed.
543
543
544
-
When the pod gets `SIGTERM`, its worker stops accepting actors and sends `SIGTERM` to the main process of each container of every actor it hosts. Each actor then has 30 minutes, counted from the pod's `SIGTERM`, to be suspended. An actor suspended in that window keeps its state and resumes on another worker like any suspended actor.
544
+
When the pod gets `SIGTERM`, its worker stops accepting actors and sends `SIGTERM` to the main process of each container of every actor it hosts. Each actor then has 30 minutes, counted from the pod's `SIGTERM`, to be suspended. The 30 minutes is the worker's shutdown budget, fixed in ateworker and kept shorter than the pod's one-hour termination grace period so that the worker, not the kubelet, kills what is left. An actor suspended in that window keeps its state and resumes on another worker like any suspended actor.
545
545
546
-
Substrate does not suspend the actor for you. Whatever drives the actor, usually its harness, has to call `SuspendActor` with `DURABILITY_MODE_SYNCHRONOUS` (`kubectl ate suspend actor --mode=sync`) within the 30 minutes: an asynchronous suspend leaves the snapshot on a node that may be about to go away. Only the actor gets the `SIGTERM`, so it has to pass that on, for example on an endpoint the harness polls.
546
+
Substrate does not suspend the actor for you. Whatever drives the actor, usually its harness, has to call `SuspendActor` with `DURABILITY_MODE_SYNCHRONOUS` (`kubectl ate suspend actor --mode=sync`) before those 30 minutes are up. An asynchronous suspend can lose the snapshot if the node goes away:
547
+
548
+
- It returns once the snapshot is on the node.
549
+
- atelet schedules the upload to object storage about `--snapshot-upload-delay` (5 minutes by default) after the snapshot is taken. Neither when the upload starts nor when it finishes is promised.
550
+
- Until atelet detaches the actor, `--snapshot-detach-delay` (30 minutes by default, unrelated to the worker's 30 minutes), the actor stays pinned to the node and resumes there.
551
+
- If the node goes away before the upload completes, the snapshot goes with it.
552
+
553
+
Only the actor gets the `SIGTERM`, so it has to pass that on, for example on an endpoint the harness polls.
547
554
548
555
Every actor that must keep its state through an eviction needs a `SIGTERM` handler that:
0 commit comments