Repository navigation
ate-api-server can apply the snapshot state Checkpoint returns, and logs stale snapshot reports - #2698
Merged
Zoe Zhao (zoez7) merged 3 commits intoOct 15, 2026
Conversation
igooch
changed the title
igooch
force-pushed
the
apply-checkpoint-state
branch
from
October 15, 2026 18:41
916aaa4 to
0485a8b
Compare
Checkpoint and PromoteSnapshot return atelet's snapshot state, whose SnapshotReport nests an atelet Snapshot message. The control plane's SnapshotReport carries the same fields inline, with its own fidelity enum and object type. snapshotReportFromAtelet converts one to the other, and fidelityFromAtelet maps the fidelity enum back. Nothing calls them yet. They let the control plane apply the state atelet returns with the same rules as a ReportActorSnapshotState report.
A ReportActorSnapshotState report whose latest_durable is older than the latest COMPLETED generation, or names a generation the actor no longer records, changes nothing. That's expected for a late report, so it isn't an error, but it is worth seeing when debugging. Log each case at Info with the reported generation and uuid, and for an older one the latest completed generation. A repeat of the latest completed generation isn't logged, since atelet retries reports freely.
igooch
force-pushed
the
apply-checkpoint-state
branch
from
October 15, 2026 18:56
0485a8b to
2d4dede
Compare
igooch
marked this pull request as ready for review
October 15, 2026 19:58
Zoe Zhao (zoez7)
previously approved these changes
Oct 15, 2026
| } | ||
| if latest := findLatestDurableSnapshot(status, ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_COMPLETED); latest.GetGeneration() >= reported.GetGeneration() { | ||
| if latest.GetGeneration() > reported.GetGeneration() { | ||
| slog.InfoContext(ctx, msgDurableOlder, |
Collaborator
There was a problem hiding this comment.
Since the variable msgDurableOlder is only used once, can we inline the string here?
Collaborator
Author
There was a problem hiding this comment.
🤖 I'd keep it. The log test matches records by this message, so inlining it would mean copying the string into the test, where the two could drift apart. crash_test does the same with actorevent.Crashed.Body.
Checkpoint returns atelet's snapshot state for the actor. applyCheckpointState records an upload that state says finished, with the same rules as a ReportActorSnapshotState report. Unlike a report, a state with no local snapshot is not a detach, since the workflow that made the call records where its snapshot lives. No workflow calls applyCheckpointState yet.
igooch
force-pushed
the
apply-checkpoint-state
branch
from
October 15, 2026 20:28
2d4dede to
b141323
Compare
Zoe Zhao (zoez7)
enabled auto-merge
October 15, 2026 20:50
Zoe Zhao (zoez7)
approved these changes
Oct 15, 2026
Merged
via the queue into
agent-substrate:main
with commit Oct 15, 2026
a68c7e1
21 of 24 checks passed
2 tasks
Shrutiya Mohan (shrutiyam-glitch)
pushed a commit
to shrutiyam-glitch/substrate
that referenced
this pull request
Oct 16, 2026
…nd finalizes (agent-substrate#2766) Part of Actor Lifecycle v2 agent-substrate#2069. atelet's `Checkpoint` and `PromoteSnapshot` return its snapshot state for the actor (agent-substrate#2677). The suspend workflow discarded it. This applies it when a suspend finalizes, with the same rules as a `ReportActorSnapshotState` report (agent-substrate#2622, agent-substrate#2698). No proto changes. - `ensureAteletSuspended` and `ensurePausedSnapshotUploaded` return the state atelet reports. - `ensureSuspendedFinalized` applies it with `applyCheckpointState` before it frees the worker, and records the result in the same update that commits `SUSPENDED`. - A durable snapshot records the fidelity atelet reports. For `Checkpoint` and `PromoteSnapshot`, that's the one requested. - An upload atelet reports finished for an older generation is marked `COMPLETED`, so finalize keeps it rather than pruning it as `PENDING`. - `applyLatestDurable` refuses a reported durable snapshot with no fidelity, rather than recording `UNSPECIFIED`. atelet echoes the fidelity it was asked for, and reports already require one. #### Notes for reviewers - **The state is never stale.** Every attempt that reaches finalize has just called atelet, so a retried suspend applies the state from its own call. - **A conflicting state fails the suspend.** Finalize applies atelet's state before it frees the worker. If the state conflicts with the record, for example a different upload URI, finalize returns the error and the actor stays `SUSPENDING` with its worker, so a retry can call atelet again. - **The lifecycle-op metric's fidelity label** stays the requested fidelity. - **With today's atelet, the record doesn't change.** The applied state matches what finalize already records. It starts to matter once atelet uploads in the background: an async suspend's response can report an older snapshot as uploaded before atelet's report arrives, and finalize then keeps that snapshot as `COMPLETED` instead of pruning it. - [ ] Tests pass - [ ] Appropriate changes to documentation are included in the PR
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of Actor Lifecycle v2 #2069, for #2570.
atelet's
Checkpointresponse carries the actor's snapshot state. ate-api-server discards it today. This adds what's needed to apply it, with the same rulesReportActorSnapshotStateuses. It also logs reports that change nothing.SnapshotReportnests an ateletSnapshot. The control plane's carries the same fields inline, with its own fidelity enum and object type.snapshotReportFromAteletconverts one to the other, andfidelityFromAteletmaps the fidelity back.applyCheckpointStaterecords an upload the state says finished, throughapplyLatestDurable. atelet echoes theSnapshotit was asked for, so the fidelity recorded is the one requested. Unlike a report, a state with no local snapshot is not a detach, since the workflow that calledCheckpointrecords where its snapshot lives.latest_durableolder than the latestCOMPLETEDgeneration, or at a generation the actor no longer records, changes nothing. That's expected for a late report, so each case is logged rather than refused. A repeat of the latest isn't logged, since atelet retries reports freely.Nothing calls
applyCheckpointStateyet. A follow-up applies it in the suspend workflow.