Repository navigation
Commit 31ae728
ateapi, atelet: make a pause durable — upload its snapshot in the background and resume it on any worker
A pause checkpoints the workload onto its worker's node and nowhere else:
the node holds the only copy of the actor's state, and the record pins the
actor to that node (LocalSnapshotInfo.node_vms_with_local_snapshots →
Constraints.RequiredNodes). A node that goes away — preempted, scaled down,
rolled — takes the state along while the record still says PAUSED there. The
next resume then finds no worker that applies, the scheduler reports the
locality miss as ResourceExhausted "no free workers available" (the message of
a full pool), the router parks the request for its whole budget and every
retry repeats this; a suspend dials the atelet on the dead node and stays
retryable forever, so the actor cannot be suspended or deleted either.
Actors parked on a question for a person sit like this for hours.
Pauses now become durable shortly after they complete. After FinalizePaused
the control plane queues the actor with a background uploader (two workers
per replica, rate-limited requeues, a resync every minute that also finds
pauses from before an upgrade or a restart). The upload goes through the
atelet on the snapshot's node, the path a suspend of a paused actor already
uses, with two differences: the destination is named after the local
snapshot, so a retry re-sends the same request and overwrites the same
objects, and the atelet keeps the local copy (UploadPausedCheckpointRequest
.keep_local) — the actor stays PAUSED and its node remains the fast path. On
success the copy is recorded as the actor's external snapshot with the local
snapshot it came from (ExternalSnapshot.source_local_snapshot_name), under a
version precondition: no lease is held while the checkpoint streams, a
resume that arrives meanwhile is not delayed, and an upload the actor has
outrun is deleted again. The external snapshot the copy replaces is released
once the record points at the copy.
That marker is what the rest keys on. A paused actor's external snapshot is
its durable copy only while it names the local snapshot the actor holds;
the snapshot of an earlier suspend, or the upload of an earlier pause, is
older state and never stands in for the pause. While no copy exists the
node stays a hard constraint, exactly as before. Once one exists:
- Placement prefers the snapshot's nodes (Constraints.PreferredNodes,
ranked ahead of every other eligible worker, excluding none) instead of
requiring them. Restore takes the local snapshot when the chosen worker
is on such a node and the durable copy — with the scope the pause
captured, memory included — anywhere else. A paused actor placed off its
node with no copy is refused before any atelet is dialed rather than
restored from an older snapshot.
- A suspend commits the copy as captured: nothing is uploaded, no
destination minted, the node not dialed except to drop the local copy
best-effort (the new PruneLocalCheckpoints RPC, bounded to ten seconds).
Narrowing the copy to the template's commit scope would need the node
the actor may be leaving.
- Deleting a paused actor collects the copy with the actor's prefix, as
for a suspended actor, and asks the node to drop the local copy where an
atelet answers.
The upload uses the scope the pause captured, not the commit scope: the
copy must restore exactly as the local snapshot would. Tests cover the
marker, the resync selection and its grace, the constraint switch, the
record commit with the replaced snapshot released, the stale upload
discarded and the concurrent replica's record left alone, the suspend
shortcut at mark and upload, the placement guard, the scheduler's preference
and its fallbacks, and — end to end against the fake atelet — the upload
after a pause, the resume onto another node restoring the copy after the
snapshot's node lost its worker, the resume that still prefers the node, the
suspend that uploads nothing, and the delete that releases both copies.
Signed-off-by: Timo Derstappen <[email protected]>1 parent 4ced1ec commit 31ae728
29 files changed
Lines changed: 1997 additions & 108 deletions
File tree
- benchmarking/locust/common
- cmd
- ateapi
- internal
- controlapi
- functionaltest
- scheduling
- atelet
- docs
- internal/proto/ateletpb
- pkg/proto/ateapipb
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
255 | 255 | | |
256 | 256 | | |
257 | 257 | | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
258 | 261 | | |
259 | 262 | | |
260 | 263 | | |
261 | 264 | | |
262 | 265 | | |
263 | 266 | | |
264 | | - | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
265 | 271 | | |
266 | 272 | | |
267 | 273 | | |
268 | 274 | | |
269 | 275 | | |
270 | 276 | | |
271 | | - | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
272 | 280 | | |
273 | 281 | | |
274 | 282 | | |
| |||
Large diffs are not rendered by default.
Large diffs are not rendered by default.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3859 | 3859 | | |
3860 | 3860 | | |
3861 | 3861 | | |
3862 | | - | |
| 3862 | + | |
3863 | 3863 | | |
3864 | 3864 | | |
3865 | 3865 | | |
| |||
3880 | 3880 | | |
3881 | 3881 | | |
3882 | 3882 | | |
| 3883 | + | |
| 3884 | + | |
| 3885 | + | |
| 3886 | + | |
| 3887 | + | |
| 3888 | + | |
| 3889 | + | |
| 3890 | + | |
| 3891 | + | |
| 3892 | + | |
| 3893 | + | |
| 3894 | + | |
| 3895 | + | |
| 3896 | + | |
| 3897 | + | |
| 3898 | + | |
| 3899 | + | |
| 3900 | + | |
| 3901 | + | |
| 3902 | + | |
| 3903 | + | |
| 3904 | + | |
| 3905 | + | |
| 3906 | + | |
| 3907 | + | |
| 3908 | + | |
| 3909 | + | |
| 3910 | + | |
| 3911 | + | |
| 3912 | + | |
| 3913 | + | |
| 3914 | + | |
| 3915 | + | |
| 3916 | + | |
| 3917 | + | |
| 3918 | + | |
| 3919 | + | |
| 3920 | + | |
| 3921 | + | |
| 3922 | + | |
| 3923 | + | |
| 3924 | + | |
| 3925 | + | |
| 3926 | + | |
| 3927 | + | |
| 3928 | + | |
| 3929 | + | |
| 3930 | + | |
| 3931 | + | |
| 3932 | + | |
| 3933 | + | |
| 3934 | + | |
| 3935 | + | |
| 3936 | + | |
| 3937 | + | |
| 3938 | + | |
| 3939 | + | |
| 3940 | + | |
| 3941 | + | |
| 3942 | + | |
| 3943 | + | |
| 3944 | + | |
| 3945 | + | |
| 3946 | + | |
| 3947 | + | |
| 3948 | + | |
| 3949 | + | |
| 3950 | + | |
| 3951 | + | |
| 3952 | + | |
| 3953 | + | |
| 3954 | + | |
| 3955 | + | |
| 3956 | + | |
| 3957 | + | |
| 3958 | + | |
| 3959 | + | |
| 3960 | + | |
| 3961 | + | |
| 3962 | + | |
| 3963 | + | |
| 3964 | + | |
| 3965 | + | |
| 3966 | + | |
| 3967 | + | |
| 3968 | + | |
| 3969 | + | |
| 3970 | + | |
| 3971 | + | |
| 3972 | + | |
| 3973 | + | |
| 3974 | + | |
| 3975 | + | |
| 3976 | + | |
| 3977 | + | |
| 3978 | + | |
| 3979 | + | |
| 3980 | + | |
| 3981 | + | |
| 3982 | + | |
| 3983 | + | |
| 3984 | + | |
| 3985 | + | |
| 3986 | + | |
| 3987 | + | |
| 3988 | + | |
| 3989 | + | |
| 3990 | + | |
| 3991 | + | |
| 3992 | + | |
| 3993 | + | |
| 3994 | + | |
| 3995 | + | |
| 3996 | + | |
| 3997 | + | |
| 3998 | + | |
| 3999 | + | |
| 4000 | + | |
| 4001 | + | |
| 4002 | + | |
| 4003 | + | |
| 4004 | + | |
| 4005 | + | |
| 4006 | + | |
| 4007 | + | |
| 4008 | + | |
| 4009 | + | |
| 4010 | + | |
| 4011 | + | |
| 4012 | + | |
| 4013 | + | |
| 4014 | + | |
| 4015 | + | |
| 4016 | + | |
| 4017 | + | |
| 4018 | + | |
| 4019 | + | |
| 4020 | + | |
| 4021 | + | |
| 4022 | + | |
| 4023 | + | |
| 4024 | + | |
| 4025 | + | |
| 4026 | + | |
| 4027 | + | |
| 4028 | + | |
| 4029 | + | |
| 4030 | + | |
| 4031 | + | |
| 4032 | + | |
| 4033 | + | |
| 4034 | + | |
| 4035 | + | |
| 4036 | + | |
| 4037 | + | |
| 4038 | + | |
| 4039 | + | |
| 4040 | + | |
| 4041 | + | |
| 4042 | + | |
| 4043 | + | |
| 4044 | + | |
| 4045 | + | |
| 4046 | + | |
| 4047 | + | |
| 4048 | + | |
| 4049 | + | |
| 4050 | + | |
| 4051 | + | |
| 4052 | + | |
| 4053 | + | |
| 4054 | + | |
| 4055 | + | |
| 4056 | + | |
| 4057 | + | |
| 4058 | + | |
| 4059 | + | |
| 4060 | + | |
| 4061 | + | |
| 4062 | + | |
| 4063 | + | |
| 4064 | + | |
| 4065 | + | |
| 4066 | + | |
| 4067 | + | |
| 4068 | + | |
| 4069 | + | |
| 4070 | + | |
| 4071 | + | |
| 4072 | + | |
| 4073 | + | |
| 4074 | + | |
| 4075 | + | |
| 4076 | + | |
| 4077 | + | |
| 4078 | + | |
| 4079 | + | |
| 4080 | + | |
| 4081 | + | |
| 4082 | + | |
| 4083 | + | |
| 4084 | + | |
| 4085 | + | |
| 4086 | + | |
| 4087 | + | |
| 4088 | + | |
| 4089 | + | |
| 4090 | + | |
| 4091 | + | |
| 4092 | + | |
| 4093 | + | |
| 4094 | + | |
| 4095 | + | |
| 4096 | + | |
| 4097 | + | |
| 4098 | + | |
| 4099 | + | |
| 4100 | + | |
| 4101 | + | |
| 4102 | + | |
| 4103 | + | |
| 4104 | + | |
| 4105 | + | |
| 4106 | + | |
| 4107 | + | |
| 4108 | + | |
| 4109 | + | |
| 4110 | + | |
| 4111 | + | |
| 4112 | + | |
| 4113 | + | |
| 4114 | + | |
| 4115 | + | |
| 4116 | + | |
| 4117 | + | |
| 4118 | + | |
| 4119 | + | |
| 4120 | + | |
| 4121 | + | |
| 4122 | + | |
| 4123 | + | |
| 4124 | + | |
| 4125 | + | |
| 4126 | + | |
| 4127 | + | |
| 4128 | + | |
| 4129 | + | |
| 4130 | + | |
| 4131 | + | |
| 4132 | + | |
| 4133 | + | |
| 4134 | + | |
| 4135 | + | |
| 4136 | + | |
| 4137 | + | |
| 4138 | + | |
| 4139 | + | |
| 4140 | + | |
| 4141 | + | |
| 4142 | + | |
| 4143 | + | |
| 4144 | + | |
| 4145 | + | |
| 4146 | + | |
| 4147 | + | |
| 4148 | + | |
| 4149 | + | |
| 4150 | + | |
| 4151 | + | |
| 4152 | + | |
| 4153 | + | |
| 4154 | + | |
| 4155 | + | |
| 4156 | + | |
| 4157 | + | |
| 4158 | + | |
| 4159 | + | |
| 4160 | + | |
| 4161 | + | |
| 4162 | + | |
| 4163 | + | |
| 4164 | + | |
| 4165 | + | |
| 4166 | + | |
| 4167 | + | |
| 4168 | + | |
| 4169 | + | |
| 4170 | + | |
| 4171 | + | |
| 4172 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
130 | 130 | | |
131 | 131 | | |
132 | 132 | | |
| 133 | + | |
| 134 | + | |
133 | 135 | | |
134 | 136 | | |
135 | 137 | | |
| |||
181 | 183 | | |
182 | 184 | | |
183 | 185 | | |
| 186 | + | |
| 187 | + | |
184 | 188 | | |
185 | 189 | | |
186 | 190 | | |
187 | 191 | | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
188 | 225 | | |
189 | 226 | | |
190 | 227 | | |
| |||
0 commit comments