Repository navigation
fix(atelet): Supersede stale system-info volume registrations in atelet - #1684
Merged
Taahir Ahmed (ahmedtd) merged 1 commit intoSep 17, 2026
Merged
Conversation
Shrutiya Mohan (shrutiyam-glitch)
requested a review
from Max Thompson (thompsonmax)
September 16, 2026 21:38
Shrutiya Mohan (shrutiyam-glitch)
marked this pull request as ready for review
September 16, 2026 21:41
Shrutiya Mohan (shrutiyam-glitch)
marked this pull request as draft
September 16, 2026 21:47
| r.actors[actorUID] = actor | ||
| r.mu.Unlock() | ||
| if dup { | ||
| panic(fmt.Sprintf("system-info volumes: actor %s registered twice", actorUID)) |
Collaborator
There was a problem hiding this comment.
I'm a bit worried that with this change, a valid Deregister call (such as those in cmd/atelet/main.go:509 and :1119) would race with a second registration and cause the new registration to get erased.
I think the solution to this is to track the "registration" somehow after its created, so Deregister knows not to delete any second registration.
Example diff:
+// registration is the entry Register created. Undo removes that entry and
+// nothing else, so a caller that fails after registering cannot drop a
+// newer registration for the same actor.
+type registration struct {
+ r *systemInfoVolumeRefresher
+ actor *registeredActor
+}
+
+// Undo drops the registration unless a later Register superseded it.
+// Safe on a nil handle so callers can defer it before Register runs.
+func (h *registration) Undo() {
+ if h == nil {
+ return
+ }
+ h.r.drop(h.actor)
+ h.actor.mu.Lock()
+ h.actor.stale = true
+ h.actor.mu.Unlock()
+}
+
+// drop removes actor from the registry if it is still the live entry.
+func (r *systemInfoVolumeRefresher) drop(actor *registeredActor) {
+ r.mu.Lock()
+ if r.actors[actor.uid] == actor {
+ delete(r.actors, actor.uid)
+ }
+ r.mu.Unlock()
+}
// Register records actorUID's system-info volumes and writes their contents
-// from current cluster state. If actorUID is already registered (for example
-// after a worker pod crash left a stale entry without Terminate), the previous
-// registration is superseded.
-func (r *systemInfoVolumeRefresher) Register(actorUID string, ref resources.ActorRef, volumes []*systemInfoVolume) error {
+// from current cluster state. A second Register for the same actorUID
+// supersedes the first, which stops receiving refreshes. A failed Register
+// leaves no entry behind.
+func (r *systemInfoVolumeRefresher) Register(actorUID string, ref resources.ActorRef, volumes []*systemInfoVolume) (*registration, error) {
actor := ®isteredActor{uid: actorUID, ref: ref, volumes: volumes}
// Held until the initial write finishes so a refresh cannot interleave.
actor.mu.Lock()
defer actor.mu.Unlock()
@@
if prev != nil {
prev.mu.Lock()
prev.stale = true
prev.mu.Unlock()
}
for _, v := range volumes {
if err := r.write(ref, actorUID, v); err != nil {
- return fmt.Errorf("while populating system-info volume %q: %w", v.Name, err)
+ // actor.mu is already held here, so this cannot go through Undo.
+ r.drop(actor)
+ actor.stale = true
+ return nil, fmt.Errorf("while populating system-info volume %q: %w", v.Name, err)
}
}
- return nil
+ return ®istration{r: r, actor: actor}, nil
--- a/cmd/atelet/main.go
+++ b/cmd/atelet/main.go
@@ func (s *AteomHerder) Run(
- defer func() {
- if err != nil {
- s.systemInfoVolumes.Deregister(actorUID)
- }
- }()
- if err := s.systemInfoVolumes.Register(actorUID, actorRef, systemInfoVolumesFor(actorUID, req.GetSpec())); err != nil {
+ reg, err := s.systemInfoVolumes.Register(actorUID, actorRef, systemInfoVolumesFor(actorUID, req.GetSpec()))
+ if err != nil {
return nil, err
}
+ defer func() {
+ if err != nil {
+ reg.Undo()
+ }
+ }()
if err := s.prepareOCIBundles(ctx, actorUID, actorRef,
@@ func (s *AteomHerder) Restore(
- // Undo the Register if the restore fails.
+ // Undo the Register if the restore fails. The prep leg writes reg and
+ // nothing reads it before g.Wait returns.
+ var reg *registration
defer func() {
if err != nil {
- s.systemInfoVolumes.Deregister(actorUID)
+ reg.Undo()
}
}()
@@
- if err = s.systemInfoVolumes.Register(actorUID, actorRef, systemInfoVolumesFor(actorUID, req.GetSpec())); err != nil {
+ if reg, err = s.systemInfoVolumes.Register(actorUID, actorRef, systemInfoVolumesFor(actorUID, req.GetSpec())); err != nil {
Collaborator
There was a problem hiding this comment.
I'm happy to tackle this myself in a follow-up too if you'd prefer that. I don't think it necessarily needs to block this fix.
Collaborator
Author
There was a problem hiding this comment.
Thanks !
Shrutiya Mohan (shrutiyam-glitch)
force-pushed
the
systemvol_fix
branch
from
September 17, 2026 00:21
5fc16f8 to
ceaf7b3
Compare
Shrutiya Mohan (shrutiyam-glitch)
marked this pull request as ready for review
September 17, 2026 07:08
When a worker pod crashes or is deleted out-of-band, ateapi transitions the actor to CRASHED and clears worker_assignment without calling atelet.Terminate. If the actor is later recovered and scheduled onto a new worker pod on the same node, systemInfoVolumeRefresher.Register previously panicked on the leftover entry in r.actors. Supersede any existing registration for the same actor UID by marking the previous entry stale under its mutex, replacing it in r.actors, and logging the supersession.
Shrutiya Mohan (shrutiyam-glitch)
force-pushed
the
systemvol_fix
branch
from
September 17, 2026 16:44
ceaf7b3 to
a56b22d
Compare
2 tasks
Max Thompson (thompsonmax)
approved these changes
Sep 17, 2026
Taahir Ahmed (ahmedtd)
merged commit Sep 17, 2026
d7c2d05
into
agent-substrate:main
11 checks passed
Open
2 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When a worker pod crashes or is deleted out-of-band,
ateapitransitions the actor toCRASHEDand clearsworker_assignmentwithout callingatelet.Terminate. If the actor is later recovered and scheduled onto a new worker pod on the same node,systemInfoVolumeRefresher.Registerpanicked on the leftover entry in r.actors.Supersede any existing registration for the same actor UID by marking the previous entry stale under its mutex and replacing it in r.actors.
Temporary fix for #1710