Skip to content

fix(atelet): Supersede stale system-info volume registrations in atelet - #1684

Merged
Taahir Ahmed (ahmedtd) merged 1 commit into
agent-substrate:mainfrom
shrutiyam-glitch:systemvol_fix
Sep 17, 2026
Merged

Taahir Ahmed (ahmedtd) merged 1 commit into
agent-substrate:mainfrom
shrutiyam-glitch:systemvol_fix

Conversation

@shrutiyam-glitch

@shrutiyam-glitch Shrutiya Mohan (shrutiyam-glitch) commented Sep 16, 2026 •

Copy link
Copy Markdown
Collaborator

When a worker pod crashes or is deleted out-of-band, ateapi transitions the actor to CRASHED and clears worker_assignment without calling atelet.Terminate. If the actor is later recovered and scheduled onto a new worker pod on the same node, systemInfoVolumeRefresher.Register panicked on the leftover entry in r.actors.

Supersede any existing registration for the same actor UID by marking the previous entry stale under its mutex and replacing it in r.actors.

Temporary fix for #1710

  • Tests pass
  • Appropriate changes to documentation are included in the PR

Comment thread cmd/atelet/systeminfovolume.go Resolved
r.actors[actorUID] = actor
r.mu.Unlock()
if dup {
panic(fmt.Sprintf("system-info volumes: actor %s registered twice", actorUID))

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a bit worried that with this change, a valid Deregister call (such as those in cmd/atelet/main.go:509 and :1119) would race with a second registration and cause the new registration to get erased.

I think the solution to this is to track the "registration" somehow after its created, so Deregister knows not to delete any second registration.

Example diff:

+// registration is the entry Register created. Undo removes that entry and
+// nothing else, so a caller that fails after registering cannot drop a
+// newer registration for the same actor.
+type registration struct {
+     r     *systemInfoVolumeRefresher
+     actor *registeredActor
+}
+
+// Undo drops the registration unless a later Register superseded it.
+// Safe on a nil handle so callers can defer it before Register runs.
+func (h *registration) Undo() {
+     if h == nil {
+             return
+     }
+     h.r.drop(h.actor)
+     h.actor.mu.Lock()
+     h.actor.stale = true
+     h.actor.mu.Unlock()
+}
+
+// drop removes actor from the registry if it is still the live entry.
+func (r *systemInfoVolumeRefresher) drop(actor *registeredActor) {
+     r.mu.Lock()
+     if r.actors[actor.uid] == actor {
+             delete(r.actors, actor.uid)
+     }
+     r.mu.Unlock()
+}
 // Register records actorUID's system-info volumes and writes their contents
-// from current cluster state. If actorUID is already registered (for example
-// after a worker pod crash left a stale entry without Terminate), the previous
-// registration is superseded.
-func (r *systemInfoVolumeRefresher) Register(actorUID string, ref resources.ActorRef, volumes []*systemInfoVolume) error {
+// from current cluster state. A second Register for the same actorUID
+// supersedes the first, which stops receiving refreshes. A failed Register
+// leaves no entry behind.
+func (r *systemInfoVolumeRefresher) Register(actorUID string, ref resources.ActorRef, volumes []*systemInfoVolume) (*registration, error) {
      actor := &registeredActor{uid: actorUID, ref: ref, volumes: volumes}
      // Held until the initial write finishes so a refresh cannot interleave.
      actor.mu.Lock()
      defer actor.mu.Unlock()
@@
      if prev != nil {
              prev.mu.Lock()
              prev.stale = true
              prev.mu.Unlock()
      }

      for _, v := range volumes {
              if err := r.write(ref, actorUID, v); err != nil {
-                     return fmt.Errorf("while populating system-info volume %q: %w", v.Name, err)
+                     // actor.mu is already held here, so this cannot go through Undo.
+                     r.drop(actor)
+                     actor.stale = true
+                     return nil, fmt.Errorf("while populating system-info volume %q: %w", v.Name, err)
              }
      }
-     return nil
+     return &registration{r: r, actor: actor}, nil
--- a/cmd/atelet/main.go
+++ b/cmd/atelet/main.go
@@ func (s *AteomHerder) Run(
-     defer func() {
-             if err != nil {
-                     s.systemInfoVolumes.Deregister(actorUID)
-             }
-     }()
-     if err := s.systemInfoVolumes.Register(actorUID, actorRef, systemInfoVolumesFor(actorUID, req.GetSpec())); err != nil {
+     reg, err := s.systemInfoVolumes.Register(actorUID, actorRef, systemInfoVolumesFor(actorUID, req.GetSpec()))
+     if err != nil {
              return nil, err
      }
+     defer func() {
+             if err != nil {
+                     reg.Undo()
+             }
+     }()
      if err := s.prepareOCIBundles(ctx, actorUID, actorRef,
@@ func (s *AteomHerder) Restore(
-     // Undo the Register if the restore fails.
+     // Undo the Register if the restore fails. The prep leg writes reg and
+     // nothing reads it before g.Wait returns.
+     var reg *registration
      defer func() {
              if err != nil {
-                     s.systemInfoVolumes.Deregister(actorUID)
+                     reg.Undo()
              }
      }()
@@
-             if err = s.systemInfoVolumes.Register(actorUID, actorRef, systemInfoVolumesFor(actorUID, req.GetSpec())); err != nil {
+             if reg, err = s.systemInfoVolumes.Register(actorUID, actorRef, systemInfoVolumesFor(actorUID, req.GetSpec())); err != nil {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm happy to tackle this myself in a follow-up too if you'd prefer that. I don't think it necessarily needs to block this fix.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks !

When a worker pod crashes or is deleted out-of-band, ateapi transitions the
actor to CRASHED and clears worker_assignment without calling atelet.Terminate.
If the actor is later recovered and scheduled onto a new worker pod on the same
node, systemInfoVolumeRefresher.Register previously panicked on the leftover
entry in r.actors.
Supersede any existing registration for the same actor UID by marking the
previous entry stale under its mutex, replacing it in r.actors, and logging
the supersession.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants