Repository navigation
fix atelet deviceplugin wedge - #3323
Merged
Tim Hockin (thockin) merged 2 commits intoOct 29, 2026
Merged
Tim Hockin (thockin) merged 2 commits into
Tim Hockin (thockin) merged 2 commits into
Conversation
the replacement pod starts as soon as this one is terminating and registers the same socket while we drain. kubelet 1.37 can then wedge the resource as "already connected" until kubelet restarts. Signed-off-by: Benjamin Elder <[email protected]>
re-registering right after registering can wedge the resource in kubelet the same way as two overlapping atelet pods Signed-off-by: Benjamin Elder <[email protected]>
Benjamin Elder (BenTheElder)
changed the title
Tim Hockin (thockin)
added
the
prio/P0
Benjamin Elder (BenTheElder)
added
kind/bug
Tim Hockin (thockin)
approved these changes
Oct 29, 2026
Tim Hockin (thockin)
assigned
Tim Hockin (thockin) and unassigned
Eitan Yarmush (EItanya)
Merged
via the queue into
agent-substrate:main
with commit Oct 29, 2026
40bf20f
8 of 9 checks passed
It may be best to not reuse the socket and instead use random socket name. See kubernetes/kubernetes#139338 |
Collaborator
Author
|
Thanks, that makes sense. I'll switch atelet to a new socket per registration in a follow-up. |
2 tasks done
Collaborator
Author
|
follow-up: #3486 IIUC we can afford to defer it to 1.0.x, what we have now will work but random socket will be preferable. [I vaguely remembered a thread about this but couldn't find it, Sergey gave us the pointer and advice, thank you!] |
Eitan Yarmush (EItanya)
pushed a commit
to kagent-dev/substrate
that referenced
this pull request
Nov 3, 2026
…ubstrate#3486) Follow-up to agent-substrate#3323, as @SergeyKanzhelev suggested there. kubelet expects a device plugin to switch to a new socket when it re-registers. Since 1.37 it rejects a Register for a socket it still has connected, and two overlapping Registers for one socket can wedge the resource as "already connected" until kubelet restarts: kubernetes/kubernetes#139338 - the device plugin listens on a new `ate.dev-kvm-<random>.sock` for each registration, and removes sockets left over from earlier atelets - the device plugin stays up until atelet exits again. agent-substrate#3323 stopped it on SIGTERM to avoid overlapping with the replacement pod, but the DaemonSet only starts the replacement once the old pod is gone, so that just left `ate.dev/kvm` unhealthy for the whole drain <!-- e.g. "AI-assisted, I reviewed and tested it." No AI co-author trailers. --> AI use: I used Claude to draft this and reviewed it. <!-- Breaking change? Describe what users must do on upgrade and ask for the `breaking-change` label. --> - [x] Tests pass - [x] Docs updated --------- Signed-off-by: Benjamin Elder <[email protected]> (cherry picked from commit 38e9ef6) Signed-off-by: Eitan Yarmush <[email protected]>
Yufan Su (yufan-su)
pushed a commit
to yufan-su/substrate
that referenced
this pull request
Nov 3, 2026
…ubstrate#3486) Follow-up to agent-substrate#3323, as @SergeyKanzhelev suggested there. kubelet expects a device plugin to switch to a new socket when it re-registers. Since 1.37 it rejects a Register for a socket it still has connected, and two overlapping Registers for one socket can wedge the resource as "already connected" until kubelet restarts: kubernetes/kubernetes#139338 - the device plugin listens on a new `ate.dev-kvm-<random>.sock` for each registration, and removes sockets left over from earlier atelets - the device plugin stays up until atelet exits again. agent-substrate#3323 stopped it on SIGTERM to avoid overlapping with the replacement pod, but the DaemonSet only starts the replacement once the old pod is gone, so that just left `ate.dev/kvm` unhealthy for the whole drain <!-- e.g. "AI-assisted, I reviewed and tested it." No AI co-author trailers. --> AI use: I used Claude to draft this and reviewed it. <!-- Breaking change? Describe what users must do on upgrade and ask for the `breaking-change` label. --> - [x] Tests pass - [x] Docs updated --------- Signed-off-by: Benjamin Elder <[email protected]>
Dmitry Berkovich (dberkov)
pushed a commit
that referenced
this pull request
Nov 3, 2026
Follow-up to #3323, as @SergeyKanzhelev suggested there. kubelet expects a device plugin to switch to a new socket when it re-registers. Since 1.37 it rejects a Register for a socket it still has connected, and two overlapping Registers for one socket can wedge the resource as "already connected" until kubelet restarts: kubernetes/kubernetes#139338 - the device plugin listens on a new `ate.dev-kvm-<random>.sock` for each registration, and removes sockets left over from earlier atelets - the device plugin stays up until atelet exits again. #3323 stopped it on SIGTERM to avoid overlapping with the replacement pod, but the DaemonSet only starts the replacement once the old pod is gone, so that just left `ate.dev/kvm` unhealthy for the whole drain <!-- e.g. "AI-assisted, I reviewed and tested it." No AI co-author trailers. --> AI use: I used Claude to draft this and reviewed it. <!-- Breaking change? Describe what users must do on upgrade and ask for the `breaking-change` label. --> - [x] Tests pass - [x] Docs updated --------- Signed-off-by: Benjamin Elder <[email protected]> (cherry picked from commit 38e9ef6) Signed-off-by: Eitan Yarmush <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why is this needed? (Or link the issue. Open one first for features, API, install or CI changes.)
Fixes #3272
Since 1.37, kubelet wedges a device plugin resource if a second Register for the same socket arrives while it is still closing the first stream. The failed connect deregisters both clients by socket path:
https://github.com/kubernetes/kubernetes/blob/v1.37.0/pkg/kubelet/cm/devicemanager/plugin/v1beta1/handler.go#L82-L86
So when the first stream ends, kubelet never calls PluginDisconnected, and every later Register fails with "device plugin already connected" until kubelet restarts:
https://github.com/kubernetes/kubernetes/blob/v1.37.0/pkg/kubelet/cm/devicemanager/plugin/v1beta1/handler.go#L137-L140
atelet keeps one fsnotify watcher across registration retries, so a kubelet.sock Create that was queued while it retried made it stop and re-register right after registering. It now ignores the Create if kubelet.sock is still the socket it registered with.
A node that is already wedged still needs a kubelet restart.
AI use: I used Claude to trace this through the kubelet source and draft the fix, and reviewed it.