Skip to content

Commit 52aa837

Browse files
authored
fix(kubernetes): force the kubevirt-csi temp NFS unmount after a timeout (#4584)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does The kubevirt-csi node plugin now unmounts its temporary NFS mount with a timeout, and retries with `umount -f` when the timeout runs out. For an NFS publish the plugin mounts the export root into an `nfs-init-*` temp dir to create `/data`. It unmounted that dir in a defer with a plain `umount`, which has no deadline. Before this change, if the NFS server was gone by then, the publish could hang on that `umount` forever and the temp mount stayed. The force-unmount helper in `k8s.io/mount-utils` doesn't work for this case. `UnmountWithForce` retries with `-f` only after its first `umount` returns. That `umount` forks the `umount.nfs` helper. When the timeout kills `umount`, the helper stays stuck on the dead server and keeps the output pipe open, and `CombinedOutput` waits for that pipe, so the `-f` retry never runs. The plugin now runs `umount` itself with `WaitDelay` set, so the pipe is closed a second after the kill, and then runs `umount -f` under its own timeout. A "not mounted" answer counts as done, same as in mount-utils. `umount2` detaches the mount before it blocks on tearing down the superblock, so the `-f` may find nothing left. Some umounts can't be killed at all, for example one stuck in uninterruptible sleep. So the whole cleanup also waits at most 30s, well inside the 2m deadline kubelet sets for NodePublishVolume (`csiTimeout` in `pkg/volume/csi/csi_plugin.go`). Once the budget runs out the publish goes on, and the unmount keeps running in the background and logs its result when it ends. The temp dir is removed by that background unmount once it succeeds, so an unmount that finishes late still cleans up after itself. A mounter other than the exec-based one keeps its plain `Unmount`, under the same budget. The temp mount now uses `nosharecache`. A forced unmount runs `nfs_umount_begin`, which kills every RPC in flight on the superblock, and NFS shares a superblock between mounts of the same export with the same options. Without this option, `-f` on the temp mount could fail I/O on the pod mount this same publish creates, and on other pods on the node that mount the same volume. The cost is a separate cache: pods that already have `/data` mounted may not see what the migration moves in until their attribute cache expires. The migration is a one-time backfill, so I'm fine with that. The leak only lives until the container restarts, because `/tmp` in the node plugin container has no hostPath and the mounts go away with the container's mount namespace. So this change stops leaked mounts from piling up inside one container. For the same reason a sweep of `nfs-init-*` dirs at startup would never find anything, so I did not add one. Two cases stay as they were. An `umount` that fails fast, for example with `target is busy`, gets no force retry, is logged, and the dir stays because `rmdir` refuses a mount point. And since the temp mount is a hard mount, a server that disappears before the unmount still hangs the mount and migration calls earlier in this path. That second case is #4581. The tests run the real exec-based mounter against a fake `umount` on `PATH`. The fake covers a plain unmount that succeeds, one that fails fast and gets no `-f`, and one that hangs while an orphaned child holds its output pipe. The last one has to end in `umount -f` inside the budget. A "not mounted" from `-f` counts as done, and any other `-f` failure is reported and keeps the dir. Fake mounters cover the background path: giving up on a stalled unmount, removing the dir after a late unmount, and logging the late result. Other tests check `nosharecache` on the temp mount and the temp dir after a failed mount. I haven't reproduced this against a real NFS server that went away, so it is not checked whether `umount -f` in the almalinux image gets as far as `umount2(MNT_FORCE)` there. The tests are in a new file, so this PR does not overlap with #4562. Closes #4561 ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> No UI change. ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note fix(kubernetes): the kubevirt-csi node plugin no longer hangs an NFS volume publish forever when unmounting its temporary mount after the NFS server went away; the unmount now times out and is retried with force ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Temporary NFS mounts use isolated caching to reduce interference with pod mounts. * Cleanup waits up to 30 seconds for unmounting. If the limit is reached, the publish operation returns an error while unmounting continues in the background. * Temporary directories are removed after a successful unmount or a failed mount, and retained if unmounting fails. * When a standard unmount times out, cleanup retries with a forced unmount when available. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents c35cc84 + 92bf90c commit 52aa837

2 files changed

Lines changed: 445 additions & 9 deletions

File tree

‎packages/apps/kubernetes/images/kubevirt-csi-driver/node.go‎

Lines changed: 107 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -2,9 +2,13 @@ package main
22

33
import (
44
"context"
5+
"errors"
56
"fmt"
67
"os"
8+
"os/exec"
79
"path/filepath"
10+
"strings"
11+
"time"
812

913
csi "github.com/container-storage-interface/spec/lib/go/csi"
1014
"google.golang.org/grpc/codes"
@@ -81,21 +85,23 @@ func (w *WrappedNodeService) NodePublishVolume(ctx context.Context, req *csi.Nod
8185
if err != nil {
8286
return nil, status.Errorf(codes.Internal, "failed to create temp mount dir: %v", err)
8387
}
84-
// Logged, not returned: a leftover temp dir does not make the volume
85-
// unusable, and failing an otherwise good publish only makes kubelet retry.
86-
defer func() {
87-
if err := os.Remove(tmpMount); err != nil {
88-
klog.Warningf("Failed to remove temp dir %s: %v", tmpMount, err)
89-
}
90-
}()
9188

9289
rootSource := fmt.Sprintf("%s:%s", host, path)
93-
rootOpts := []string{"nfsvers=4.2", fmt.Sprintf("port=%s", port)}
90+
// nosharecache gives the temp mount a superblock of its own: the forced
91+
// unmount below kills every RPC in flight on its superblock, which would
92+
// otherwise be the one the pods on this node use for the same export.
93+
// The price is a cache of its own too: pods already holding /data may not
94+
// see the entries the migration moves in until their attribute cache
95+
// expires (acdirmax).
96+
rootOpts := []string{"nfsvers=4.2", fmt.Sprintf("port=%s", port), "nosharecache"}
9497
if err := w.mounter.Mount(rootSource, tmpMount, "nfs", rootOpts); err != nil {
98+
if rmErr := os.Remove(tmpMount); rmErr != nil {
99+
klog.Warningf("Failed to remove temp dir %s: %v", tmpMount, rmErr)
100+
}
95101
return nil, status.Errorf(codes.Internal, "NFS temp mount failed: %v", err)
96102
}
97103
defer func() {
98-
if err := w.mounter.Unmount(tmpMount); err != nil {
104+
if err := unmountTempMount(w.mounter, tmpMount); err != nil {
99105
klog.Warningf("Failed to unmount temp dir %s: %v", tmpMount, err)
100106
}
101107
}()
@@ -140,6 +146,98 @@ func (w *WrappedNodeService) NodePublishVolume(ctx context.Context, req *csi.Nod
140146
return &csi.NodePublishVolumeResponse{}, nil
141147
}
142148

149+
const (
150+
// Well inside the 2m kubelet puts on NodePublishVolume (csiTimeout in
151+
// kubernetes pkg/volume/csi/csi_plugin.go). Bounds the unmount only: the
152+
// temp mount is a hard mount, so the mount and the migration before it
153+
// still block for as long as the server is gone.
154+
tempUnmountTimeout = 30 * time.Second
155+
// How long umount's output pipe may stay open once umount has exited or
156+
// been killed; the umount.nfs helper it forked can hold it.
157+
umountPipeGrace = time.Second
158+
)
159+
160+
// unmountTempMount bounds the unmount: a plain umount of an NFS mount whose
161+
// server is gone blocks forever, which hangs the publish and leaks the mount.
162+
func unmountTempMount(m mount.Interface, dir string) error {
163+
return unmountWithin(m, dir, tempUnmountTimeout)
164+
}
165+
166+
// unmountWithin gives the whole unmount one deadline. On timeout the unmount
167+
// keeps running in the background and logs its result when it ends; an umount
168+
// stuck in uninterruptible sleep cannot be killed, so no deadline on the
169+
// process itself can be relied on.
170+
//
171+
// The dir is removed by the worker once the unmount succeeds, so an unmount
172+
// that finishes after the deadline still cleans up; removed by the caller it
173+
// would still be a mount point at that moment. A failed unmount keeps the dir.
174+
// Plain Remove rather than RemoveAll: if the unmount did not take, the dir's
175+
// contents are the export's.
176+
func unmountWithin(m mount.Interface, dir string, budget time.Duration) error {
177+
done := make(chan error)
178+
abandoned := make(chan struct{})
179+
go func() {
180+
var err error
181+
if _, ok := m.(*mount.Mounter); ok {
182+
err = umountForcedAfter(dir, budget/3)
183+
} else {
184+
err = m.Unmount(dir)
185+
}
186+
if err == nil {
187+
if rmErr := os.Remove(dir); rmErr != nil && !os.IsNotExist(rmErr) {
188+
klog.Warningf("Failed to remove temp dir %s: %v", dir, rmErr)
189+
}
190+
}
191+
select {
192+
case done <- err:
193+
case <-abandoned:
194+
if err != nil {
195+
klog.Warningf("unmount of %s failed after the deadline: %v", dir, err)
196+
} else {
197+
klog.Infof("unmount of %s finished after the deadline", dir)
198+
}
199+
}
200+
}()
201+
select {
202+
case err := <-done:
203+
return err
204+
case <-time.After(budget):
205+
close(abandoned)
206+
return fmt.Errorf("unmount of %s still running after %v", dir, budget)
207+
}
208+
}
209+
210+
// umountForcedAfter replaces mount-utils' UnmountWithForce, which retries with
211+
// -f only once its first umount has returned. That umount forks umount.nfs;
212+
// killed on the deadline, it leaves the helper stuck on the vanished server
213+
// holding the output pipe, and CombinedOutput waits for that pipe, so the -f
214+
// never runs. WaitDelay closes the pipe instead.
215+
func umountForcedAfter(dir string, timeout time.Duration) error {
216+
err := runUmount(timeout, dir)
217+
if !errors.Is(err, context.DeadlineExceeded) {
218+
return err
219+
}
220+
klog.V(2).Infof("Timed out waiting for unmount of %s, trying with -f", dir)
221+
return runUmount(timeout, "-f", dir)
222+
}
223+
224+
func runUmount(timeout time.Duration, args ...string) error {
225+
ctx, cancel := context.WithTimeout(context.Background(), timeout)
226+
defer cancel()
227+
cmd := exec.CommandContext(ctx, "umount", args...)
228+
cmd.WaitDelay = umountPipeGrace
229+
out, err := cmd.CombinedOutput()
230+
if ctx.Err() != nil {
231+
return fmt.Errorf("umount %s: %w", strings.Join(args, " "), ctx.Err())
232+
}
233+
// umount2 detaches the mount before it blocks tearing the superblock down,
234+
// so the -f after a killed umount can find nothing left to unmount.
235+
if err != nil && !strings.Contains(string(out), "not mounted") {
236+
return fmt.Errorf("umount %s: %v: %s", strings.Join(args, " "), err, out)
237+
}
238+
return nil
239+
}
240+
143241
// NodeExpandVolume for NFS volumes is a no-op (LINSTOR handles NFS resize automatically).
144242
// This should not normally be called for NFS since ControllerExpandVolume returns
145243
// NodeExpansionRequired=false, but we handle it gracefully as a safety net.

0 commit comments

Comments
 (0)