Skip to content

Commit 953b965

Browse files
authored
fix(kubernetes): keep NFS publish from hanging on a dead server (#4791)
## What this PR does When the kubevirt-csi node plugin publishes an NFS volume, it first mounts the export root into an `nfs-init-*` temp dir to create `/data` and move old files into it. That temp mount was hard. If the NFS server stopped answering at that point, the publish call never returned. kubelet gives up after 2m and retries, each retry blocked the same way, and goroutines and temp dirs piled up until the server came back or the plugin restarted. The fix does three things: 1. The temp root is now mounted `soft` with `retry=0`. It only does metadata work that a retried publish redoes, so an error there is harmless, and a stuck call now ends on its own once the connection to the server is declared dead. The mount the pod gets stays hard, because a soft mount can report a failed write that the server did apply. 2. A publish to a target path that already has one running now returns `Aborted` instead of starting another. This is what stops the pile-up: at most one call per target path is stuck at a time. 3. Moving old files into `/data` now checks the destination with `lstat` and fails the publish on any error other than a missing file. With the soft mount, the check can fail with EIO on a file that `/data` already holds, and a rename issued after the server comes back would replace that file. `lstat` keeps a user's symlink in `/data` that loops or dangles from failing every publish. Such a symlink counts as present and is left alone, while before this change the rename replaced a dangling one. I kept the default retry timeouts on purpose, although the issue suggested lowering `timeo` and `retrans`. The Linux client shares one connection per server for all of that server's mounts on the node, and the first mount sets the connection's TCP timeouts. Lowering them on the temp mount would have changed the timeouts of the pods' hard mounts too. Over NFSv4, a soft RPC fails only after the connection is declared dead, so a stuck call takes minutes to end, longer than kubelet's 2m deadline. The guard keeps kubelet's retries from adding new calls during that time. The guard works per target path, not per volume. Two pods on one node that use the same RWX volume still publish in parallel, and each pod can have at most one stuck call. The unit tests pin the mount options on both mounts, the guard behaviour including its release after a failed publish, and the destination check: a failed check stops the move, and a symlink in `/data` stays in place. They use the fake mounter. I did not test against a real NFS server that went away. Not covered here, and tracked in #4814: - If the server's TCP connection stays up while the NFS service hangs, a soft NFSv4 RPC does not time out. The guard still limits it to one stuck call per target. - `Mount` from mount-utils runs `mount` with no deadline. This predates this change, and the guard bounds it the same way. - If the target is already hard-mounted from the dead server, the check of the target at the start of a publish blocks for good. The guard keeps this to one call per target but does not end it. Unpublish is not coordinated with a publish that is still running either. Both predate this change. - A follow-up comment on the issue reported two leftovers of the bounded unmount: an orphaned `umount.nfs` after a timeout, and a possible `stat` before `umount2`. Both may end sooner now that the temp mount is soft, but I have not verified either. Refs #4581 ### Screenshots Not a UI change. ### Downstream repositories The change is limited to the kubevirt-csi node plugin's Go code. None of the triggers in the contributing guide's map apply: no package added or renamed, no platform values, variants, release assets, ApplicationDefinition semantics or developer tooling touched. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(kubernetes): an NFS volume publish in tenant Kubernetes clusters no longer hangs forever when the NFS server goes away during the publish, and kubelet retries no longer pile up while a publish is stuck. The plugin's temporary mount of the export root is now soft, and a second publish to the same target path is refused while the first is still running. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Concurrent NFS publish requests for the same target now return an `Aborted` error. Different targets remain available, and a target can be used again after a request finishes, including after a failed attempt. * Migration now stops if checking a destination encounters an unexpected error. Existing destinations, including symlinks, are left in place. * Temporary-directory creation and mount errors now identify the affected volume. * **Updates** * Temporary NFS root mounts now use soft behavior with retries disabled. The final data mount remains hard. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents b865138 + f76dedf commit 953b965

2 files changed

Lines changed: 318 additions & 10 deletions

File tree

‎packages/apps/kubernetes/images/kubevirt-csi-driver/node.go‎

Lines changed: 49 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -4,10 +4,12 @@ import (
44
"context"
55
"errors"
66
"fmt"
7+
"io/fs"
78
"os"
89
"os/exec"
910
"path/filepath"
1011
"strings"
12+
"sync"
1113
"time"
1214

1315
csi "github.com/container-storage-interface/spec/lib/go/csi"
@@ -25,6 +27,16 @@ var _ csi.NodeServer = &WrappedNodeService{}
2527
type WrappedNodeService struct {
2628
*service.NodeService
2729
mounter mount.Interface
30+
// Target paths with an NFS publish running. kubelet retries a publish
31+
// that outlives its 2m deadline while the first call still runs, and on
32+
// a dead server every retry would block the same way: the soft temp mount
33+
// gives up only after minutes, and the stat of a target that is already
34+
// hard-mounted does not give up at all.
35+
publishing sync.Map
36+
// Replaced in tests, which cannot get the soft temp mount's EIO locally;
37+
// nil means the os function.
38+
lstat func(string) (os.FileInfo, error)
39+
mkdir func(string, os.FileMode) error
2840
}
2941

3042
// NodeStageVolume for NFS volumes is a no-op (NFS doesn't need staging).
@@ -53,7 +65,19 @@ func (w *WrappedNodeService) NodePublishVolume(ctx context.Context, req *csi.Nod
5365
return nil, status.Errorf(codes.Internal, "failed to parse NFS export: %v", err)
5466
}
5567

68+
lstat, mkdir := w.lstat, w.mkdir
69+
if lstat == nil {
70+
lstat = os.Lstat
71+
}
72+
if mkdir == nil {
73+
mkdir = os.Mkdir
74+
}
75+
5676
targetPath := req.GetTargetPath()
77+
if _, busy := w.publishing.LoadOrStore(targetPath, struct{}{}); busy {
78+
return nil, status.Errorf(codes.Aborted, "publish of NFS volume %s at %s already in progress", req.GetVolumeId(), targetPath)
79+
}
80+
defer w.publishing.Delete(targetPath)
5781

5882
// Check if already mounted
5983
notMnt, err := w.mounter.IsLikelyNotMountPoint(targetPath)
@@ -83,7 +107,7 @@ func (w *WrappedNodeService) NodePublishVolume(ctx context.Context, req *csi.Nod
83107
// user files that were written before this fix (backward compatibility).
84108
tmpMount, err := os.MkdirTemp("", fmt.Sprintf("nfs-init-%s-", req.GetVolumeId()))
85109
if err != nil {
86-
return nil, status.Errorf(codes.Internal, "failed to create temp mount dir: %v", err)
110+
return nil, status.Errorf(codes.Internal, "failed to create temp mount dir for volume %s: %v", req.GetVolumeId(), err)
87111
}
88112

89113
rootSource := fmt.Sprintf("%s:%s", host, path)
@@ -93,12 +117,21 @@ func (w *WrappedNodeService) NodePublishVolume(ctx context.Context, req *csi.Nod
93117
// The price is a cache of its own too: pods already holding /data may not
94118
// see the entries the migration moves in until their attribute cache
95119
// expires (acdirmax).
96-
rootOpts := []string{"nfsvers=4.2", fmt.Sprintf("port=%s", port), "nosharecache"}
120+
//
121+
// soft lets an RPC fail with EIO instead of blocking forever. That is
122+
// safe here because this mount only does metadata work that a retried
123+
// publish redoes, and it is why the volume mount below stays hard. Over
124+
// NFSv4 a soft RPC fails only once the TCP connection is declared dead,
125+
// which takes minutes with the default timeo and retrans. Those are not
126+
// lowered: the first mount of a server sets them for the connection that
127+
// every later mount of it on this node shares, the pods' hard mounts
128+
// included. retry=0 stops mount.nfs from retrying a failed mount for 2m.
129+
rootOpts := []string{"nfsvers=4.2", fmt.Sprintf("port=%s", port), "nosharecache", "soft", "retry=0"}
97130
if err := w.mounter.Mount(rootSource, tmpMount, "nfs", rootOpts); err != nil {
98131
if rmErr := os.Remove(tmpMount); rmErr != nil {
99132
klog.Warningf("Failed to remove temp dir %s: %v", tmpMount, rmErr)
100133
}
101-
return nil, status.Errorf(codes.Internal, "NFS temp mount failed: %v", err)
134+
return nil, status.Errorf(codes.Internal, "NFS temp mount for volume %s failed: %v", req.GetVolumeId(), err)
102135
}
103136
defer func() {
104137
if err := unmountTempMount(w.mounter, tmpMount); err != nil {
@@ -107,8 +140,10 @@ func (w *WrappedNodeService) NodePublishVolume(ctx context.Context, req *csi.Nod
107140
}()
108141

109142
dataDir := filepath.Join(tmpMount, "data")
110-
if err := os.MkdirAll(dataDir, 0777); err != nil {
111-
return nil, status.Errorf(codes.Internal, "failed to create /data subdir: %v", err)
143+
// Not MkdirAll: with the server gone its stat fails on the mount point,
144+
// and it then reports the local temp dir as existing instead of the EIO.
145+
if err := mkdir(dataDir, 0777); err != nil && !errors.Is(err, fs.ErrExist) {
146+
return nil, status.Errorf(codes.Internal, "failed to create /data subdir for volume %s: %v", req.GetVolumeId(), err)
112147
}
113148

114149
// Auto-migrate: move user files from root into /data (skip internal artifacts).
@@ -124,8 +159,14 @@ func (w *WrappedNodeService) NodePublishVolume(ctx context.Context, req *csi.Nod
124159
}
125160
src := filepath.Join(tmpMount, name)
126161
dst := filepath.Join(dataDir, name)
127-
if _, err := os.Stat(dst); err == nil {
128-
continue // already exists in /data, skip
162+
// Lstat: a user's symlink in /data counts as present whatever it
163+
// points at, and is not replaced.
164+
if _, err := lstat(dst); err == nil {
165+
klog.Warningf("Not migrating %s for volume %s: /data/%s already exists, so %s stays in the export root", name, req.GetVolumeId(), name, name)
166+
continue
167+
} else if !os.IsNotExist(err) {
168+
// The rename would replace whatever the failed stat could not see.
169+
return nil, status.Errorf(codes.Internal, "failed to check /data/%s for volume %s: %v", name, req.GetVolumeId(), err)
129170
}
130171
klog.Infof("Migrating %s to /data/%s for volume %s", name, name, req.GetVolumeId())
131172
if err := os.Rename(src, dst); err != nil {
@@ -148,9 +189,7 @@ func (w *WrappedNodeService) NodePublishVolume(ctx context.Context, req *csi.Nod
148189

149190
const (
150191
// Well inside the 2m kubelet puts on NodePublishVolume (csiTimeout in
151-
// kubernetes pkg/volume/csi/csi_plugin.go). Bounds the unmount only: the
152-
// temp mount is a hard mount, so the mount and the migration before it
153-
// still block for as long as the server is gone.
192+
// kubernetes pkg/volume/csi/csi_plugin.go).
154193
tempUnmountTimeout = 30 * time.Second
155194
// How long umount's output pipe may stay open once umount has exited or
156195
// been killed; the umount.nfs helper it forked can hold it.

0 commit comments

Comments
 (0)