Repository navigation
[Backport release-1.7] fix(kubernetes): keep NFS publish from hanging on a dead server - #5017
Merged
Merged
Conversation
The kubevirt-csi node plugin mounts the NFS export root into a temp dir to create /data and migrate old files. That mount was hard, so with the server gone the mkdir, readdir and rename calls on it never returned. kubelet gives up after 2m and retries, and every retry blocked the same way, piling up goroutines and nfs-init-* dirs. Mount the temp root soft, so its RPCs fail once the connection to the server is declared dead instead of blocking forever; it only does metadata work that a retried publish redoes. retry=0 keeps mount.nfs from retrying a failed temp mount for 2m. timeo and retrans keep their defaults, because the first mount of a server sets them for the connection the pods' mounts of it share. The volume mount the pod gets stays hard, since a soft mount can report a failed write the server applied. Over NFSv4 that soft failure takes minutes, and a stat of a target that is already hard-mounted can block for good, so a publish for a target path that already has one running now returns Aborted instead of starting another. A soft stat can fail with EIO on a file /data already holds, and a rename issued once the server is back would replace it, so the migration now fails the publish on any stat error other than a missing destination. It checks the destination with lstat, so a user's symlink in /data that loops or dangles counts as present instead of failing every publish or being replaced by the rename. The skip it makes for a destination already in /data leaves the root entry where the pod cannot see it, so it is now logged as a warning. /data is created with a plain mkdir, since with the server gone MkdirAll fails its stat on the mount point and then reports the local temp dir as already existing instead of the EIO. The temp mount errors now name the volume, like the migration errors do. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]> (cherry picked from commit f76dedf)
1 of 12 tasks
github-actions
Bot
requested review from
IvanHunters,
Andrei Kvapil (kvaps),
Aleksei Sviridkin (lexfrei),
Timofei Larkin (lllamnyp),
Matthieu ROBIN (matthieu-robin),
myasnikovdaniil and
scooby87
as code owners
October 21, 2026 09:17
Contributor
|
Important Review skippedBot user detected. To trigger a single review, invoke the ⚙️ Run configuration
You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Backport of #4791 to
release-1.7.