Repository navigation
Commit 953b965
authored
fix(kubernetes): keep NFS publish from hanging on a dead server (#4791)
## What this PR does
When the kubevirt-csi node plugin publishes an NFS volume, it first
mounts the export root into an `nfs-init-*` temp dir to create `/data`
and move old files into it. That temp mount was hard. If the NFS server
stopped answering at that point, the publish call never returned.
kubelet gives up after 2m and retries, each retry blocked the same way,
and goroutines and temp dirs piled up until the server came back or the
plugin restarted.
The fix does three things:
1. The temp root is now mounted `soft` with `retry=0`. It only does
metadata work that a retried publish redoes, so an error there is
harmless, and a stuck call now ends on its own once the connection to
the server is declared dead. The mount the pod gets stays hard, because
a soft mount can report a failed write that the server did apply.
2. A publish to a target path that already has one running now returns
`Aborted` instead of starting another. This is what stops the pile-up:
at most one call per target path is stuck at a time.
3. Moving old files into `/data` now checks the destination with `lstat`
and fails the publish on any error other than a missing file. With the
soft mount, the check can fail with EIO on a file that `/data` already
holds, and a rename issued after the server comes back would replace
that file. `lstat` keeps a user's symlink in `/data` that loops or
dangles from failing every publish. Such a symlink counts as present and
is left alone, while before this change the rename replaced a dangling
one.
I kept the default retry timeouts on purpose, although the issue
suggested lowering `timeo` and `retrans`. The Linux client shares one
connection per server for all of that server's mounts on the node, and
the first mount sets the connection's TCP timeouts. Lowering them on the
temp mount would have changed the timeouts of the pods' hard mounts too.
Over NFSv4, a soft RPC fails only after the connection is declared dead,
so a stuck call takes minutes to end, longer than kubelet's 2m deadline.
The guard keeps kubelet's retries from adding new calls during that
time.
The guard works per target path, not per volume. Two pods on one node
that use the same RWX volume still publish in parallel, and each pod can
have at most one stuck call.
The unit tests pin the mount options on both mounts, the guard behaviour
including its release after a failed publish, and the destination check:
a failed check stops the move, and a symlink in `/data` stays in place.
They use the fake mounter. I did not test against a real NFS server that
went away.
Not covered here, and tracked in #4814:
- If the server's TCP connection stays up while the NFS service hangs, a
soft NFSv4 RPC does not time out. The guard still limits it to one stuck
call per target.
- `Mount` from mount-utils runs `mount` with no deadline. This predates
this change, and the guard bounds it the same way.
- If the target is already hard-mounted from the dead server, the check
of the target at the start of a publish blocks for good. The guard keeps
this to one call per target but does not end it. Unpublish is not
coordinated with a publish that is still running either. Both predate
this change.
- A follow-up comment on the issue reported two leftovers of the bounded
unmount: an orphaned `umount.nfs` after a timeout, and a possible `stat`
before `umount2`. Both may end sooner now that the temp mount is soft,
but I have not verified either.
Refs #4581
### Screenshots
Not a UI change.
### Downstream repositories
The change is limited to the kubevirt-csi node plugin's Go code. None of
the triggers in the contributing guide's map apply: no package added or
renamed, no platform values, variants, release assets,
ApplicationDefinition semantics or developer tooling touched.
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
### Release note
```release-note
fix(kubernetes): an NFS volume publish in tenant Kubernetes clusters no longer hangs forever when the NFS server goes away during the publish, and kubelet retries no longer pile up while a publish is stuck. The plugin's temporary mount of the export root is now soft, and a second publish to the same target path is refused while the first is still running.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Concurrent NFS publish requests for the same target now return an
`Aborted` error. Different targets remain available, and a target can be
used again after a request finishes, including after a failed attempt.
* Migration now stops if checking a destination encounters an unexpected
error. Existing destinations, including symlinks, are left in place.
* Temporary-directory creation and mount errors now identify the
affected volume.
* **Updates**
* Temporary NFS root mounts now use soft behavior with retries disabled.
The final data mount remains hard.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->2 files changed
Lines changed: 318 additions & 10 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
4 | 4 | | |
5 | 5 | | |
6 | 6 | | |
| 7 | + | |
7 | 8 | | |
8 | 9 | | |
9 | 10 | | |
10 | 11 | | |
| 12 | + | |
11 | 13 | | |
12 | 14 | | |
13 | 15 | | |
| |||
25 | 27 | | |
26 | 28 | | |
27 | 29 | | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
28 | 40 | | |
29 | 41 | | |
30 | 42 | | |
| |||
53 | 65 | | |
54 | 66 | | |
55 | 67 | | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
56 | 76 | | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
57 | 81 | | |
58 | 82 | | |
59 | 83 | | |
| |||
83 | 107 | | |
84 | 108 | | |
85 | 109 | | |
86 | | - | |
| 110 | + | |
87 | 111 | | |
88 | 112 | | |
89 | 113 | | |
| |||
93 | 117 | | |
94 | 118 | | |
95 | 119 | | |
96 | | - | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
97 | 130 | | |
98 | 131 | | |
99 | 132 | | |
100 | 133 | | |
101 | | - | |
| 134 | + | |
102 | 135 | | |
103 | 136 | | |
104 | 137 | | |
| |||
107 | 140 | | |
108 | 141 | | |
109 | 142 | | |
110 | | - | |
111 | | - | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
112 | 147 | | |
113 | 148 | | |
114 | 149 | | |
| |||
124 | 159 | | |
125 | 160 | | |
126 | 161 | | |
127 | | - | |
128 | | - | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
129 | 170 | | |
130 | 171 | | |
131 | 172 | | |
| |||
148 | 189 | | |
149 | 190 | | |
150 | 191 | | |
151 | | - | |
152 | | - | |
153 | | - | |
| 192 | + | |
154 | 193 | | |
155 | 194 | | |
156 | 195 | | |
| |||
0 commit comments