Skip to content

Commit 7739d1e

Browse files
committed
feat(multus)!: stage reference CNI plugins from the multus image
Multus-backed NetworkAttachmentDefinitions delegate to the upstream containernetworking reference plugins (bridge, macvlan, ipvlan, ...), and nothing guarantees a node has them in the directory Multus searches. On k3s they live under /var/lib/rancher/k3s/data/cni while Multus and Cilium install into /opt/cni/bin, so a bridge-type NAD fails there with `failed to find plugin "bridge" in path [/opt/cni/bin]`. Bundle 14 of the release's plugins into the multus image and add an install-cni-plugins init container that installs them into the host /opt/cni/bin, mirroring the existing install-multus-binary container. loopback, dhcp, tap and dummy are left out; the Dockerfile records why. Gate it on a new chart value, stageCniPlugins, resolved per bundle and on for both. Talos bundles only bridge, firewall, flannel, host-local, loopback and portmap, so the other ten are missing there exactly as on generic Linux; the four that overlap are rewritten with the same upstream version, v1.9.1 on both sides, rather than an older one. A marker in the image Dockerfile records the Talos release that pin was compared against, and a test holds it to the release the platform ships. multus therefore moves out of system.common-packages into the two variant branches, the way linstor already does for talos.enabled. networking.stageCniPlugins overrides the variant default in either direction. Where staging is on the install is unconditional: a plugin is replaced on every pod recreation, not only when the destination is missing. A destination that is already a directory is reported and left alone, since replacing it is not something this can do safely. Turning the value off afterwards does not undo it; nothing here removes or restores a plugin. The container is not privileged and takes no mount propagation. It runs in the spc_t SELinux domain, without which an enforcing node refuses the write. Each plugin is installed under a temporary name and swapped in with a rename: /opt/cni/bin is shared with cilium and kube-ovn. The plugin tarball is checksummed before extraction. A plugin that cannot be installed is reported on stderr and to /dev/termination-log, and the container still exits 0: Multus stays the node's primary CNI and nothing removes its config, so an init container that never completes would leave the shim with no daemon behind it. The manifest change lives in customize-deployment.patch so it survives make update. BREAKING CHANGE: enabling stageCniPlugins replaces reference CNI plugins in /opt/cni/bin on existing nodes, Talos included -- four of the names it writes are ones the Talos node image also carries. Opt out before upgrading: declining afterwards does not restore what was replaced. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
1 parent ee26a50 commit 7739d1e

17 files changed

Lines changed: 1995 additions & 20 deletions

‎docs/agents/image-refs.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -32,9 +32,9 @@ A reference can be stored in three different kinds of file, and **all three are
3232

3333
1. **`packages/<group>/<pkg>/values.yaml`** — the common case, in five YAML sub-shapes (single string, split map with `digest`, split map with the digest inside `tag`, chart-global `global.registry.address` + `global.images[]`, and the OCI-artifact `platformSourceUrl`/`platformSourceRef` pair). The sub-shapes exist because vendored upstream charts choose their own values layout; the list is empirical, not closed.
3434
2. **`packages/<group>/<pkg>/images/<name>.tag`** — a plain file holding one reference, read by templates via `.Files.Get` rather than through values. Easy to forget precisely because it never appears in a values file; otherwise an ordinary first-party reference that carries the cozystack version like any other.
35-
3. **A declared file that embeds a reference in surrounding text** — used when a package vendors an upstream manifest verbatim and its `image:` target `sed`s the built reference straight into it. `packages/system/multus/templates/multus-daemonset-thick.yml` is the only instance today.
35+
3. **A declared file that embeds a reference in surrounding text** — used when a package vendors an upstream manifest and its `image:` target `sed`s the built reference straight into it. `packages/system/multus/templates/multus-daemonset-thick.yml` is the only instance today. It is vendored *and* patched: it carries Helm conditionals, so `yq` cannot parse it at all and the YAML pass yields nothing for it. What keeps its reference visible is the textual scrape `collect_image_refs` runs over every declared extra — for this entry that leg is not a backstop but the only one that works.
3636

37-
`hack/lib/image-refs.sh` is the single enumeration of all three, and the only place that knowledge is allowed to live. `image_ref_files <root>` yields the files; `collect_image_refs <root>` yields the references inside them. Shape 3 is an explicit list (`IMAGE_REF_EXTRA_FILES`) rather than a `templates/**` glob, because such a glob would sweep in vendored upstream chart templates and Helm-templated `image:` lines where a blind rewrite corrupts a value that `make update` regenerates.
37+
`hack/lib/image-refs.sh` is the single enumeration of all three, and the only place that knowledge is allowed to live. `image_ref_files <root>` yields the files; `collect_image_refs <root>` yields the references inside them. Shape 3 is an explicit list (`IMAGE_REF_EXTRA_FILES`) rather than a `templates/**` glob, because such a glob would sweep in vendored upstream chart templates and Helm-templated `image:` lines where a blind rewrite corrupts a value that `make update` regenerates. The declared entry is safe from that despite now containing Helm conditionals: its `image:` lines are plain, and declaring a file is the deliberate statement that its references are stamped rather than templated.
3838

3939
### Adding a reference in a new location
4040

‎docs/vm-external-vlan.md‎

Lines changed: 32 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,9 +23,40 @@ Three pieces cooperate:
2323
## Prerequisites
2424

2525
- The `multus` package is enabled (it provides the `NetworkAttachmentDefinition` CRD and the secondary-network plumbing).
26-
- The `bridge` CNI plugin is present in `/opt/cni/bin` on every node. This binary comes from the node's CNI-plugins bundle, not from a Cozystack chart: on Talos it ships with the node image, while k3s keeps its copy under `/var/lib/rancher/k3s/data/current/bin` and leaves `/opt/cni/bin` (where Multus and Cilium install) without it. Verify with `ls /opt/cni/bin/bridge`; a missing binary makes the NAD fail with `failed to find plugin "bridge" in path [/opt/cni/bin]`. If it is absent, stage the upstream reference plugins (`bridge`, `macvlan`, `ipvlan`, …) into `/opt/cni/bin` as part of node provisioning.
26+
- The `bridge` CNI plugin is present in `/opt/cni/bin` on every node. The `multus` package puts it there itself on every platform, which is the subject of the next section — read it before upgrading a cluster whose `/opt/cni/bin` you provision yourself. Verify with `ls /opt/cni/bin/bridge`; a missing binary makes the NAD fail with `failed to find plugin "bridge" in path [/opt/cni/bin]`.
2727
- There is no IPAM plugin in this path — addresses are assigned inside the guest, not by the CNI. Plan static addresses per VM.
2828

29+
## Staging the reference CNI plugins
30+
31+
The `multus` package installs the reference plugins into `/opt/cni/bin` itself, on every platform, because no platform supplies all of them. That is the `stageCniPlugins` chart value, and the platform turns it on for every bundle. What differs is only how much is missing before staging runs. Talos puts six binaries in that directory — `bridge`, `firewall`, `host-local`, `loopback` and `portmap` from the same upstream release, plus `flannel`, which is a different project — so a NAD naming `macvlan`, `ipvlan` or `vlan` fails there too; staging adds the ten it does not carry and rewrites the four that overlap with the same upstream version (`v1.9.1` on both sides), not an older one. Generic Linux offers no guarantee at all — k3s keeps its own copies under `/var/lib/rancher/k3s/data/cni` (`/var/lib/rancher/k3s/data/current/bin` on releases older than October 2024) and Cilium installs only `cilium-cni` and `loopback`, so `/opt/cni/bin` there can lack even `bridge`.
32+
33+
Carrying the plugins costs roughly 67 MiB uncompressed per architecture in the `multus` image, which every node pulls whether staging is on or off. Size airgap mirrors accordingly.
34+
35+
Where staging is on, it tries to replace any file of the same name already in that directory, every time the Multus pod is recreated, so a hand-placed or locally patched `bridge` will normally not survive — carry such a change in the `multus` image instead. A plugin it fails to write is reported and skipped rather than removed, so whatever was there stays — an older copy where the node had one, and nothing at all where it did not. It installs 14 of the upstream reference plugins and leaves `loopback`, `dhcp`, `dummy` and `tap` alone. Leaving them alone is not a promise that something else provides them: `loopback` is installed by Cilium and kube-ovn, while `dhcp`, `dummy` and `tap` are simply not staged — `dhcp` needs a host daemon this package does not run, and the other two have no consumer here. If you need one of those, put it on the node yourself.
36+
37+
**If you provision `/opt/cni/bin` yourself, set the opt-out before upgrading.** The replacement happens as soon as Multus rolls, and turning `stageCniPlugins` off afterwards does not undo it: nothing in this package removes or restores a plugin, so the copies stay until you re-provision the node. Operators pinned to a particular plugin version should decline first and upgrade second. This applies on Talos too, where four of the names staging writes are ones the node image also carries. Declining looks like this, on the `platform` component of the `cozystack.cozystack-platform` Package CR:
38+
39+
```yaml
40+
apiVersion: cozystack.io/v1alpha1
41+
kind: Package
42+
metadata:
43+
name: cozystack.cozystack-platform
44+
spec:
45+
components:
46+
platform:
47+
values:
48+
networking:
49+
stageCniPlugins: false
50+
```
51+
52+
Editing the multus HelmRelease or the child `Package cozystack.multus` directly does not hold: both are re-rendered by the platform reconcile. An unreadable value there — a spelling that is neither a `true` nor a `false` form — stops the whole platform render rather than resolving to off, so a typo in this key halts reconciliation cluster-wide.
53+
54+
If `bridge` is missing on a node where staging is on, read the log with `kubectl --namespace cozy-multus logs <multus-pod> --container install-cni-plugins`. The line `no /cni-plugins in this image; skipping staging` means the running `multus` image predates the plugin staging and installed nothing: the image reference and the manifest are re-pinned on different schedules, so a tree taken between a Dockerfile change and the next release bake can carry one without the other. Move to a release whose `multus` image carries the plugins. A line beginning `failed to install:` names plugins the container could not write, one of which may be the `bridge` you are looking for. The reason for each is earlier in the same log, printed as that plugin was attempted rather than next to the summary — so with more than one failure the causes are spread above it, not adjacent to it, and the one naming your plugin is the one to read. The container reports these and exits successfully on purpose, so the pod shows Ready and `kubectl get` will not point you here: failing it instead would leave the node unable to start any pod that goes through CNI, since Multus remains the primary CNI with no daemon behind it — host-network pods bypass CNI and still start. That means a NAD failing with `failed to find plugin` is the symptom to trust, and this log is where the reason is.
55+
56+
If the init container is absent from the Multus pod altogether, staging is off for this cluster. That means `networking.stageCniPlugins` was set to `false` somewhere: no bundle defaults it off, on Talos or anywhere else. Turn it back on with the same block as above and `true` in place of `false`, or stage the plugins through node provisioning instead.
57+
58+
Changing this value edits the Multus DaemonSet's pod template, so Multus rolls node by node. While a node's `multus-daemon` is down its CNI shim has no daemon behind it, and pod creation and deletion on that node fail for the length of the restart. Flip it during a window where that is acceptable, not while something else is rolling.
59+
2960
## Step 1 — Linux bridge on the node
3061

3162
Create a bridge that enslaves the tagged VLAN sub-interface. The VLAN sub-interface itself carries no address; the bridge carries the host's presence on that VLAN (optional, but useful for a gateway-reachability sanity path and for any host-side service the VMs must reach).

0 commit comments

Comments
 (0)