Repository navigation
Description
The K3s CNI binaries are installed alongside the rest of the bundled userspace, and the managed containerd config is updated on restart to point at the current bin dir under /var/lib/rancher/k3s/data/XXX/bin. This makes it difficult to install custom CNI plugins, as the path used by containerd changes every time k3s is upgraded.
This was an obstacle to our packaging Multus with K3s:
This has been complained about on Users Slack:
The thing is that Cilium installs itself in /var/lib/rancher/k3s/data/[long_id]/bin and, following a k3s upgrade, the cni gets broken as the cluster can't find the cilium-cni binary anymore, and I need to restart the cilium daemonset in order for the cluster to work again. This is why I was looking at changing the cni binary location. Otherwise, I may need to use a clusterPolicy with something like kyverno to check for a kubernetes upgrade and then restart the pods accordingly, which isn't ideal.
Steps to validate issue:
- Install K3s:
curl -sL get.k3s.io | INSTALL_K3S_VERSION=v1.31.0+k3s1 sh -s - - Deploy multus with whereabouts using legacy "current" bin dir:
apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
name: multus
namespace: kube-system
spec:
repo: https://rke2-charts.rancher.io
chart: rke2-multus
targetNamespace: kube-system
valuesContent: |-
manifests:
configMap:
true
config:
fullnameOverride: multus
cni_conf:
confDir: /var/lib/rancher/k3s/agent/etc/cni/net.d
binDir: /var/lib/rancher/k3s/data/current/bin
kubeconfig: /var/lib/rancher/k3s/agent/etc/cni/net.d/multus.d/multus.kubeconfig
rke2-whereabouts:
fullnameOverride: whereabouts
enabled: true
cniConf:
confDir: /var/lib/rancher/k3s/agent/etc/cni/net.d
binDir: /var/lib/rancher/k3s/data/current/bin3: Validate that multus and whereabouts are installed on nodes:
root@systemd-node-1:~# /var/lib/rancher/k3s/data/current/bin/multus
meta-plugin that delegates to other CNI plugins
CNI protocol versions supported: 0.1.0, 0.2.0, 0.3.0, 0.3.1, 0.4.0, 1.0.0, 1.1.0
root@systemd-node-1:~# /var/lib/rancher/k3s/data/current/bin/whereabouts
whereabouts v0.8.0-8c381170 linux/amd64
CNI protocol versions supported: 0.1.0, 0.2.0, 0.3.0, 0.3.1, 0.4.0, 1.0.0, 1.1.04: Upgrade K3s: curl -sL get.k3s.io | INSTALL_K3S_VERSION=v1.31.1+k3s1 sh -s -
5: Note that CNI binaries are missing after upgrade - the "current" directory has changed:
root@systemd-node-1:~# /var/lib/rancher/k3s/data/current/bin/multus
bash: /var/lib/rancher/k3s/data/current/bin/multus: No such file or directory
root@systemd-node-1:~# /var/lib/rancher/k3s/data/current/bin/whereabouts
bash: /var/lib/rancher/k3s/data/current/bin/whereabouts: No such file or directorySteps to validate fix:
- Install K3s:
curl -sL get.k3s.io | INSTALL_K3S_COMMIT=xxx sh -s - - Deploy multus with whereabouts using new fixed bin dir:
apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
name: multus
namespace: kube-system
spec:
repo: https://rke2-charts.rancher.io
chart: rke2-multus
targetNamespace: kube-system
valuesContent: |-
manifests:
configMap:
true
config:
fullnameOverride: multus
cni_conf:
confDir: /var/lib/rancher/k3s/agent/etc/cni/net.d
binDir: /var/lib/rancher/k3s/data/cni
kubeconfig: /var/lib/rancher/k3s/agent/etc/cni/net.d/multus.d/multus.kubeconfig
rke2-whereabouts:
fullnameOverride: whereabouts
enabled: true
cniConf:
confDir: /var/lib/rancher/k3s/agent/etc/cni/net.d
binDir: /var/lib/rancher/k3s/data/cni3: Validate that multus and whereabouts are installed on nodes:
root@systemd-node-1:~# /var/lib/rancher/k3s/data/cni/multus
meta-plugin that delegates to other CNI plugins
CNI protocol versions supported: 0.1.0, 0.2.0, 0.3.0, 0.3.1, 0.4.0, 1.0.0, 1.1.0
root@systemd-node-1:~# /var/lib/rancher/k3s/data/cni/whereabouts
whereabouts v0.8.0-8c381170 linux/amd64
CNI protocol versions supported: 0.1.0, 0.2.0, 0.3.0, 0.3.1, 0.4.0, 1.0.0, 1.1.04: Upgrade K3s: curl -sL get.k3s.io | INSTALL_K3S_COMMIT=yyy sh -s -
5: Note that CNI binaries are still in place after the upgrade:
root@systemd-node-1:~# /var/lib/rancher/k3s/data/cni/multus
meta-plugin that delegates to other CNI plugins
CNI protocol versions supported: 0.1.0, 0.2.0, 0.3.0, 0.3.1, 0.4.0, 1.0.0, 1.1.0
root@systemd-node-1:~# /var/lib/rancher/k3s/data/cni/whereabouts
whereabouts v0.8.0-8c381170 linux/amd64
CNI protocol versions supported: 0.1.0, 0.2.0, 0.3.0, 0.3.1, 0.4.0, 1.0.0, 1.1.0Metadata
Metadata
Assignees
Labels
Type
Projects
- StatusShow more project fieldsDone Issue
Activity
brandond commentedon Oct 18, 2024
This has introduced a regression. K3s fails to start after upgrading from one commit (or release) with this change to another, due to the CNI symlinks failing to update.
log shows:
brandond commentedon Oct 19, 2024
To test use of multus+whereabouts, you can use the following manifest. This is for a host on a
172.17.0.0/24network and allows whereabouts to assign addresses from the top end of that range. Adjust it for the actual network your host is on, and it will assign the pod an IP on your LAN on eth1.Note that whereabouts MUST be passed the correct paths to the configuration and kubeconfig files, by default it is hardcoded to search for these under /etc/cni/net.d/.
aganesh-suse commentedon Oct 23, 2024
Closing based on #10928
fix(e2e): point host CNI paths at k3s dirs before Multus install
fix(cni): correct k3s CNI bin path and stale registration docs
fix(cni): correct k3s CNI bin path and stale registration docs
fix(cni): correct k3s CNI bin path and stale registration docs
cicd: migrate k3s multus scenarios to multus v4
fix(cni): correct k3s CNI bin path and stale registration docs