What happened?
After we updated two of our clusters from 1.22 to v1.24.4 we noticed a huge amount of logs (tens of entries per second) in kubelet:
kubelet_getters.go:300] "Path does not exist" path="/var/lib/kubelet/pods/86cba354-348c-4826-9837-df9c616a8862/volumes"
Neither does the volumes subdir exist, nor the corresponding subdir for the pod ("/var/lib/kubelet/pods/86cba354-348c-4826-9837-df9c616a8862 in this case).
Kubelet currently produces multiple gigabytes of logs each day since the upgrade.
What did you expect to happen?
No log spam (at least not with verbosity=0).
How can we reproduce it (as minimally and precisely as possible)?
Not really sure. We changed nothing in the setup. We just upgraded to v1.24.4. The cluster does not have any volume providers running. Just a 'vanilla' kubernetes (ubuntu 20.04, kubeadm, cillium, cri-o).
Anything else we need to know?
No response
Kubernetes version
Details
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.4", GitCommit:"95ee5ab382d64cfe6c28967f36b53970b8374491", GitTreeState:"clean", BuildDate:"2022-08-17T18:54:23Z", GoVersion:"go1.18.5", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.4
Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.4", GitCommit:"95ee5ab382d64cfe6c28967f36b53970b8374491", GitTreeState:"clean", BuildDate:"2022-08-17T18:47:37Z", GoVersion:"go1.18.5", Compiler:"gc", Platform:"linux/amd64"}Cloud provider
Details
bare metal
OS version
Details
$ cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.4 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.4 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
$ uname -a
Linux k8s-master-p-1 5.15.0-46-generic #49~20.04.1-Ubuntu SMP Thu Aug 4 19:15:44 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Install tools
Details
kubeadm
Container runtime (CRI) and version (if applicable)
Details
CRI-O
Version: 1.24.2
GitCommit: bd548b04f78a30e1e9d7c17162714edd50edd6ca
GitTreeState: clean
BuildDate: 2022-08-09T16:42:39Z
GoVersion: go1.18.2
Compiler: gc
Platform: linux/amd64
Linkmode: dynamic
BuildTags: apparmor, exclude_graphdriver_devicemapper, containers_image_ostree_stub, seccomp
SeccompEnabled: true
AppArmorEnabled: true
Related plugins (CNI, CSI, ...) and versions (if applicable)
Details
cilium v1.12.0
What happened?
After we updated two of our clusters from 1.22 to v1.24.4 we noticed a huge amount of logs (tens of entries per second) in kubelet:
Neither does the
volumessubdir exist, nor the corresponding subdir for the pod ("/var/lib/kubelet/pods/86cba354-348c-4826-9837-df9c616a8862in this case).Kubelet currently produces multiple gigabytes of logs each day since the upgrade.
What did you expect to happen?
No log spam (at least not with verbosity=0).
How can we reproduce it (as minimally and precisely as possible)?
Not really sure. We changed nothing in the setup. We just upgraded to v1.24.4. The cluster does not have any volume providers running. Just a 'vanilla' kubernetes (ubuntu 20.04, kubeadm, cillium, cri-o).
Anything else we need to know?
No response
Kubernetes version
Details
Cloud provider
Details
bare metalOS version
Details
Install tools
Details
kubeadmContainer runtime (CRI) and version (if applicable)
Details
CRI-OVersion: 1.24.2
GitCommit: bd548b04f78a30e1e9d7c17162714edd50edd6ca
GitTreeState: clean
BuildDate: 2022-08-09T16:42:39Z
GoVersion: go1.18.2
Compiler: gc
Platform: linux/amd64
Linkmode: dynamic
BuildTags: apparmor, exclude_graphdriver_devicemapper, containers_image_ostree_stub, seccomp
SeccompEnabled: true
AppArmorEnabled: true
Related plugins (CNI, CSI, ...) and versions (if applicable)
Details
cilium v1.12.0