Orphaned pods fail to get cleaned up #38498
Description
Kubernetes version
Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.6", GitCommit:"e569a27d02001e343cb68086bc06d47804f62af6", GitTreeState:"clean", BuildDate:"2016-11-12T05:16:27Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}
Environment:
- Cloud provider or hardware configuration:
AWS - OS (e.g. from /etc/os-release):
Ubuntu 16.04.1 LTS - Kernel (e.g.
uname -a):Linux 4.4.0-53-generic #74-Ubuntu SMP Fri Dec 2 15:59:10 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
What happened:
syslogs are getting spammed every 2 seconds with these kubelet errors:
Dec 9 13:14:02 ip-10-50-242-179 start-kubelet.sh[31129]: E1209 13:14:02.300355 31129 kubelet_volumes.go:159] Orphaned pod "ff614192-bcc4-11e6-a20e-0a591a8e83d7" found, but error open /var/lib/kubelet/pods/ff614192-bcc4-11e6-a20e-0a591a8e83d7/volumes: no such file or directory occured during reading volume dir from disk
Dec 9 13:14:02 ip-10-50-242-179 start-kubelet.sh[31129]: E1209 13:14:02.300373 31129 kubelet_getters.go:249] Could not read directory /var/lib/kubelet/pods/ff769116-bcf4-11e6-a20e-0a591a8e83d7/volumes: open /var/lib/kubelet/pods/ff769116-bcf4-11e6-a20e-0a591a8e83d7/volumes: no such file or directory
We get the above 2 log entries for all the non-running pods (2150) every 2 seconds.
So our logs get into the Gb pretty quickly
There are 2160 pods in /var/lib/kubelet/pods/
~# ls /var/lib/kubelet/pods/ | wc -l
2160
But only 10 are running and attached to volumes
~# df -h | grep kubelet
/dev/xvdf 256G 232M 256G 1% /var/lib/kubelet
tmpfs 7.4G 8.0K 7.4G 1% /var/lib/kubelet/pods/5b884f1f-bbcd-11e6-a20e-0a591a8e83d7/volumes/kubernetes.io~secret/secrets
tmpfs 7.4G 12K 7.4G 1% /var/lib/kubelet/pods/5b884f1f-bbcd-11e6-a20e-0a591a8e83d7/volumes/kubernetes.io~secret/default-token-lfu24
tmpfs 7.4G 12K 7.4G 1% /var/lib/kubelet/pods/15302286-bbaa-11e6-a20e-0a591a8e83d7/volumes/kubernetes.io~secret/default-token-m0h9s
tmpfs 7.4G 12K 7.4G 1% /var/lib/kubelet/pods/b0395433-a546-11e6-9670-0a591a8e83d7/volumes/kubernetes.io~secret/default-token-n79fe
tmpfs 7.4G 12K 7.4G 1% /var/lib/kubelet/pods/1198c11a-bd25-11e6-a20e-0a591a8e83d7/volumes/kubernetes.io~secret/default-token-np531
tmpfs 7.4G 12K 7.4G 1% /var/lib/kubelet/pods/473d7d51-bd25-11e6-a20e-0a591a8e83d7/volumes/kubernetes.io~secret/default-token-smuz3
tmpfs 7.4G 12K 7.4G 1% /var/lib/kubelet/pods/e17b1a95-bd36-11e6-a20e-0a591a8e83d7/volumes/kubernetes.io~secret/default-token-1xs9g
tmpfs 7.4G 12K 7.4G 1% /var/lib/kubelet/pods/2a36441b-bd57-11e6-a20e-0a591a8e83d7/volumes/kubernetes.io~secret/default-token-qbw68
tmpfs 7.4G 12K 7.4G 1% /var/lib/kubelet/pods/cf6c04f4-bd64-11e6-a20e-0a591a8e83d7/volumes/kubernetes.io~secret/default-token-n79fe
tmpfs 7.4G 8.0K 7.4G 1% /var/lib/kubelet/pods/24130c15-bdf5-11e6-98c0-0615e1fbbfc7/volumes/kubernetes.io~secret/secrets
tmpfs 7.4G 12K 7.4G 1% /var/lib/kubelet/pods/24130c15-bdf5-11e6-98c0-0615e1fbbfc7/volumes/kubernetes.io~secret/default-token-9ksrm
tmpfs 7.4G 12K 7.4G 1% /var/lib/kubelet/pods/a271290c-bdf6-11e6-98c0-0615e1fbbfc7/volumes/kubernetes.io~secret/default-token-n79fe