Skip to content

projects/kubernetes: kubelet container mounts not visible to dockerd container #2131

@btheado

Description

@btheado

Description

Commit 0b70e7e

When I start the kubernetes master from projects/kubernetes and then run kubeadm-init.sh, neither of the kube-proxy or weave-net pods come up. In both the pod's logs there are errors complaining about files missing in /var/run/secrets/kubernetes.io/serviceaccount.

It looks to me like the kubelet is creating those files in the kubelet container, but the dockerd container is not able to see those same files.

If I go back to an earlier commit (e0bcac3) from over a month ago, both kube-proxy and weave-net can come up fine and when I look in the dockerd container, it can see the same kubernetes secrets mounts as in the kublet container.

Steps to reproduce the issue:

  1. cd projects/kubernetes
  2. make build-vm-images
  3. ./boot.sh
  4. ctr exec --tty kubelet ash
  5. kubeadm-init.sh
  6. kubectl --namespace=kube-system get pods
  7. Repeat step 6 until the kube-proxy and weave-net pods to go to Error or CrashLoopBackOff state
  8. kubectl --namespace=kube-system logs kube-proxy-7wdtn ;# Replace with your actual value from step 6
  9. Look for a complaint in the output about /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: no such file or directory
  10. kubectl --namespace=kube-system logs weave-net-wqt0m weave ;# Replace with your actual value from step 6
  11. Look for a complaint in the output about /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
  12. df -h | grep secret and notice this kubelet container has two tmpfs mounts for each of weave-net and kube-proxy
  13. exit ;# to leave the kubelet container
  14. ctr exec docker df -h | grep secret and notice there is no output
  15. ctr exec docker docker ps -a and look for the container id for a kube-proxy instance
  16. ctr exec docker docker inspect <containerid from previous step> | grep secret and notice the source of the mount is the same path as the tmpfs from step 12. But step 13 shows those tmpfs mounts are missing from the dockerd container. This is why steps 9 and 11 are complaining about no such file or directory.

Describe the results you received:

Neither the weave-net nor the kube-proxy pods came up due to the missing secrets mount

Describe the results you expected:

The weave-net and the kube-proxy pods should have come up

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions