Description

Containers running on Kubernetes that mount the /var/lib/kubelet/pods directory via hostPath are not able to be started if another container happens to mount a path via mountPath that contains double quotes.
The container with the hostPath transitions into CreateContainerError phase with following error:

  Warning  Failed     3m18s (x3 over 3m43s)  kubelet, kube-node01  (combined from similar events): Error: failed to generate container "a1fc3e89a4cc2ec280eeb31d7dcfc71b129df29d3b54dd04f082c048a1f38c8a" spec: parsing '/kubelet/pods/7b5b5252-053a-4918-88d1-bae7711eb2a4/volumes/kubernetes.io~configmap/content/"bar"' failed: unable to unquote root field: invalid syntax

The error comes from parsing the mountInfo: https://github.com/containerd/containerd/blob/v1.3.4/mount/mountinfo_linux.go#L90

Steps to reproduce the issue:

1. Deploy a Pod on a specific node containing a mountPath with double quotes:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: content
   data:
     test: |-
       foo
   ---
   apiVersion: v1
   kind: Pod
   metadata:
     name: busybox-quotemount
     labels:
       app: busybox
   spec:
     affinity:
       nodeAffinity:
         requiredDuringSchedulingIgnoredDuringExecution:
           nodeSelectorTerms:
           - matchExpressions:
             - key: kubernetes.io/hostname
               operator: In
               values:
               - kube-node01
     containers:
     - image: busybox
       command:
         - /bin/sh
         - -c
         - "tail -f /dev/null"
       imagePullPolicy: IfNotPresent
       name: busybox
       volumeMounts:
       - name: content
         mountPath: /tmp
       - name: content
         mountPath: '/tmp/"bar"'
         subPath: '"bar"'
     restartPolicy: Always
     volumes:
     - name: content
       configMap:
         name: content

2. Deploy a Pod that mounts the kubelet pods host path onto the same node:

   apiVersion: v1
   kind: Pod
   metadata:
     name: busybox-hostmount
     namespace: kube-system
     labels:
       app: busybox
   spec:
     affinity:
       nodeAffinity:
         requiredDuringSchedulingIgnoredDuringExecution:
           nodeSelectorTerms:
           - matchExpressions:
             - key: kubernetes.io/hostname
               operator: In
               values:
               - kube-node01
     containers:
     - image: busybox
       securityContext:
         privileged: true
       command:
         - /bin/sh
         - -c
         - "tail -f /dev/null"
       imagePullPolicy: IfNotPresent
       name: busybox
       volumeMounts:
       - name: pods-mount-dir
         mountPath: /var/lib/kubelet/pods
         mountPropagation: Bidirectional
     restartPolicy: Always
     volumes:
     - hostPath:
         path: /var/lib/kubelet/pods
         type: Directory
       name: pods-mount-dir

Describe the results you received:

The first pod gets created fine while the second cannot be started:

default       busybox-quotemount                                             1/1     Running                0          38s     10.6.8.57    kube-node01     <none>           <none>
kube-system   busybox-hostmount                                              0/1     CreateContainerError   0          30s     10.6.8.58    kube-node01     <none>           <none>

Describe the results you expected:

Actually not sure what to expect. The problem I see here is that Kubernetes establishes a mount even though the path contains a double quote but then containerd parsing fails if the path of the kubelet pods from the host is to be mounted into another container.

Output of containerd --version:

containerd github.com/containerd/containerd v1.3.4 814b7956fafc7a0980ea07e950f983d0837e5578

There was another issue opened in the cloud-provider-openstack repo because the cinder-csi-plugin was affected by this issue.

So at this point I'm not actually sure if this behavior is "intended" because it might be that using double quotes is actually not desired. On the other side Kubernetes does not perform a validation during this step and the mountpoint with double quotes is created and can be used. Also creating a directory on Linux directly and mounting it works fine:

mkdir '/tmp/"foo"'
mkdir /tmp/bar
sudo mount --bind /tmp/bar '/tmp/"foo"'

cat /proc/self/mountinfo | grep foo
1046 30 253:1 /tmp/bar /tmp/"foo" rw,relatime shared:1 - ext4 /dev/mapper/ubuntu--vg-root rw,errors=remount-ro

Do you think there should be a validation happen before actually creating the mount (e.g. validating the mountPath via Kubernetes) or should containerd be able to cope with double quotes?