Skip to content

Image volumes (KEP-4639) silently mount empty: ImageVolume gate is on but bundled containerd lacks CRI support #14323

Closed

Description

@lexfrei

Environment

  • k3s: v1.36.1+k3s1
  • Bundled containerd: v2.2.3-k3s1
  • Node OS: Ubuntu 24.04

What happens

A Pod using a native image: volume (KEP-4639 / OCI VolumeSource) is admitted and runs, but the volume mounts empty instead of exposing the referenced image's filesystem. No error is surfaced — the container just sees an empty directory.

Reproducer

apiVersion: v1
kind: Pod
metadata:
  name: imgvol-test
spec:
  restartPolicy: Never
  containers:
  - name: c
    image: busybox:1.36
    command: ["sh","-c","ls -la /img; ls /img/bin"]
    volumeMounts:
    - name: testimg
      mountPath: /img
  volumes:
  - name: testimg
    image:
      reference: busybox:1.36
      pullPolicy: IfNotPresent

Observed:

total 8
drwxrwxrwx 2 root root 4096 ... .
drwxr-xr-x 1 root root 4096 ... ..
ls: /img/bin: No such file or directory

Expected: /img contains busybox's root filesystem (/bin, /etc, …).

Root cause

ImageVolume is Beta and enabled by default since Kubernetes 1.33, so the apiserver and kubelet accept image: volumes. But the bundled containerd does not advertise image-volume support in its CRI RuntimeFeatures. From k3s crictl info, the only features reported are:

features: { supplemental_groups_policy: true }
runtimeHandlers[].features: { recursive_read_only_mounts: true, user_namespaces: true }

Image-volume CRI support landed in containerd as alpha in v2.1 with beta tracked separately (containerd/containerd#11853). The v2.2.3-k3s1 build here does not appear to implement/advertise it, so the result is a silent empty mount instead of a clear error.

Impact

Any workload relying on image volumes silently gets an empty directory. Concrete downstream: KubeVirt v1.9's containerDisk path delivers its container-disk binary via an image volume, so on k3s every containerDisk-backed VM fails to start with exec: "/container-disk-binary/usr/bin/container-disk": no such file or directory (kubevirt#17460).

Ask

Either build/enable image-volume CRI support in the bundled containerd so the Beta feature works, or — if it is intentionally unsupported — surface an error instead of mounting empty (fail loudly), and/or document the gap.

Activity

  1. brandond commented on Jul 3, 2026

    @brandond
    Member

    build/enable image-volume CRI support in the bundled containerd

    This isn't an optional feature that we have omitted and can simply switch on in our build. Our bundled versions should support all the same features as the corresponding upstream version.

    Note that k3s v1.36.2 includes containerd v2.3.2, which I believe does support image volumes and subpaths.

    if it is intentionally unsupported — surface an error instead of mounting empty (fail loudly), and/or document the gap.

    This is not a bug in k3s or a feature request that we can accommodate. This is just how Kubernetes project maintainers decided to handle runtimes without image volume support. If you think Kubernetes should raise an error instead of just not mounting anything there, please open an issue with that project.

  2. moved this from To Triage to Done Issue in K3s Developmenton Jul 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions