runtime options seem to be ignored (again) with v2.0.1 #11169

mathias-ioki · 2024-12-16T14:29:04Z

Description

The behaviour looks exactly the same, as described here: #10249

TL;TR: We are using crun instead of runc and have therefore a dedicated section for it in our containerd config. With the latest release, this seems to be ignored again.

Error:

error="rpc error: code = Unknown desc = failed to create containerd task: failed to validate OCI runtime features: runtime info: failed to run [/opt/containerd/bin/containerd-shim-runc-v2 -info]: exit status 1 (stderr: \"io.containerd.runc.v2: failed to look up the path of \\\"runc\\\": exec: \\\"runc\\\": executable file not found in $PATH\")"

Steps to reproduce the issue

Put crun in a non-default path

Configure the path in the config:

[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc.options]
       BinaryName = '/opt/crun/crun'

Restart containerd

Describe the results you received and expected

Results I expect:

BinaryName is respected and crun is used from this path

Results I received:

BinaryName seems to be ignored and containers are not able to start with Error:

error="rpc error: code = Unknown desc = failed to create containerd task: failed to validate OCI runtime features: runtime info: failed to run [/opt/containerd/bin/containerd-shim-runc-v2 -info]: exit status 1 (stderr: \"io.containerd.runc.v2: failed to look up the path of \\\"runc\\\": exec: \\\"runc\\\": executable file not found in $PATH\")"

What version of containerd are you using?

v2.0.1

Any other relevant information

/opt/crun/crun --version

crun version 1.19
commit: db31c42ac46e20b5527f5339dcbf6f023fcd539c
rundir: /run/crun
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL

Show configuration if it is related to CRI plugin.

Related config section - was working fine with 2.0.0:

    [plugins.'io.containerd.cri.v1.runtime'.containerd]
      default_runtime_name = 'crun'
      ignore_blockio_not_enabled_errors = false
      ignore_rdt_not_enabled_errors = false

      [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes]
        [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc]
          runtime_type = 'io.containerd.runc.v2'
          runtime_path = ''
          pod_annotations = []
          container_annotations = []
          privileged_without_host_devices = false
          privileged_without_host_devices_all_devices_allowed = false
          base_runtime_spec = ''
          cni_conf_dir = ''
          cni_max_conf_num = 0
          snapshotter = ''
          sandboxer = 'podsandbox'
          io_type = ''

          [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc.options]
            BinaryName = ''
            CriuImagePath = ''
            CriuWorkPath = ''
            IoGid = 0
            IoUid = 0
            NoNewKeyring = false
            Root = ''
            ShimCgroup = ''
            SystemdCgroup = true

        [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.crun]
          runtime_type = 'io.containerd.runc.v2'
          runtime_path = ''
          pod_annotations = []
          container_annotations = []
          privileged_without_host_devices = false
          privileged_without_host_devices_all_devices_allowed = false
          base_runtime_spec = ''
          cni_conf_dir = ''
          cni_max_conf_num = 0
          snapshotter = ''
          sandboxer = 'podsandbox'
          io_type = ''

          [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.crun.options]
            BinaryName = '/opt/crun/crun'
            CriuImagePath = ''
            CriuWorkPath = ''
            IoGid = 0
            IoUid = 0
            NoNewKeyring = false
            Root = ''
            ShimCgroup = ''
            SystemdCgroup = true

The text was updated successfully, but these errors were encountered:

djdongjin · 2024-12-28T19:22:27Z

@mathias-ioki I tried your crun config, and it seems to work in my case (at least it can find the crun binary but failed on a crun error)

Curious how you encounter this, e.g., how/where did you specify the runtime/runtime-handler? Is it through a k8s/pod runtime-handler or something else?

containerd config: default generated on main + your crun cri runtime + option

$ cat pod-config.json
{
    "metadata": {
        "name": "nginx-crun-sandbox",
        "namespace": "default",
        "attempt": 1,
        "uid": "hdishd83djaidwnduwk28bcsb"
    },
    "log_directory": "/tmp",
    "linux": {
    }
}
$ sudo crictl runp --runtime=crun pod-config.json
E1228 19:10:24.081804  542569 remote_runtime.go:193] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to start sandbox \"375d3d696ea415b511f4c92894237526b85504e97a87a2155c94815f0f5ee05f\": failed to create containerd task: failed to create shim task: OCI runti
me create failed: sd-bus call: Invalid unit name or type.: Invalid argument"
FATA[0000] run pod sandbox: rpc error: code = Unknown desc = failed to start sandbox "375d3d696ea415b511f4c92894237526b85504e97a87a2155c94815f0f5ee05f": failed to create containerd task: failed to create shim task: OCI runtime create failed: sd-bus call: Invalid unit name or type.: Invalid argument

The sd-bus call: Invalid unit name or type.: Invalid argument error looks like generated by crun

And crun is not in my path:

$ which crun
crun not found
$ /opt/crun/crun --version
crun version 1.19
commit: db31c42ac46e20b5527f5339dcbf6f023fcd539c
rundir: /run/user/1000/crun
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL

mathias-ioki · 2025-01-06T09:24:37Z

Hey @djdongjin ,

I don't specify the runtime explicitly, as I have set it as default one in the containerd config:

[plugins.'io.containerd.cri.v1.runtime'.containerd]
      default_runtime_name = 'crun'

Looks like, this parameter isn't working then, if you can specify it manually. So with version 2.0.0 it should use the crun runtime, if you don't specify anything else, but not with 2.0.1. Can you confirm?

djdongjin · 2025-01-06T13:25:54Z

Hi @mathias-ioki I tried again without specifying the runtime explicitly. It still works (on both main branch and v.2.0.1.

# run sandbox still fails on the same crun error
$ sudo crictl runp pod-config.json
E0106 13:14:06.938129  462892 remote_runtime.go:193] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to start sandbox \"ecb1afa0a95d187a5ca14b0aaacfedeee63a6b09636cd74a048279cc210501d8\": failed to create containerd task: failed to create shim task: OCI runtime create failed: sd-bus call: Invalid unit name or type.: Invalid argument"
FATA[0000] run pod sandbox: rpc error: code = Unknown desc = failed to start sandbox "ecb1afa0a95d187a5ca14b0aaacfedeee63a6b09636cd74a048279cc210501d8": failed to create containerd task: failed to create shim task: OCI runtime create failed: sd-bus call: Invalid unit name or type.: Invalid argument
# the `crun` runtime is used, as well as the `BinaryName:/opt/crun/crun` option
$ sudo journalctl -u containerd | grep ecb1afa0a95d
...
Jan 06 13:14:06 ctd.us-central1-c.c.fair-app-446404-m2.internal containerd[462812]: time="2025-01-06T13:14:06.714955709Z" level=debug msg="use OCI runtime {Type:io.containerd.runc.v2 Path: PodAnnotations:[] ContainerAnnotations:[] Options:map[BinaryName:/opt/crun/crun CriuImagePath: CriuWorkPath: IoGid:0 IoUid:0 NoNewKeyring:false Root: ShimCgroup: SystemdCgroup:true] PrivilegedWithoutHostDevices:false PrivilegedWithoutHostDevicesAllDevicesAllowed:false BaseRuntimeSpec: NetworkPluginConfDir: NetworkPluginMaxConfNum:0 Snapshotter: Sandboxer:podsandbox IOType:}" podsandboxid=ecb1afa0a95d187a5ca14b0aaacfedeee63a6b09636cd74a048279cc210501d8

Attached is the full pod-config and containerd config. Can you try this on your end and see if the issue still persist?

{
    "metadata": {
        "name": "nginx-crun-sandbox-default-runtime2",
        "namespace": "default",
        "attempt": 1,
        "uid": "hdishd83djaidwnduwk28b"
    },
    "log_directory": "/tmp",
    "linux": {
    }
}

version = 3
root = '/var/lib/containerd'
state = '/run/containerd'
temp = ''
plugin_dir = ''
disabled_plugins = []
required_plugins = []
oom_score = 0
imports = []

[grpc]
  address = '/run/containerd/containerd.sock'
  tcp_address = ''
  tcp_tls_ca = ''
  tcp_tls_cert = ''
  tcp_tls_key = ''
  uid = 0
  gid = 0
  max_recv_message_size = 16777216
  max_send_message_size = 16777216

[ttrpc]
  address = ''
  uid = 0
  gid = 0

[debug]
  address = ''
  uid = 0
  gid = 0
  level = 'debug'
  format = ''

[metrics]
  address = ''
  grpc_histogram = false

[plugins]
  [plugins.'io.containerd.cri.v1.images']
    default_runtime_name = 'crun'
    snapshotter = 'overlayfs'
    disable_snapshot_annotations = true
    discard_unpacked_layers = false
    max_concurrent_downloads = 3
    image_pull_progress_timeout = '5m0s'
    image_pull_with_sync_fs = false
    stats_collect_period = 10

    [plugins.'io.containerd.cri.v1.images'.pinned_images]
      sandbox = 'registry.k8s.io/pause:3.10'

    [plugins.'io.containerd.cri.v1.images'.registry]
      config_path = ''

    [plugins.'io.containerd.cri.v1.images'.image_decryption]
      key_model = 'node'

  [plugins.'io.containerd.cri.v1.runtime']
    enable_selinux = false
    selinux_category_range = 1024
    max_container_log_line_size = 16384
    disable_apparmor = false
    restrict_oom_score_adj = false
    disable_proc_mount = false
    unset_seccomp_profile = ''
    tolerate_missing_hugetlb_controller = true
    disable_hugetlb_controller = true
    device_ownership_from_security_context = false
    ignore_image_defined_volumes = false
    netns_mounts_under_state_dir = false
    enable_unprivileged_ports = true
    enable_unprivileged_icmp = true
    enable_cdi = true
    cdi_spec_dirs = ['/etc/cdi', '/var/run/cdi']
    drain_exec_sync_io_timeout = '0s'
    ignore_deprecation_warnings = []

    [plugins.'io.containerd.cri.v1.runtime'.containerd]
      default_runtime_name = 'crun'
      ignore_blockio_not_enabled_errors = false
      ignore_rdt_not_enabled_errors = false

      [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes]
        [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc]
          runtime_type = 'io.containerd.runc.v2'
          runtime_path = ''
          pod_annotations = []
          container_annotations = []
          privileged_without_host_devices = false
          privileged_without_host_devices_all_devices_allowed = false
          base_runtime_spec = ''
          cni_conf_dir = ''
          cni_max_conf_num = 0
          snapshotter = ''
          sandboxer = 'podsandbox'
          io_type = ''

          [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc.options]
            BinaryName = ''
            CriuImagePath = ''
            CriuWorkPath = ''
            IoGid = 0
            IoUid = 0
            NoNewKeyring = false
            Root = ''
            ShimCgroup = ''

        [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.crun]
          runtime_type = 'io.containerd.runc.v2'
          runtime_path = ''
          pod_annotations = []
          container_annotations = []
          privileged_without_host_devices = false
          privileged_without_host_devices_all_devices_allowed = false
          base_runtime_spec = ''
          cni_conf_dir = ''
          cni_max_conf_num = 0
          snapshotter = ''
          sandboxer = 'podsandbox'
          io_type = ''

          [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.crun.options]
            BinaryName = '/opt/crun/crun'
            CriuImagePath = ''
            CriuWorkPath = ''
            IoGid = 0
            IoUid = 0
            NoNewKeyring = false
            Root = ''
            ShimCgroup = ''
            SystemdCgroup = true

    [plugins.'io.containerd.cri.v1.runtime'.cni]
      bin_dir = '/opt/cni/bin'
      conf_dir = '/etc/cni/net.d'
      max_conf_num = 1
      setup_serially = false
      conf_template = ''
      ip_pref = ''
      use_internal_loopback = false

  [plugins.'io.containerd.gc.v1.scheduler']
    pause_threshold = 0.02
    deletion_threshold = 0
    mutation_threshold = 100
    schedule_delay = '0s'
    startup_delay = '100ms'

  [plugins.'io.containerd.grpc.v1.cri']
    disable_tcp_service = true
    stream_server_address = '127.0.0.1'
    stream_server_port = '0'
    stream_idle_timeout = '4h0m0s'
    enable_tls_streaming = false

    [plugins.'io.containerd.grpc.v1.cri'.x509_key_pair_streaming]
      tls_cert_file = ''
      tls_key_file = ''

  [plugins.'io.containerd.image-verifier.v1.bindir']
    bin_dir = '/opt/containerd/image-verifier/bin'
    max_verifiers = 10
    per_verifier_timeout = '10s'

  [plugins.'io.containerd.internal.v1.opt']
    path = '/opt/containerd'

  [plugins.'io.containerd.internal.v1.tracing']

  [plugins.'io.containerd.metadata.v1.bolt']
    content_sharing_policy = 'shared'

  [plugins.'io.containerd.monitor.container.v1.restart']
    interval = '10s'

  [plugins.'io.containerd.monitor.task.v1.cgroups']
    no_prometheus = false

  [plugins.'io.containerd.nri.v1.nri']
    disable = false
    socket_path = '/var/run/nri/nri.sock'
    plugin_path = '/opt/nri/plugins'
    plugin_config_path = '/etc/nri/conf.d'
    plugin_registration_timeout = '5s'
    plugin_request_timeout = '2s'
    disable_connections = false

  [plugins.'io.containerd.runtime.v2.task']
    platforms = ['linux/amd64']

  [plugins.'io.containerd.service.v1.diff-service']
    default = ['walking']
    sync_fs = false

  [plugins.'io.containerd.service.v1.tasks-service']
    blockio_config_file = ''
    rdt_config_file = ''

  [plugins.'io.containerd.shim.v1.manager']
    env = []

  [plugins.'io.containerd.snapshotter.v1.blockfile']
    root_path = ''
    scratch_file = ''
    fs_type = ''
    mount_options = []
    recreate_scratch = false

  [plugins.'io.containerd.snapshotter.v1.btrfs']
    root_path = ''

  [plugins.'io.containerd.snapshotter.v1.devmapper']
    root_path = ''
    pool_name = ''
    base_image_size = ''
    async_remove = false
    discard_blocks = false
    fs_type = ''
    fs_options = ''

  [plugins.'io.containerd.snapshotter.v1.native']
    root_path = ''

  [plugins.'io.containerd.snapshotter.v1.overlayfs']
    root_path = ''
    upperdir_label = false
    sync_remove = false
    slow_chown = false
    mount_options = []

  [plugins.'io.containerd.snapshotter.v1.zfs']
    root_path = ''

  [plugins.'io.containerd.tracing.processor.v1.otlp']

  [plugins.'io.containerd.transfer.v1.local']
    max_concurrent_downloads = 3
    max_concurrent_uploaded_layers = 3
    config_path = ''

[cgroup]
  path = ''

[timeouts]
  'io.containerd.timeout.bolt.open' = '0s'
  'io.containerd.timeout.metrics.shimstats' = '2s'
  'io.containerd.timeout.shim.cleanup' = '5s'
  'io.containerd.timeout.shim.load' = '5s'
  'io.containerd.timeout.shim.shutdown' = '3s'
  'io.containerd.timeout.task.state' = '2s'

[stream_processors]
  [stream_processors.'io.containerd.ocicrypt.decoder.v1.tar']
    accepts = ['application/vnd.oci.image.layer.v1.tar+encrypted']
    returns = 'application/vnd.oci.image.layer.v1.tar'
    path = 'ctd-decoder'
    args = ['--decryption-keys-path', '/etc/containerd/ocicrypt/keys']
    env = ['OCICRYPT_KEYPROVIDER_CONFIG=/etc/containerd/ocicrypt/ocicrypt_keyprovider.conf']

  [stream_processors.'io.containerd.ocicrypt.decoder.v1.tar.gzip']
    accepts = ['application/vnd.oci.image.layer.v1.tar+gzip+encrypted']
    returns = 'application/vnd.oci.image.layer.v1.tar+gzip'
    path = 'ctd-decoder'
    args = ['--decryption-keys-path', '/etc/containerd/ocicrypt/keys']
    env = ['OCICRYPT_KEYPROVIDER_CONFIG=/etc/containerd/ocicrypt/ocicrypt_keyprovider.conf']

Start a pod by sudo crictl runp pod-config.json

mathias-ioki · 2025-01-09T14:00:57Z

Hey @djdongjin,

thank you for testing it again. After testing a bit more, the issue seems to be a bit more complicated. My findings so far:

creation of the pod isn't a problem - this works in all cases. That's the reason, why you wasn't able to re-produce it

another important factor is to enable user-namespaces for the container

without the user namespace feature, everything works quite well
as soon as user namespaces are enabled for the container, containerd isn't able to start the container

from the logs I can see, that containerd is checking the runtime features via

/opt/containerd/bin/containerd-shim-runc-v2 -info

so it seems to me, that this command is the root cause of the issue. It looks like, that this command isn't aware of the runtime config and therefore fails with an error

Jan 09 13:48:13 molecule-containerd containerd[11066]: time="2025-01-09T13:48:13.868633651Z" level=error msg="StartContainer for \"f1221b0885a24c6d3fde38bde51c6ba9dc516ec51224df23486697cb83c8b6ef\" failed" error="rpc error: code = Unknown desc = failed to create containerd task: failed to validate OCI runtime features: runtime info: failed to run [/opt/containerd/bin/containerd-shim-runc-v2 -info]: exit status 1 (stderr: \"io.containerd.runc.v2: failed to look up the path of \\\"runc\\\": exec: \\\"runc\\\": executable file not found in $PATH\")"

TL;TR: You have to enable user namespaces for your container, to make it fail. If you do that, the pod is created, but the container within the pod can't be started.

mathias-ioki · 2025-01-27T10:36:01Z

Hey folks,

it's still an issue with version 2.0.2. Is there anything I can do to help find a solution?

djdongjin · 2025-01-28T19:46:42Z

Hi @mathias-ioki , I haven't got time to repro with a user namespace case.

Do you have an existing pod-config.json/container-config.json that uses user namespace, and that I can use to repro with crictl?

Previously, PluginInfo was called with task options as the primary value, resulting in opts.BinaryName being omitted. Consequently, the containerd-shim-runc-v2 fell back to the system's runc binary in the PATH rather than the explicitly specified one. This change inverts the option fallback by preferring runtime options over task options, ensuring the correct binary is used for the PluginInfo request. Closes: containerd#11169 Signed-off-by: Jose Fernandez <[email protected]> Reviewed-by: Erikson Tung <[email protected]>

jfernandez · 2025-02-26T22:15:52Z

@mathias-ioki @djdongjin, we believe we encountered this issue when we enabled user namespaces. In particular, the BinaryName config was not picked up by the PluginInfo call. Here is our proposed fix: #11442

Previously, PluginInfo was called with task options as the primary value, resulting in opts.BinaryName being omitted. Consequently, the containerd-shim-runc-v2 fell back to the system's runc binary in the PATH rather than the explicitly specified one. This change inverts the option fallback by preferring runtime options over task options, ensuring the correct binary is used for the PluginInfo request. Closes: containerd#11169 Signed-off-by: Jose Fernandez <[email protected]> Reviewed-by: Erikson Tung <[email protected]>

mathias-ioki added the kind/bug label Dec 16, 2024

dosubot bot added the area/runtime Runtime label Dec 16, 2024

dmcgowan added this to Issue Management Jan 22, 2025

github-project-automation bot moved this to Todo in Issue Management Jan 22, 2025

jfernandez mentioned this issue Feb 26, 2025

Prefer runtime options for PluginInfo request #11442

Merged

AkihiroSuda closed this as completed in #11442 Feb 27, 2025

github-project-automation bot moved this from Todo to Done in Issue Management Feb 27, 2025

jfernandez mentioned this issue Mar 4, 2025

RuntimeOptions are not passed to the shim when user namespace is enabled #11480

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

runtime options seem to be ignored (again) with v2.0.1 #11169

runtime options seem to be ignored (again) with v2.0.1 #11169

mathias-ioki commented Dec 16, 2024

djdongjin commented Dec 28, 2024 •

edited

Loading

mathias-ioki commented Jan 6, 2025

djdongjin commented Jan 6, 2025 •

edited

Loading

mathias-ioki commented Jan 9, 2025

mathias-ioki commented Jan 27, 2025

djdongjin commented Jan 28, 2025

jfernandez commented Feb 26, 2025

runtime options seem to be ignored (again) with v2.0.1 #11169

runtime options seem to be ignored (again) with v2.0.1 #11169

Comments

mathias-ioki commented Dec 16, 2024

Description

Steps to reproduce the issue

Describe the results you received and expected

What version of containerd are you using?

Any other relevant information

Show configuration if it is related to CRI plugin.

djdongjin commented Dec 28, 2024 • edited Loading

mathias-ioki commented Jan 6, 2025

djdongjin commented Jan 6, 2025 • edited Loading

mathias-ioki commented Jan 9, 2025

mathias-ioki commented Jan 27, 2025

djdongjin commented Jan 28, 2025

jfernandez commented Feb 26, 2025

djdongjin commented Dec 28, 2024 •

edited

Loading

djdongjin commented Jan 6, 2025 •

edited

Loading