Description

When NRI RunPodSandbox fails after StartSandbox has already created a container record in bolt DB, the deferred cleanup path does not delete the container. This can be triggered by NRI plugins rejecting the sandbox after it has been started, as well as CNI setup timeouts, ttrpc deadline errors, or other failures that occur after StartSandbox. If kubelet retries the same pod and succeeds, multiple containers with the same sandbox name but different IDs coexist in bolt DB. On containerd restart, recover() calls sandboxNameIndex.Reserve() for each container and hits a name conflict, causing a fatal exit that makes the node completely unusable.

Steps to reproduce the issue

1. On a node with an NRI plugin that rejects RunPodSandbox under certain conditions, create a pod that triggers the NRI rejection
2. Since StartSandbox has already created a container in bolt DB before the NRI hook, and the deferred cleanup does not delete it, a stale container remains in bolt DB
3. Retry creating the same pod — NRI now allows it — a new container with the same sandbox name but different ID is written to bolt DB
4. Restart containerd — recover() finds both containers and calls Reserve() with the same name for different IDs — fatal

Describe the results you received and expected

Received: containerd exits with fatal error:
level=fatal msg="Failed to run CRI service" error="failed to recover state: failed to reserve sandbox name "nginx-1_default_075f8559-d013-487d-8c31-31b8ba24ca99_0": name "nginx-1_default_075f8559-d013-487d-8c31-31b8ba24ca99_0" is reserved for "c315533e05821dd235f05b40515d4c8ec4f6455230d907557ec12b3394a54b1f""
After the fatal, ctr cannot connect to containerd. The node becomes completely unusable.
Expected: containerd should either:

• Clean up the container created by StartSandbox() when RunPodSandbox fails after it (root cause fix), OR
• Treat the name conflict as a non-fatal warning during recovery, allowing the node to remain operational (mitigation)

What version of containerd are you using?

containerd 2.2.1

Any other relevant information

k8s 1.36.0

Show configuration if it is related to CRI plugin.

version = 3
root = '/data0/containerd'
oom_score = -999

[metrics]
address = '127.0.0.1:1338'
default_runtime_name = "runc"
discard_unpacked_layers = true

[plugins]
[plugins.'io.containerd.cri.v1.images']
snapshotter = 'overlayfs'

[plugins.'io.containerd.cri.v1.runtime']
[plugins.'io.containerd.cri.v1.runtime'.containerd]
default_runtime_name = 'runc'
[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes]
[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc]
runtime_type = 'io.containerd.runc.v2'
[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc.options]
SystemdCgroup = true

    [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.nvidia]
      runtime_type = 'io.containerd.runc.v2'
      [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.nvidia.options]
        BinaryName = 'nvidia-container-runtime'
        SystemdCgroup = true

[plugins.'io.containerd.nri.v1.nri']
disable = true
socket_path = '/var/run/nri/nri.sock'
plugin_path = '/opt/nri/plugins'
plugin_config_path = '/etc/nri/conf.d'
plugin_registration_timeout = '10s'
plugin_request_timeout = '5s'
disable_connections = false

[plugins.'io.containerd.nri.v1.nri'.default_validator]
  enable = false
  reject_oci_hook_adjustment = false
  reject_runtime_default_seccomp_adjustment = false
  reject_unconfined_seccomp_adjustment = false
  reject_custom_seccomp_adjustment = false
  reject_namespace_adjustment = false
  reject_sysctl_adjustment = false
  required_plugins = ['node-resource-agent']
  tolerate_missing_plugins_annotation = 'nri.containerd.io/skip-validation'