Skip to content

1.6.x inside a Docker container fails to run containers (cannot enter cgroupv2 "/sys/fs/cgroup/default" with domain controllers -- it is in an invalid state) #6659

@tianon

Description

@tianon

Description

On 1.5.x (with either of runc 1.0.3 or 1.1.0) I can successfully run a Docker container with --privileged and /var/lib/containerd on a volume, run containerd inside, and run containers on that containerd instance without issue.

On 1.6.x (again, with either recent runc version) I get the following error when I try to run a container:

ctr: failed to create shim task: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:385: applying cgroup configuration for process caused: cannot enter cgroupv2 "/sys/fs/cgroup/default" with domain controllers -- it is in an invalid state: unknown

Steps to reproduce the issue

  1. build an image with the version of containerd + runc desired
  2. docker run --name test --privileged --volume /var/lib/containerd your-docker-image containerd
    (feel free to use the tianon/containerd:1.6.1 or tianon/containerd:1.5.10 images if you do not want to build your own)
  3. docker exec -it test bash -c 'ctr image pull docker.io/library/bash:latest && ctr run -t --rm docker.io/library/bash:latest test'

Describe the results you received and expected

Expected:

$ docker exec -it test bash -c 'ctr image pull docker.io/library/bash:latest && ctr run -t --rm docker.io/library/bash:latest test
docker.io/library/bash:latest:                                                    resolved       |++++++++++++++++++++++++++++++++++++++| 
index-sha256:fc742d0c3d9d8f5fb2681062398c04b710cd08c46dac1a8f0a5515687018acb9:    done           |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:20f58dc05f356808d12635065b568d37966bb68f472c53edaa98ba8e16430d26: done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:832a0baf56be374fb9d9151f3ef6a82c5754f69117f2c58fd235c11c801725cb:    done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:53e520fda1825ccb2b42bb9b8d5d3d9842f11063ff519347da83ca928b18bedd:   done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:59bf1c3509f33515622619af21ed55bbe26d24913cedbca106468a5fb37a50c3:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:c3141377f232ef4d065034b3332237dc2ca236ac742f83b93aa2d5f703b5e3bd:    done           |++++++++++++++++++++++++++++++++++++++| 
elapsed: 2.2 s                                                                    total:  2.7 Mi (1.2 MiB/s)                                       
unpacking linux/amd64 sha256:fc742d0c3d9d8f5fb2681062398c04b710cd08c46dac1a8f0a5515687018acb9...
done: 215.141188ms	
bash-5.1# 

Recevied:

$ docker exec -it test bash -c 'ctr image pull docker.io/library/bash:latest && ctr run -t --rm docker.io/library/bash:latest test
docker.io/library/bash:latest:                                                    resolved       |++++++++++++++++++++++++++++++++++++++| 
index-sha256:fc742d0c3d9d8f5fb2681062398c04b710cd08c46dac1a8f0a5515687018acb9:    done           |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:20f58dc05f356808d12635065b568d37966bb68f472c53edaa98ba8e16430d26: done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:832a0baf56be374fb9d9151f3ef6a82c5754f69117f2c58fd235c11c801725cb:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:59bf1c3509f33515622619af21ed55bbe26d24913cedbca106468a5fb37a50c3:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:c3141377f232ef4d065034b3332237dc2ca236ac742f83b93aa2d5f703b5e3bd:    done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:53e520fda1825ccb2b42bb9b8d5d3d9842f11063ff519347da83ca928b18bedd:   done           |++++++++++++++++++++++++++++++++++++++| 
elapsed: 2.2 s                                                                    total:  4.5 Mi (2.0 MiB/s)                                       
unpacking linux/amd64 sha256:fc742d0c3d9d8f5fb2681062398c04b710cd08c46dac1a8f0a5515687018acb9...
done: 176.456151ms	
ctr: failed to create shim task: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:385: applying cgroup configuration for process caused: cannot enter cgroupv2 "/sys/fs/cgroup/default" with domain controllers -- it is in an invalid state: unknown

What version of containerd are you using?

containerd github.com/containerd/containerd v1.6.1 10f428d

Any other relevant information

I originally thought this was somehow related to opencontainers/runc#3387, but I'm a lot less sure of that now.

Show configuration if it is related to CRI plugin.

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions