Repository navigation
Bug: K3s OCI runtime create failed: runc create failed: unable to start container process: unable to apply cgroup configuration: #591
Description
Describe the bug
When using the K3s container on my ARM Mac with Colima I am unable to run deployments in the K3s cluster. The K3s cluster from Colima itself works fine.
To Reproduce
k3s = K3SContainer()
k3s.start()
env = os.environ.copy()
with NamedTemporaryFile(mode="w", delete=False) as tmp_file:
tmp_file.write(k3s.config_yaml())
tmp_file.close()
env["KUBECONFIG"] = tmp_file.name
subprocess.check_call(
["kubectl", "run", "busybox", "--image=busybox"],
env=env,
)this will start the cluster and insert a pod. But the pod will not start and instead see a logs like
Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: unable to apply cgroup configuration: failed to write 4188: write /sys/fs/cgroup/kubepods/besteffort/pod86fac936-f8f2-4cfa-8a62-77c892d5ef3d/ba75980054e795ee5b552902df784e5be731cee71fa20aacb51d4459d5f49173/cgroup.procs: no such file or directory: unknown
the logs from the K3s container look similarly for services like coreDNS.
Runtime environment
Darwin MABP-JRP4R0W3XJ 23.3.0 Darwin Kernel Version 23.3.0: Wed Dec 20 21:31:00 PST 2023; root:xnu-10002.81.5~7/RELEASE_ARM64_T6020 arm64
Python 3.11.6
docker info:
Server:
Containers: 26
Running: 2
Paused: 0
Stopped: 24
Images: 21
Server Version: 24.0.7
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: d8f198a4ed8892c764191ef7b3b06d8a2eeb5c7f
runc version: v1.1.10-0-g18a0cb0
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: builtin
cgroupns
Kernel Version: 6.5.0-35-generic
Operating System: Ubuntu 23.10
OSType: linux
Architecture: aarch64
CPUs: 4
Total Memory: 3.816GiB
Name: colima
ID: 264a16ae-9445-4626-99ba-d62c497cf7e5
Docker Root Dir: /var/lib/docker
Debug Mode: false
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Activity
tharwan commentedon May 29, 2024
Quick update. Same results on the linux machine (without Colima) from a colleague.
tharwan commentedon May 29, 2024
Second update, if I remove this line everything works fine.
I could not quite figure out what this line is supposed to do.
alexanderankin commentedon May 30, 2024
if you want, i can add a constructor boolean parameter for that, i have no idea what it is intended to do.
I'd also add a warning if that parameter is passed in as experimental as again, no idea what its for, and also boolean parameters are not great for maintainability, so we'd probably want to figure out what its for and avoid the extra param or generalize a bit.
tharwan commentedon May 30, 2024
Maybe @ash1425 still knows? I assume it has something to do with k3s-io/k3s#7265 but I am not sure. From the k3s docs I don’t get why the volume mount might be needed.
But since we can’t get it to work on two different systems, I thing an option would be good. Otherwise we will just use our own implementation, which is also fine.
alexanderankin commentedon May 30, 2024
might as well accept the contribution might help someone else, might also learn the answer if someone comes back to this issue with it. opened #592
alexanderankin commentedon May 30, 2024
if it works for you ill release it @tharwan
alexanderankin commentedon May 30, 2024
fix(k3s): add configuration parameter for disabling cgroup mount to a…
alexanderankin commentedon Jun 3, 2024
@tharwan this is out now