Running container killed with failure to write to cgroup.procs #1326
Description
On our Jenkins CI infrastructure we run Maven builds inside a Docker container. Unfortunately once in a while the build container crashes during the execution of the Maven build with a failure writing a pid to the cgroup.proc file.
Feb` 13 23:56:41 myhost dockerd-current[5659]: time="2017-02-13T23:56:41.695619134+01:00" level=error msg="Error running exec in container: rpc error: code = 13 desc = invalid header field value "oci runtime error: exec failed: container_linux.go:247: starting container process caused \"process_linux.go:87: adding pid 21890 to cgroups caused \\\"failed to write 21890 to cgroup.procs: write /sys/fs/cgroup/cpu,cpuacct/system.slice/docker-f2ea7bd5f37f4d5719fec4a05fdb58401207c98b6abbfa02e497af4bc167ec08.scope/cgroup.procs: invalid argument\\\"\"\n\
Feb 14 04:34:52 myhost dockerd-current[5659]: time="2017-02-14T04:34:52.084545467+01:00" level=error msg="Error running exec in container: rpc error: code = 13 desc = invalid header field value "oci runtime error: exec failed: container_linux.go:247: starting container process caused \"process_linux.go:87: adding pid 36656 to cgroups caused \\\"failed to write 36656 to cgroup.procs: write /sys/fs/cgroup/cpu,cpuacct/system.slice/docker-2b412ea4871f6b1ed33547224bac344ea677f1814604a501a44e42ce84b64854.scope/cgroup.procs: invalid argument\\\"\"\n""
Feb 14 06:20:22 myhost dockerd-current[5659]: time="2017-02-14T06:20:22.751841415+01:00" level=error msg="Error running exec in container: rpc error: code = 13 desc = invalid header field value "oci runtime error: exec failed: container_linux.go:247: starting container process caused \"process_linux.go:87: adding pid 30239 to cgroups caused \\\"failed to write 30239 to cgroup.procs: write /sys/fs/cgroup/cpu,cpuacct/system.slice/docker-1418520af414c01054dce8ca3777b616289ee41a0de7ad135af8e2e740472a49.scope/cgroup.procs: invalid argument\\\"\"\n""```
I assume the error is thrown from https://github.com/opencontainers/runc/blob/v1.0.0-rc2/libcontainer/cgroups/utils.go#L422 , which boils down to https://github.com/golang/go/blob/master/src/io/ioutil/ioutil.go#L76 and https://github.com/golang/go/blob/master/src/os/file.go#L139
Running: 3
Paused: 0
Stopped: 7
Images: 54
Server Version: 1.12.6
Storage Driver: devicemapper
Pool Name: docker-thinpool
Pool Blocksize: 524.3 kB
Base Device Size: 10.74 GB
Backing Filesystem: xfs
Data file:
Metadata file:
Data Space Used: 10.18 GB
Data Space Total: 45.1 GB
Data Space Available: 34.92 GB
Metadata Space Used: 4.526 MB
Metadata Space Total: 4.295 GB
Metadata Space Available: 4.29 GB
Thin Pool Minimum Free Space: 4.509 GB
Udev Sync Supported: true
Deferred Removal Enabled: true
Deferred Deletion Enabled: true
Deferred Deleted Device Count: 0
Library Version: 1.02.136 (2016-11-05)
Logging Driver: journald
Cgroup Driver: systemd
Plugins:
Volume: local
Network: overlay host null bridge
Swarm: inactive
Runtimes: oci runc
Default Runtime: oci
Security Options: seccomp selinux
Kernel Version: 4.9.7-201.fc25.x86_64
Operating System: Fedora 25 (Atomic Host)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 2
CPUs: 56
Total Memory: 125.8 GiB
Name: myhost
ID: XDZT:BINX:3JZJ:BABH:6WSS:T2D5:Z5XJ:FM3Y:HOG7:XB33:T22Z:F2IS
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Insecure Registries:
127.0.0.0/8
Registries: docker.io (secure) ```