Containers fail to start when using --memory in docker 17.09.0-ce #35123

TimWolla · 2017-10-07T17:47:32Z

Note: This already happened with 17.06, but was fixed with 17.07, it reappears with 17.09, see #34616

Description

Containers fail to start with --memory set.

Steps to reproduce the issue:

docker run -it --rm --memory 1G debian bash

Describe the results you received:

docker: Error response from daemon: oci runtime error: container_linux.go:265: starting container process caused "process_linux.go:284: applying cgroup configuration for process caused \"No such device or address\"".

Describe the results you expected:
A bash running inside a Debian container.

Additional information you deem important (e.g. issue happens only occasionally):
Happens even after a reboot of the complete machine.

Output of docker version:

Client:
 Version:      17.09.0-ce
 API version:  1.32
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:42:09 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.09.0-ce
 API version:  1.32 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:40:48 2017
 OS/Arch:      linux/amd64
 Experimental: false

Output of docker info:

Containers: 24
 Running: 24
 Paused: 0
 Stopped: 0
Images: 138
Server Version: 17.09.0-ce
Storage Driver: zfs
 Zpool: rpool
 Zpool Health: ONLINE
 Parent Dataset: rpool/docker
 Space Used By Parent: 3726835712
 Space Available: 770689671168
 Parent Quota: no
 Compression: on
Logging Driver: json-file
Cgroup Driver: systemd
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0
runc version: 3f2f8b84a77f73d38244dd690525642a72156c64
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.9.0-4-amd64
Operating System: Debian GNU/Linux 9 (stretch)
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.57GiB
Name: *snip*
ID: *snip*
Docker Root Dir: /docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: true

Additional environment details (AWS, VirtualBox, physical, etc.):
Dedicated bare metal host running Debian Stretch.

The text was updated successfully, but these errors were encountered:

thaJeztah · 2017-10-07T18:18:18Z

ping @tiborvass

tiny1990 · 2017-10-13T02:54:01Z

it's works in cgroupfs mode , you can change to test, but i do not know why

huzhengchuan · 2017-10-16T03:11:56Z

I meet the same question in centos7.2.

[root@k8s1-8-master ~]# docker version
Client:
Version: 17.09.0-ce
API version: 1.32
Go version: go1.8.3
Git commit: afdb6d4
Built: Tue Sep 26 22:41:23 2017
OS/Arch: linux/amd64

Server:
Version: 17.09.0-ce
API version: 1.32 (minimum version 1.12)
Go version: go1.8.3
Git commit: afdb6d4
Built: Tue Sep 26 22:42:49 2017
OS/Arch: linux/amd64
Experimental: false

docker-containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --metrics-interval=0 --start-timeout 2m --state-dir /var/run/docker/libcontainerd/containerd --shim docker-containerd-shim --runtime docker-runc --runtime-args --systemd-cgroup=true
use systemd as group manage

huzhengchuan · 2017-10-16T03:55:55Z

upgrade to docker 17.10.0-ce-rc2 work well.
But I didnot know the issue in moby. who know, can ref it.

gyliu513 · 2017-10-24T23:30:59Z

Yes, seems using Cgroup Driver: cgroupfs works fine, can anyone explain why does this works?

thaJeztah · 2017-10-25T11:43:09Z

The systemd cgroups implementation is known to be troublesome, and frequently breaks between systemd versions. For that reason, the default is to use cgroupfs (see #17704, #16238, #16256, and various other issues). Unless you have a very specific reason to require systemd cgroups, I'd recommend using the cgroupfs option.

@TimWolla does switching to cgroupfs resolve your issue as well?

TimWolla · 2017-10-26T16:03:17Z

@TimWolla does switching to `cgroupfs` resolve your issue as well?

Yes, it does. Am 25.10.2017 um 13:44 schrieb Sebastiaan van Stijn:

…

The systemd cgroups implementation is known to be troublesome, and frequently breaks between systemd versions. For that reason, the default is to use `cgroupfs` (see #17704, #16238, #16256, and various other issues). Unless you have a very specific reason to require systemd cgroups, I'd recommend using the cgroupfs option. @TimWolla does switching to `cgroupfs` resolve your issue as well?

tiny1990 · 2017-10-27T02:14:20Z

@thaJeztah what's difference between cgroupfs and systemd

tossmilestone · 2018-02-06T09:23:42Z

@thaJeztah Sorry to bother, I want to make sure that if this issue is fixed on 17.12, because we may upgrade to this version if it was fixed.

darshan-karia · 2018-02-09T19:14:28Z

@tossmilestone I tried 17.12 in past and it did not work. The version which worked for me was 17.10

thaJeztah · 2018-02-10T00:55:38Z

@darshan-karia what problem did you run into? Testing this on a 17.12 machine:

$ docker run -it --rm --memory 1G alpine echo hello world
WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
hello world

That warning indicates that the machine doesn't support swap accounting. The container started, but limiting swap is not applied; after enabling (see Your kernel does not support cgroup swap limit capabilities);

$ docker run -it --rm --memory 1G alpine echo hello world
hello world

darshan-karia · 2018-03-06T10:57:23Z

@thaJeztah The issue I was encountering was when running a systemd container with memory option.
But, I tried running systemd container after your comment, and that seems to be working.

I must have confused the versions, as I was trying many for my experiment, sorry about confusion.

This is a test case for issue moby#35123, making sure we can set container's memory limit when using `native.cgroupdriver=systemd`. [v2: skip if no systemd present] [v3: add --iptables=false to avoid flaky tests with t.Parallel()] [v4: rebase after PR#36507 merge] Signed-off-by: Kir Kolyshkin <[email protected]>

This is a test case for issue moby/moby#35123, making sure we can set container's memory limit when using `native.cgroupdriver=systemd`. [v2: skip if no systemd present] [v3: add --iptables=false to avoid flaky tests with t.Parallel()] [v4: rebase after PR#36507 merge] Signed-off-by: Kir Kolyshkin <[email protected]> (cherry picked from commit 4ca5c5361059e29ed31074ca5b96f8b2030b5f99) Signed-off-by: Sebastiaan van Stijn <[email protected]>

thaJeztah · 2018-03-22T13:00:31Z

The fix for this should be in Docker 17.12.1, and a test is added in #36172. I'll go ahead and close this 👍

This is a test case for issue moby/moby#35123, making sure we can set container's memory limit when using `native.cgroupdriver=systemd`. [v2: skip if no systemd present] [v3: add --iptables=false to avoid flaky tests with t.Parallel()] [v4: rebase after PR#36507 merge] Signed-off-by: Kir Kolyshkin <[email protected]> Upstream-commit: 4ca5c5361059e29ed31074ca5b96f8b2030b5f99 Component: engine

This is a test case for issue moby/moby#35123, making sure we can set container's memory limit when using `native.cgroupdriver=systemd`. [v2: skip if no systemd present] [v3: add --iptables=false to avoid flaky tests with t.Parallel()] [v4: rebase after PR#36507 merge] Signed-off-by: Kir Kolyshkin <[email protected]> (cherry picked from commit 4ca5c53) Signed-off-by: Sebastiaan van Stijn <[email protected]>

GordonTheTurtle added area/runtime version/17.09 labels Oct 7, 2017

kolyshkin mentioned this issue Jan 31, 2018

Test for systemd cgroupdriver memory setting #36172

Merged

thaJeztah mentioned this issue Mar 22, 2018

[18.03] Test for systemd cgroupdriver memory setting docker-archive/docker-ce#476

Merged

thaJeztah closed this as completed Mar 22, 2018

SpheMakh mentioned this issue Jun 19, 2018

restrict cpu/ram usage ratt-ru/Stimela-classic#263

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Containers fail to start when using --memory in docker 17.09.0-ce #35123

Containers fail to start when using --memory in docker 17.09.0-ce #35123

TimWolla commented Oct 7, 2017

thaJeztah commented Oct 7, 2017

tiny1990 commented Oct 13, 2017

huzhengchuan commented Oct 16, 2017 •

edited

Loading

huzhengchuan commented Oct 16, 2017

gyliu513 commented Oct 24, 2017

thaJeztah commented Oct 25, 2017

TimWolla commented Oct 26, 2017 via email

tiny1990 commented Oct 27, 2017 •

edited

Loading

tossmilestone commented Feb 6, 2018

darshan-karia commented Feb 9, 2018 •

edited

Loading

thaJeztah commented Feb 10, 2018

darshan-karia commented Mar 6, 2018

thaJeztah commented Mar 22, 2018

Containers fail to start when using --memory in docker 17.09.0-ce #35123

Containers fail to start when using --memory in docker 17.09.0-ce #35123

Comments

TimWolla commented Oct 7, 2017

thaJeztah commented Oct 7, 2017

tiny1990 commented Oct 13, 2017

huzhengchuan commented Oct 16, 2017 • edited Loading

huzhengchuan commented Oct 16, 2017

gyliu513 commented Oct 24, 2017

thaJeztah commented Oct 25, 2017

TimWolla commented Oct 26, 2017 via email

tiny1990 commented Oct 27, 2017 • edited Loading

tossmilestone commented Feb 6, 2018

darshan-karia commented Feb 9, 2018 • edited Loading

thaJeztah commented Feb 10, 2018

darshan-karia commented Mar 6, 2018

thaJeztah commented Mar 22, 2018

huzhengchuan commented Oct 16, 2017 •

edited

Loading

tiny1990 commented Oct 27, 2017 •

edited

Loading

darshan-karia commented Feb 9, 2018 •

edited

Loading