Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

System error in /sys/fs/cgroup/cpu,cpuactt #16256

Closed
fommil opened this issue Sep 12, 2015 · 52 comments
Closed

System error in /sys/fs/cgroup/cpu,cpuactt #16256

fommil opened this issue Sep 12, 2015 · 52 comments

Comments

@fommil
Copy link

fommil commented Sep 12, 2015

I'm on Debian Unstable with the docker.io images (debian no longer repackages). This is what happens when I try to run any image:

~ docker run -it debian:jessie
Error response from daemon: Cannot start container c35192b15227f0a93170415f688bd7ff974aaf7af63a168398d865d857ed2f75: [8] System error: open /sys/fs/cgroup/cpu,cpuacct/init.scope/system.slice/docker-c35192b15227f0a93170415f688bd7ff974aaf7af63a168398d865d857ed2f75.scope/cpu.shares: no such file or directory

and indeed, that structure does not exist on my directory

~ ls /sys/fs/cgroup/cpu,cpuacct/init.scope/
cgroup.clone_children  cgroup.procs  cpuacct.stat  cpuacct.usage  cpuacct.usage_percpu  cpu.shares  notify_on_release  tasks

i.e. no system.slice.

I'm running a very new linux kernel, has the sys directory perhaps changed to the point that docker breaks?


~ uname -a
Linux Samskara 4.1.0-2-amd64 #1 SMP Debian 4.1.6-1 (2015-08-23) x86_64 GNU/Linux

~ docker version
Client version: 1.7.1
Client API version: 1.19
Go version (client): go1.4.2
Git commit (client): 786b29d
OS/Arch (client): linux/amd64
Server version: 1.7.1
Server API version: 1.19
Go version (server): go1.4.2
Git commit (server): 786b29d
OS/Arch (server): linux/amd64

~ docker -D info
Containers: 3
Images: 13
Storage Driver: devicemapper
 Pool Name: docker-8:5-27000835-pool
 Pool Blocksize: 65.54 kB
 Backing Filesystem: extfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 1.601 GB
 Data Space Total: 107.4 GB
 Data Space Available: 105.8 GB
 Metadata Space Used: 1.978 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.146 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Data loop file: /mnt/archive/docker/devicemapper/devicemapper/data
 Metadata loop file: /mnt/archive/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.104 (2015-08-10)
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 4.1.0-2-amd64
Operating System: Debian GNU/Linux stretch/sid (containerized)
CPUs: 4
Total Memory: 15.62 GiB
Name: Samskara
ID: EM6A:FAHK:YV4K:3AUD:LWIU:RJ2Q:ZPOP:Q4C7:IIAX:ZIR6:P5UY:67GB
Username: fommil
Registry: https://index.docker.io/v1/
WARNING: No memory limit support
WARNING: No swap limit support
@fommil
Copy link
Author

fommil commented Sep 12, 2015

that auto response is very useful... are you using a service for that or did you hack something up? (I want to use it for my projects if it is possible to customise it to only respond once to a submitter to avoid spamming people)

@bosr
Copy link

bosr commented Sep 12, 2015

Hi @fommil

If you are on Debian Unstable, there is a chance you have just upgraded to systemd 226, which introduces the init.scope and places init (pid 1) inside it:

$ cat /proc/1/cgroup
9:cpuset:/
8:memory:/
7:devices:/init.scope
6:cpu,cpuacct:/
5:perf_event:/
4:net_cls,net_prio:/
3:freezer:/
2:blkio:/
1:name=systemd:/init.scope

source: http://news.softpedia.com/news/systemd-226-adds-new-features-to-the-dhcp-implementation-of-networkd-more-491265.shtml

I have the same error message as you.

@cdd-aix
Copy link

cdd-aix commented Sep 12, 2015

Downgrading all systemd components to 225 on Debian resolves the issue.

Bug #798778 opened against debian sid's systemd package.

Long term, docker should handle systemd's new init.scope. 4 days notice is insufficient :-).
Short term, no distribution should release new versions of systemd that make major changes to cgroup space without testing them against other packages that use cgroup space first.

@bosr
Copy link

bosr commented Sep 12, 2015

Agreed!

Romain

On 12 Sep 2015, at 16:48, Chris Dukes [email protected] wrote:

Downgrading all systemd components to 225 on Debian resolves the issue.

Bug #798778 opened against debian sid's systemd package.

Long term, docker should handle systemd's new init.scope. 4 days notice is insufficient :-).
Short term, no distribution should release new versions of systemd that make major changes to cgroup space without testing them against other packages that use cgroup space first.


Reply to this email directly or view it on GitHub.

@LK4D4
Copy link
Contributor

LK4D4 commented Sep 13, 2015

You can use fs cgroups if you want, they're much more stable and has more features.
--exec-opt native.cgroupdriver=cgroupfs to daemon flags.

@cdd-aix
Copy link

cdd-aix commented Sep 13, 2015

Alexander,

Interesting. That does appear to resolve the issue.

Thank you.

I have questions.

  1. If cgroupfs is preferable to systemd, why is the default to check for
    systemd then fallback to cgroupfs?
  2. Has systemd done other experiments with cgroup space with similar
    fallout?
  3. If not, what were the other historical instabilities?
  4. What are the additional features provided by cgroupfs as the driver?
    https://docs.docker.com/reference/commandline/daemon/
    is somewhat silent on the above questions.

I see the differences in /sys/fs/cgroup between the two. But the context
on the nature of the differences eludes me.

Again, Thank You.

On Sat, Sep 12, 2015 at 11:54 PM, Alexander Morozov <
[email protected]> wrote:

You can use fs cgroups if you want, they're much more stable and has more
features.
--exec-opt native.cgroupdriver=cgroupfs to daemon flags.


Reply to this email directly or view it on GitHub
#16256 (comment).

@fommil
Copy link
Author

fommil commented Sep 13, 2015

--exec-opt native.cgroupdriver=cgroupfs works for me! :-)

@fommil fommil closed this as completed Sep 13, 2015
@thaJeztah
Copy link
Member

@LK4D4 are there changes to be made, or docs to be added w.r.t #16256 (comment)? I don't know the answers to those, but let me know if the documentation should have extra information about that 👍

@fommil
Copy link
Author

fommil commented Sep 13, 2015

It would be good if this setting was automatically enabled for relevant systems

@thaJeztah
Copy link
Member

@fommil feel free to create a separate issue for that, if you can clearly describe what you think needs to be done (and please add a link to this issue as well, for reference).

@runcom
Copy link
Member

runcom commented Sep 13, 2015

Or libcontainer should handle the fact that with systemd >225 cgroups paths are different?

@epifanio
Copy link

Hi, i'm running in the same problem (debian sid) how/whre should I use the option : --exec-opt native.cgroupdriver=cgroupfs ?

@cdd-aix
Copy link

cdd-aix commented Sep 14, 2015

/etc/default
On Sep 14, 2015 2:03 PM, "epifanio" [email protected] wrote:

Hi, i'm running in the same problem (debian sid) how/whre should I use the
option : --exec-opt native.cgroupdriver=cgroupfs ?


Reply to this email directly or view it on GitHub
#16256 (comment).

@epifanio
Copy link

I tried .. that's what i got :

service docker stop
service docker start
docker daemon --exec-opt native.cgroupdriver=cgroupfs
docker: 'daemon' is not a docker command.
See 'docker --help'.

@thaJeztah
Copy link
Member

@epifanio the docker daemon command was added in docker 1.8, so either upgrade to docker 1.8, or use docker -d, which was the "old" way of starting the docker daemon

@epifanio
Copy link

@thaJeztah , thanks!
docker -d --exec-opt native.cgroupdriver=cgroupfs
works fine (eventually I'll upgrade my docker installation).
Question, i runned docker -d ... from a standard ssh shell on my server using nohup to avoid to kill the demon when i close the ssh session. How can i make those changes permanent, so that I don't need to issue it and leave supervisord to do the job?

@LK4D4
Copy link
Contributor

LK4D4 commented Sep 15, 2015

btw, it's still an issue, it breaks oom for systemd as well. I'm not sure from where this init.scope crap is came, because mine devices is in user.slice, but inside container it's mounted as /init.scope/system.slice
Also I don't see mention of init.scope anywhere in systemd docs. Which is another reason to use fs implementation.

@LK4D4 LK4D4 reopened this Sep 15, 2015
@cdd-aix
Copy link

cdd-aix commented Sep 15, 2015

Take a peek at /proc/1/cgroup.
devices and blkio are in /init.scope.

In systemd 225 and older, those are in /.

Suspicion:
native.cgroupdriver=systemd uses systemd's cgroups in creating paths for
docker cgroups.

The release notes for systemd 226 does not address the why.

systemd/init's metrics are easy to track when fenced in a separate cgroup
hierarchy.

I have not rebooted since enabling memory cgroups and swap accounting, but
I suspect I will be pleasantly surprised.

On Tue, Sep 15, 2015 at 12:42 PM, Alexander Morozov <
[email protected]> wrote:

btw, it's still an issue, it breaks oom for systemd as well. I'm not sure
from where this init.scope crap is came, because mine devices is in
user.slice, but inside container it's mounted as /init.scope/system.slice


Reply to this email directly or view it on GitHub
#16256 (comment).

@LK4D4
Copy link
Contributor

LK4D4 commented Sep 15, 2015

@cdd-aix funny. Seems like we're messed /proc/self with /proc/1 somehow.

@jeremyeder
Copy link

Seems no such issue exists in Fedora rawhide:

# rpm -q docker systemd
docker-1.9.0-2.gitf8950e0.fc24.x86_64
systemd-226-1.fc24.x86_64

# cat /proc/1/cgroup
10:hugetlb:/
9:memory:/init.scope
8:cpuset:/
7:net_cls,net_prio:/
6:perf_event:/
5:devices:/init.scope
4:blkio:/init.scope
3:freezer:/
2:cpu,cpuacct:/init.scope
1:name=systemd:/init.scope

# docker run fedora echo hi
hi
#

@venthur
Copy link

venthur commented Sep 17, 2015

docker -d --exec-opt native.cgroupdriver=cgroupfs works on current Debian/Unstable.

@paulbdavis
Copy link

The issue only seems to manifest when using -c (though I guess some other options that utilize cgroups would also cause the issue).

% docker version
Client:
 Version:      1.8.1
 API version:  1.20
 Go version:   go1.4.2
 Git commit:   d12ea79
 Built:        Sat Aug 15 17:29:10 UTC 2015
 OS/Arch:      linux/amd64

Server:
 Version:      1.8.1
 API version:  1.20
 Go version:   go1.4.2
 Git commit:   d12ea79
 Built:        Sat Aug 15 17:29:10 UTC 2015
 OS/Arch:      linux/amd64
 % pacman -Qi systemd
Name           : systemd
Version        : 226-1
Description    : system and service manager
Architecture   : x86_64
URL            : http://www.freedesktop.org/wiki/Software/systemd
Licenses       : GPL2  LGPL2.1
Groups         : None
Provides       : nss-myhostname  systemd-tools=226  udev=226
Depends On     : acl  bash  dbus  iptables  kbd  kmod  hwids  libcap  libgcrypt  libsystemd  libidn  lz4  pam  libseccomp  util-linux  xz
Optional Deps  : cryptsetup: required for encrypted block devices [installed]
                 libmicrohttpd: remote journald capabilities
                 quota-tools: kernel-level quota management
                 systemd-sysvcompat: symlink package to provide sysvinit binaries [installed]
                 polkit: allow administration as unprivileged user [installed]
Required By    : accountsservice  android-udev  bluez-utils  chromium  colord  device-mapper  dhcpcd  docker  gnome-system-monitor  lib32-systemd  libgusb  libinput  libpulse
                 libusb  lvm2  media-player-info  mesa  mkinitcpio  netctl  pcmciautils  pcsclite  plymouth-legacy  polkit  python-pyudev  qt5-base  qtwebkit  rtkit  subversion
                 systemd-sysvcompat  udisks2  upower  xf86-input-evdev
Optional For   : None
Conflicts With : nss-myhostname  systemd-tools  udev
Replaces       : nss-myhostname  systemd-tools  udev
Installed Size :  27.24 MiB
Packager       : Dave Reisner <[email protected]>
Build Date     : Tue 08 Sep 2015 01:57:54 PM EDT
Install Date   : Sun 20 Sep 2015 03:17:27 PM EDT
Install Reason : Installed as a dependency for another package
Install Script : Yes
Validated By   : Signature
 % uname -r
4.1.6-1-ARCH
 % docker run --rm ubuntu echo hi
hi
 % docker run --rm -c 1 ubuntu echo hi
Error response from daemon: Cannot start container c44478b6b516c75fa72e090e61db49a08453f25c7fe59e877e641214dc3dd721: [8] System error: open /sys/fs/cgroup/cpu,cpuacct/init.scope/system.slice/docker-c44478b6b516c75fa72e090e61db49a08453f25c7fe59e877e641214dc3dd721.scope/cpu.shares: no such file or directory

@flixi
Copy link

flixi commented Sep 23, 2015


[root@alarm services]# docker version
Client version: 1.7.1
Client API version: 1.19
Go version (client): go1.4.2
Git commit (client): 786b29d
OS/Arch (client): linux/arm
Server version: 1.7.1
Server API version: 1.19
Go version (server): go1.4.2
Git commit (server): 786b29d
OS/Arch (server): linux/arm
uname -r
4.2.0-1-ARCH

If I try to

docker run redis:latest

I get the error:

689d0208f5907a16b153bc213b9fb78777b7b2486890ddf8b8039c6c982e6396
Error response from daemon: Cannot start container 689d0208f5907a16b153bc213b9fb78777b7b2486890ddf8b8039c6c982e6396: [8] System error: open /sys/fs/cgroup/cpu,cpuacct/init.scope/system.slice/docker-689d0208f5907a16b153bc213b9fb78777b7b2486890ddf8b8039c6c982e6396.scope/cpu.shares: no such file or directory

Running ArchLinux on BeagleBone Black rev4.

@paulbdavis
Copy link

On arch, the fd it is looking for exists at /sys/fs/cgroup/cpu,cpuacct/system.slice/docker-blahblah/cpu.shares instead of /sys/fs/cgroup/cpu,cpuacct/init.scope/system.slice/docker-blahblah/cpu.shares

It's there, just not under the init.scope directory

@tinti
Copy link

tinti commented Sep 25, 2015

docker -d --exec-opt native.cgroupdriver=cgroupfs works on Arch.

docker

$ docker version
Client:
 Version:      1.8.2
 API version:  1.20
 Go version:   go1.5.1
 Git commit:   0a8c2e3-dirty
 Built:        Mon Sep 14 12:09:36 UTC 2015
 OS/Arch:      linux/amd64

Server:
 Version:      1.8.2
 API version:  1.20
 Go version:   go1.5.1
 Git commit:   0a8c2e3-dirty
 Built:        Mon Sep 14 12:09:36 UTC 2015
 OS/Arch:      linux/amd64

kernel

uname -r
4.1.6-1-ARCH

systemd

$ pacman -Qi systemd
Name           : systemd
Version        : 226-1
Description    : system and service manager
Architecture   : x86_64
URL            : http://www.freedesktop.org/wiki/Software/systemd
Licenses       : GPL2  LGPL2.1
Groups         : None
Provides       : nss-myhostname  systemd-tools=226  udev=226
Depends On     : acl  bash  dbus  iptables  kbd  kmod  hwids  libcap  libgcrypt  libsystemd  libidn  lz4  pam  libseccomp  util-linux  xz
Optional Deps  : cryptsetup: required for encrypted block devices [installed]
                 libmicrohttpd: remote journald capabilities
                 quota-tools: kernel-level quota management
                 systemd-sysvcompat: symlink package to provide sysvinit binaries [installed]
                 polkit: allow administration as unprivileged user [installed]
Required By    : accountsservice  android-udev  chromium  colord  device-mapper  dhcpcd  docker  lib32-systemd  libgusb  libinput  libmbim  libpulse  libusb  lvm2  media-player-info  mesa  mkinitcpio  modemmanager  netctl  pcmciautils
                 polkit  qt5-base  qtwebkit  rtkit  subversion  systemd-sysvcompat  udisks2  upower  xf86-input-evdev
Optional For   : None
Conflicts With : nss-myhostname  systemd-tools  udev
Replaces       : nss-myhostname  systemd-tools  udev
Installed Size :  27.24 MiB
Packager       : Dave Reisner <[email protected]>
Build Date     : Tue 08 Sep 2015 02:57:54 PM BRT
Install Date   : Mon 21 Sep 2015 07:45:44 AM BRT
Install Reason : Installed as a dependency for another package
Install Script : Yes
Validated By   : Signature

@Chris00
Copy link
Contributor

Chris00 commented Oct 6, 2015

Note that on Debian testing (with systemd), you have to add the option --exec-opt native.cgroupdriver=cgroupfs in /lib/systemd/system/docker.service:

...
[Service]
ExecStart=/usr/bin/docker -d -H fd:// --exec-opt native.cgroupdriver=cgroupfs
...

@johnjelinek
Copy link

👍 --exec-opt native.cgroupdriver=cgroupfs works for CentOS 7.1 too. I wish docker-machine create would configure this automatically.

@thaJeztah
Copy link
Member

@johnjelinek we recently merged a PR to use if as default for docker 1.10; #17704 there's still some discussion on that though.

@LK4D4 should this be resolved with the changes in #17704?

@johnjelinek
Copy link

hrmm .. I think I spoke too soon:

$ eval "$(docker-machine env --swarm my-swarm)"
$ docker run --rm redis

                _._
           _.-``__ ''-._
      _.-``    `.  `_.  ''-._           Redis 3.0.5 (00000000/0) 64 bit
  .-`` .-```.  ```\/    _.,_ ''-._
 (    '      ,       .-`  | `,    )     Running in standalone mode
 |`-._`-...-` __...-.``-._|'` _.-'|     Port: 6379
 |    `-._   `._    /     _.-'    |     PID: 1
  `-._    `-._  `-./  _.-'    _.-'
 |`-._`-._    `-.__.-'    _.-'_.-'|
 |    `-._`-._        _.-'_.-'    |           http://redis.io
  `-._    `-._`-.__.-'_.-'    _.-'
 |`-._`-._    `-.__.-'    _.-'_.-'|
 |    `-._`-._        _.-'_.-'    |
  `-._    `-._`-.__.-'_.-'    _.-'
      `-._    `-.__.-'    _.-'
          `-._        _.-'
              `-.__.-'
...

but then, with this docker-compose.yml:

web:
  build: .
  command: python app.py
  ports:
     - "5000"
  hostname: hello.weave.local
  environment:
     - "affinity:container!=app_web_*"
redis:
  image: redis
  hostname: redis.weave.local

it fails on the redis spin up:

$ docker-compose up -d --force-recreate
Recreating app_web_1
Recreating app_redis_1
ERROR: Cannot start container 3d19005a57b6237bddbd868d5f158126001eed9cbcc8027e0c330f3410b23d69: [8] System error: write /sys/fs/cgroup/memory/system.slice/docker/3d19005a57b6237bddbd868d5f158126001eed9cbcc8027e0c330f3410b23d69/memory.swappiness: invalid argument

All the swarm dockers are configured with --exec-opt native.cgroupdriver=cgroupfs.

@thaJeztah
Copy link
Member

@johnjelinek could that be related to #15159?

@johnjelinek
Copy link

I don't think so -- I didn't unmount anything. The problem existed before I did the --exec-opt native.cgroupdriver=cgroupfs fix above. How can I troubleshoot further?

@thaJeztah
Copy link
Member

@johnjelinek hm actually, it looks like #17879, could be related to some API changes w.r.t default values, but not sure

@johnjelinek
Copy link

I manually created this file: /sys/fs/cgroup/memory/system.slice/docker/8983d7cd867be40dcfd68747c79fac2f713243a979c3011dc1997dcdf8665150/memory.swappiness because that's what was complaining. When I did docker-compose up -d again, the 898...150 folder went missing. I guess it was deleted when the container was recreated.

@johnjelinek
Copy link

@thaJeztah: I think you're right. It's likely #17879.

@marineam
Copy link

Just for the sake of completeness since I didn't see it explicitly stated, for recent versions of systemd that use init.scope for pid 1 libcontainer should not be including it in this join: https://github.com/opencontainers/runc/blob/master/libcontainer/cgroups/systemd/apply_systemd.go#L384

@marineam
Copy link

Instead of assuming things about systemd's cgroup behavior it could be asked instead:

$ busctl get-property org.freedesktop.systemd1 /org/freedesktop/systemd1/unit/system_2eslice org.freedesktop.systemd1.Slice ControlGroup
s "/system.slice"

That should work regardless of systemd version

@thaJeztah
Copy link
Member

@marineam are you still seeing this on docker 1.11.1? We're using runC now for the runtime, and a lot has changed since this was reported

@marineam
Copy link

This code is in runc and unchanged since docker 1.10 but I haven't actually tried docker 1.11 yet.

@HelenaZheng
Copy link

I fixed by restart the docker serivce

systemctl restart docker

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests