Skip to content

[WSL 2.9 regression] Rootless Docker/Podman cannot start any container — standard /sys/fs/cgroup/user.slice path missing after cgroup hierarchy moved under /wsl-user/distro-N/ #41492

Closed

Description

Windows Version

10.0.26300.9278

WSL Version

2.9.9.0 (Pre-release channel)

Full wsl --version:

WSL version: 2.9.9.0
Kernel version: 6.18.40.1-1
WSLg version: 1.0.79
MSRDC version: 1.2.7214
Direct3D version: 1.611.1-81528511
DXCore version: 10.0.26100.1-240331-1435.ge-release
Windows version: 10.0.26300.9278

Are you using WSL 1 or WSL 2?

  • WSL 2

Kernel Version

6.18.40.1-microsoft-standard-WSL2

Distro Version

Fedora Linux 44 (WSL). Also reproduced on Fedora Linux 42 (WSL) after upgrading WSL to 2.9.x.

Other Software

Docker CE rootless mode (dockerd-rootless), running as a systemctl --user service:

Docker version 29.7.2, build a7dcaa6
runc version 1.4.3
rootlesskit version 3.0.2
containerd containerd v2.3.4

Repro Steps

Rootless Docker (and, by the same root cause, rootless Podman) can no longer start any container after updating WSL to 2.9.x. The identical setup works on WSL 2.7.x.

  1. On WSL 2.9.9 with systemd enabled, run rootless Docker as a user service (standard dockerd-rootless-setuptool.sh install).

  2. Attempt to run any container as the (non-root) user:

    $ docker run --rm hello-world
    docker: Error response from daemon: open /sys/fs/cgroup/user.slice/user-1000.slice/cgroup.controllers: no such file or directory

    Setting a CPU limit fails the same way (the daemon reports no CFS support):

    $ docker run --rm --cpus=1.0 hello-world
    docker: Error response from daemon: NanoCPUs can not be set, as your kernel does not support CPU CFS scheduler or the cgroup is not mounted

Root cause (isolated)

WSL 2.9 relocated the entire cgroup v2 hierarchy under a per-distro root (/wsl-user/distro-<N>/…), leaving the standard top-level /sys/fs/cgroup/user.slice/… path that rootless Docker/rootlesskit expect absent:

# The path the rootless daemon looks for — MISSING:
$ ls /sys/fs/cgroup/user.slice/user-1000.slice/cgroup.controllers
ls: cannot access '/sys/fs/cgroup/user.slice/user-1000.slice/cgroup.controllers': No such file or directory

# Where the cgroups actually live under 2.9:
$ cut -d: -f3 /proc/self/cgroup
/wsl-user/distro-282/non-systemd

$ systemctl show [email protected] -p ControlGroup --value
/wsl-user/distro-282/systemd/user.slice/user-1000.slice/[email protected]

The following rule out the kernel and Docker as the cause:

  • Kernel supports CFS bandwidth and the interface exists (so this is not a missing kernel config):

    $ zcat /proc/config.gz | grep CFS_BANDWIDTH
    CONFIG_CFS_BANDWIDTH=y
    $ ls /sys/fs/cgroup/wsl-user/distro-282/systemd/user.slice/user-1000.slice/[email protected]/cpu.max
    .../cpu.max        # present
  • Rootful (system) Docker works, including CPU limits — isolating the fault to the rootless / systemd-user cgroup handling under WSL 2.9:

    $ sudo docker run --rm --cpus=1.0 hello-world     # succeeds
  • I also observed more than one distro cgroup root appearing across sessions (/wsl-user/distro-282/… and /wsl-user/distro-285/…), which suggests the per-distro cgroup subtree and controller delegation are not being set up consistently.

Expected Behavior

Rootless containers start and per-container resource limits (--cpus, etc.) work — the behavior on WSL 2.7.x. The standard cgroup v2 delegation (/sys/fs/cgroup/user.slice/user-<uid>.slice/… with the cpu/memory controllers delegated to user@<uid>.service) should be present so that rootlesskit/dockerd and podman operate normally.

Actual Behavior

After updating to WSL 2.9.x, rootless Docker cannot start any container. rootlesskit/dockerd fail resolving /sys/fs/cgroup/user.slice/user-1000.slice/cgroup.controllers (which no longer exists because the hierarchy was moved under /wsl-user/distro-<N>/…), and the daemon reports no CFS/CPU-quota support even though the kernel provides it. This is a regression from WSL 2.7.x, where the same rootless setup works. Related prior reports of rootless-container/systemd-user cgroup breakage on the WSL tracker: #13053, #13143.

Diagnostic Logs

Key values are inline above (wsl --version, kernel, /proc/self/cgroup, the missing path, and CONFIG_CFS_BANDWIDTH). I can attach a full collect-wsl-logs.ps1 bundle and strace of the failing dockerd/rootlesskit on request.

Activity

  1. github-actions commented on Sep 1, 2026

    @github-actions
    Contributor

    Logs are required for review from WSL team

    If this a feature request, please reply with '/feature'. If this is a question, reply with '/question'.
    Otherwise, please attach logs by following the instructions below, your issue will not be reviewed unless they are added. These logs will help us understand what is going on in your machine.

    How to collect WSL logs

    Download and execute collect-wsl-logs.ps1 in an administrative powershell prompt:

    Invoke-WebRequest -UseBasicParsing "https://raw.githubusercontent.com/microsoft/WSL/master/diagnostics/collect-wsl-logs.ps1" -OutFile collect-wsl-logs.ps1
    Set-ExecutionPolicy Bypass -Scope Process -Force
    .\collect-wsl-logs.ps1
    

    The script will output the path of the log file once done.

    If this is a networking issue, please use .\collect-wsl-logs.ps1 -LogProfile networking instead, following the instructions in Collect WSL logs for networking issues

    Once completed please upload the output files to this GitHub issue.

    See Collect WSL logs (recommended method).

    If you choose to email these logs instead of attaching them to the bug, please send them to [email protected] with the GitHub issue number in the subject, and include a link to your GitHub issue comment in the message body, and reply with '/emailed-logs'.

  2. josenobile commented on Sep 1, 2026

    @josenobile
    Author

    /emailed-logs

  3. github-actions commented on Sep 1, 2026

    @github-actions
    Contributor
    Diagnostic information
    Found '/emailed-logs', adding tag 'emailed-logs'
    
  4. josenobile commented on Sep 1, 2026

    @josenobile
    Author

    Update — diagnostic logs emailed

    The collect-wsl-logs.ps1 bundle has been captured on WSL 2.9.9.0 (kernel 6.18.40.1-microsoft-standard-WSL2, Fedora 44) with a live reproduction of the failure inside the trace window, and emailed to [email protected] per the bot's instructions (/emailed-logs reply follows).

    Deterministic repro (100% on 2.9.x) — even a no-limit container fails:

    $ docker run --rm hello-world
    docker: Error response from daemon: open /sys/fs/cgroup/user.slice/user-1000.slice/cgroup.controllers: no such file or directory

    Root cause — the standard cgroup path is absent because 2.9 relocated the hierarchy under /wsl-user/distro-<N>/:

    $ cut -d: -f3 /proc/self/cgroup
    /wsl-user/distro-280/non-systemd
    
    $ ls /sys/fs/cgroup/user.slice/user-1000.slice/cgroup.controllers
    ls: cannot access '.../cgroup.controllers': No such file or directory

    Confirmed NOT the kernel and NOT Docker:

    • Kernel provides CFS bandwidth: CONFIG_CFS_BANDWIDTH=y.
    • Rootful (system) Docker works, including --cpus — isolating the fault to the rootless / systemd --user cgroup delegation under 2.9.

    Contrast — WSL 2.7.12 (same distro/kernel line), everything works:

    $ cut -d: -f3 /proc/self/cgroup
    /init.scope
    
    $ ls /sys/fs/cgroup/user.slice/user-1000.slice/cgroup.controllers
    /sys/fs/cgroup/user.slice/user-1000.slice/cgroup.controllers      # present
    
    $ docker run --rm --cpus=1.0 hello-world                           # succeeds

    This pins the regression to the WSL 2.9 cgroup-hierarchy change: the standard top-level /sys/fs/cgroup/user.slice/user-<uid>.slice/… (with cpu/memory delegated to user@<uid>.service) is no longer presented, so rootlesskit/dockerd (and rootless podman, cf. #13053) cannot operate.

  5. github-actions commented on Sep 1, 2026

    @github-actions
    Contributor
    Diagnostic information
    Found '/emailed-logs', adding tag 'emailed-logs'
    
  6. chemwolf6922 commented on Sep 1, 2026

    @chemwolf6922
    Contributor

    Hi Jose Nobile (@josenobile) .
    I'll look into it. You can disable the per-distro cgroup with this in your .wslconfig

    [wsl2]
    isolateDistroCgroup=false
    

    The different cgroup name in different sessions is normal. As that number is the init's pid. And it is not supposed to be stable across boots.

  7. chemwolf6922 commented on Sep 1, 2026

    @chemwolf6922
    Contributor

    Hi Jose Nobile (@josenobile) . I tested with podman. It's rootless mode works. The problem with docker is that it hard codes the user slice path regardless of where systemd is at.

  8. chemwolf6922 commented on Sep 1, 2026

    @chemwolf6922
    Contributor

    I created an issue in moby to track this: moby/moby#53531

  9. AkihiroSuda commented on Sep 1, 2026

    @AkihiroSuda

    This issue will probably affect nerdctl and possibly other container engines too:

  10. bricss commented on Sep 5, 2026

    @bricss

    I'm getting this error 🔴 on my side:

    Error: crun: controller `pids` is not available under /sys/fs/cgroup/wsl-user/distro-252/machine.slice/libpod-c19754532653dcf8077450326754f140ec2a0fffa76e7f13cd2592ed86f1451d.scope/container/cgroup.controllers: OCI runtime error
    • WSL: 2.9.10
    • Podman: 6.1.1 w/ Fedora 44 (rootful)

    The only working ⚙️ solution right now: podman run --cgroups=disabled IMAGE

  11. chemwolf6922 commented on Sep 7, 2026

    @chemwolf6922
    Contributor

    Hi Yahor Siarheyenka (@bricss) .
    Could you inspect the layout from /sys/fs/cgroup/ to /sys/fs/cgroup/wsl-user/distro-<pid>/machine.slice to see at which level did the controller went missing?
    Please also share the exact podman command and crun version that triggerd this. And if systemd is enabled for your distro.
    A wsl log collection will also help diagnose this.

  12. chemwolf6922 commented on Sep 7, 2026

    @chemwolf6922
    Contributor

    I created a build here https://github.com/chemwolf6922/build-wsl/actions/runs/34079611427 based on #41512. Please help verify if this build resolves any cgroup related issues.
    Please note this is a dev build and may contain other issues. It's recommended to switch back to a release or pre-release version after the test.

  13. bricss commented on Sep 12, 2026

    @bricss

    Here we go: ⤵️

    find /sys/fs/cgroup -name cgroup.controllers -print -exec sh -c 'echo $1; cat "$1"' _ {} \;
    /sys/fs/cgroup/wsl-user/distro-254/sys-fs-fuse-connections.mount/cgroup.controllers
    /sys/fs/cgroup/wsl-user/distro-254/sys-fs-fuse-connections.mount/cgroup.controllers
    /sys/fs/cgroup/wsl-user/distro-254/sys-kernel-config.mount/cgroup.controllers
    /sys/fs/cgroup/wsl-user/distro-254/sys-kernel-config.mount/cgroup.controllers
    /sys/fs/cgroup/wsl-user/distro-254/sys-kernel-debug.mount/cgroup.controllers
    /sys/fs/cgroup/wsl-user/distro-254/sys-kernel-debug.mount/cgroup.controllers
    /sys/fs/cgroup/wsl-user/distro-254/dev-mqueue.mount/cgroup.controllers
    /sys/fs/cgroup/wsl-user/distro-254/dev-mqueue.mount/cgroup.controllers
    /sys/fs/cgroup/wsl-user/distro-254/user.slice/cgroup.controllers
    /sys/fs/cgroup/wsl-user/distro-254/user.slice/cgroup.controllers
    /sys/fs/cgroup/wsl-user/distro-254/user.slice/user-1000.slice/[email protected]/session.slice/dbus-broker.service/cgroup.controllers
    /sys/fs/cgroup/wsl-user/distro-254/user.slice/user-1000.slice/[email protected]/session.slice/dbus-broker.service/cgroup.controllers
    /sys/fs/cgroup/wsl-user/distro-254/user.slice/user-1000.slice/[email protected]/session.slice/cgroup.controllers
    /sys/fs/cgroup/wsl-user/distro-254/user.slice/user-1000.slice/[email protected]/session.slice/cgroup.controllers
    /sys/fs/cgroup/wsl-user/distro-254/user.slice/user-1000.slice/[email protected]/user.slice/podman-pause-5ba36d0a.scope/cgroup.controllers
    /sys/fs/cgroup/wsl-user/distro-254/user.slice/user-1000.slice/[email protected]/user.slice/podman-pause-5ba36d0a.scope/cgroup.controllers
    /sys/fs/cgroup/wsl-user/distro-254/user.slice/user-1000.slice/[email protected]/user.slice/cgroup.controllers
    /sys/fs/cgroup/wsl-user/distro-254/user.slice/user-1000.slice/[email protected]/user.slice/cgroup.controllers
    /sys/fs/cgroup/wsl-user/distro-254/user.slice/user-1000.slice/[email protected]/app.slice/dbus.socket/cgroup.controllers
    /sys/fs/cgroup/wsl-user/distro-254/user.slice/user-1000.slice/[email protected]/app.slice/dbus.socket/cgroup.controllers
    /sys/fs/cgroup/wsl-user/distro-254/user.slice/user-1000.slice/[email protected]/app.slice/cgroup.controllers
    /sys/fs/cgroup/wsl-user/distro-254/user.slice/user-1000.slice/[email protected]/app.slice/cgroup.controllers
    /sys/fs/cgroup/wsl-user/distro-254/user.slice/user-1000.slice/[email protected]/init.scope/cgroup.controllers
    /sys/fs/cgroup/wsl-user/distro-254/user.slice/user-1000.slice/[email protected]/init.scope/cgroup.controllers
    /sys/fs/cgroup/wsl-user/distro-254/user.slice/user-1000.slice/[email protected]/cgroup.controllers
    /sys/fs/cgroup/wsl-user/distro-254/user.slice/user-1000.slice/[email protected]/cgroup.controllers
    /sys/fs/cgroup/wsl-user/distro-254/user.slice/user-1000.slice/cgroup.controllers
    /sys/fs/cgroup/wsl-user/distro-254/user.slice/user-1000.slice/cgroup.controllers
    /sys/fs/cgroup/wsl-user/distro-254/user.slice/user-1000.slice/session-c2.scope/cgroup.controllers
    /sys/fs/cgroup/wsl-user/distro-254/user.slice/user-1000.slice/session-c2.scope/cgroup.controllers
    /sys/fs/cgroup/wsl-user/distro-254/sys-kernel-tracing.mount/cgroup.controllers
    /sys/fs/cgroup/wsl-user/distro-254/sys-kernel-tracing.mount/cgroup.controllers
    /sys/fs/cgroup/wsl-user/distro-254/init.scope/cgroup.controllers
    /sys/fs/cgroup/wsl-user/distro-254/init.scope/cgroup.controllers
    /sys/fs/cgroup/wsl-user/distro-254/cgroup.controllers
    /sys/fs/cgroup/wsl-user/distro-254/cgroup.controllers
    cpuset cpu io memory hugetlb pids rdma
    /sys/fs/cgroup/wsl-user/distro-254/system.slice/systemd-udevd.service/udev/cgroup.controllers
    /sys/fs/cgroup/wsl-user/distro-254/system.slice/systemd-udevd.service/udev/cgroup.controllers
    /sys/fs/cgroup/wsl-user/distro-254/system.slice/systemd-udevd.service/cgroup.controllers
    /sys/fs/cgroup/wsl-user/distro-254/system.slice/systemd-udevd.service/cgroup.controllers
    /sys/fs/cgroup/wsl-user/distro-254/system.slice/dbus-broker.service/cgroup.controllers
    /sys/fs/cgroup/wsl-user/distro-254/system.slice/dbus-broker.service/cgroup.controllers
    /sys/fs/cgroup/wsl-user/distro-254/system.slice/systemd-homed.service/cgroup.controllers
    /sys/fs/cgroup/wsl-user/distro-254/system.slice/systemd-homed.service/cgroup.controllers
    /sys/fs/cgroup/wsl-user/distro-254/system.slice/podman.socket/cgroup.controllers
    /sys/fs/cgroup/wsl-user/distro-254/system.slice/podman.socket/cgroup.controllers
    /sys/fs/cgroup/wsl-user/distro-254/system.slice/system-sshd\x2dkeygen.slice/cgroup.controllers
    /sys/fs/cgroup/wsl-user/distro-254/system.slice/system-sshd\x2dkeygen.slice/cgroup.controllers
    /sys/fs/cgroup/wsl-user/distro-254/system.slice/system-modprobe.slice/cgroup.controllers
    /sys/fs/cgroup/wsl-user/distro-254/system.slice/system-modprobe.slice/cgroup.controllers
    /sys/fs/cgroup/wsl-user/distro-254/system.slice/systemd-journald.service/cgroup.controllers
    /sys/fs/cgroup/wsl-user/distro-254/system.slice/systemd-journald.service/cgroup.controllers
    /sys/fs/cgroup/wsl-user/distro-254/system.slice/sshd.service/cgroup.controllers
    /sys/fs/cgroup/wsl-user/distro-254/system.slice/sshd.service/cgroup.controllers
    /sys/fs/cgroup/wsl-user/distro-254/system.slice/gssproxy.service/cgroup.controllers
    /sys/fs/cgroup/wsl-user/distro-254/system.slice/gssproxy.service/cgroup.controllers
    /sys/fs/cgroup/wsl-user/distro-254/system.slice/tmp.mount/cgroup.controllers
    /sys/fs/cgroup/wsl-user/distro-254/system.slice/tmp.mount/cgroup.controllers
    /sys/fs/cgroup/wsl-user/distro-254/system.slice/systemd-userdbd.service/cgroup.controllers
    /sys/fs/cgroup/wsl-user/distro-254/system.slice/systemd-userdbd.service/cgroup.controllers
    /sys/fs/cgroup/wsl-user/distro-254/system.slice/cgroup.controllers
    /sys/fs/cgroup/wsl-user/distro-254/system.slice/cgroup.controllers
    /sys/fs/cgroup/wsl-user/distro-254/system.slice/rhsmcertd.service/cgroup.controllers
    /sys/fs/cgroup/wsl-user/distro-254/system.slice/rhsmcertd.service/cgroup.controllers
    /sys/fs/cgroup/wsl-user/distro-254/system.slice/systemd-logind.service/cgroup.controllers
    /sys/fs/cgroup/wsl-user/distro-254/system.slice/systemd-logind.service/cgroup.controllers
    /sys/fs/cgroup/wsl-user/distro-254/system.slice/var-lib-nfs-rpc_pipefs.mount/cgroup.controllers
    /sys/fs/cgroup/wsl-user/distro-254/system.slice/var-lib-nfs-rpc_pipefs.mount/cgroup.controllers
    /sys/fs/cgroup/wsl-user/distro-254/dev-hugepages.mount/cgroup.controllers
    /sys/fs/cgroup/wsl-user/distro-254/dev-hugepages.mount/cgroup.controllers
    /sys/fs/cgroup/wsl-user/cgroup.controllers
    /sys/fs/cgroup/wsl-user/cgroup.controllers
    cpuset cpu io memory hugetlb pids rdma
    /sys/fs/cgroup/wsl-user/non-distro/cgroup.controllers
    /sys/fs/cgroup/wsl-user/non-distro/cgroup.controllers
    cpuset cpu io memory hugetlb pids rdma
    /sys/fs/cgroup/cgroup.controllers
    /sys/fs/cgroup/cgroup.controllers
    cpuset cpu io memory hugetlb pids rdma
    crun --version
    crun version 1.28
    commit: 54f16ffbefcd022bf032af768b5c5ce075c18bfc
    rundir: /run/crun
    spec: 1.0.0
    +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +JSON_C
    podman --version
    podman version 6.1.1
    wsl --version
    WSL version: 2.9.11.0
    Kernel version: 6.18.40.1-1
    WSLg version: 1.0.79
    MSRDC version: 1.2.7214
    Direct3D version: 1.611.1-81528511
    DXCore version: 10.0.26100.1-240331-1435.ge-release
    Windows version: 10.0.26300.9445

    Literally any podman run causing error ⚠️ unless --cgroups=disabled added ➕ to it >_

  14. chemwolf6922 commented on Sep 14, 2026

    @chemwolf6922
    Contributor

    Hi Yahor Siarheyenka (@bricss) . Thanks for the info. This is an issue with how processes are arranged with the systemd-less boots. There are processes put into the distro-N group which prohibits the subgroup controllers to be enabled.
    I updated that PR to resolve this. If you want to, please try out this dev build:
    https://github.com/chemwolf6922/build-wsl/actions/runs/34798617405
    Please note this is a dev build and may contain other issues. It's recommended to switch back to a release or pre-release version after the test.

  15. l0rd commented on Sep 22, 2026

    @l0rd

    I can still reproduce the problem after applying #41512.

    I have tested by building WSL from the latest commit on master branch.

    $ wsl --version
    WSL version: 2.9.1.360
    Kernel version: 6.18.40.1-1
    WSLg version: 1.0.79
    MSRDC version: 1.2.7214
    Direct3D version: 1.611.1-81528511
    DXCore version: 10.0.26100.1-240331-1435.ge-release
    Windows version: 10.0.26200.9457
    MSBuild version: 1944
    Commit: 56244fdb
    Build time: 16:59:12 Sep 22 2026
    

    We have opened an issue Podman side and are evaluating turning off systemd delegation for cgroups on WSL.

    I am testing on an ARM64 laptop.

  16. l0rd commented on Sep 23, 2026

    @l0rd

    For completeness, the Podman WSL distribution doesn't set systemd=true in /etc/wsl.conf but starts systemd in a namespace (for historical reasons). We want to change that, and it would fix this problem too, but for now, systemd cgroup propagation seems broken for us.

  17. chemwolf6922 commented on Sep 28, 2026

    @chemwolf6922
    Contributor

    Hi Mario Loriedo (@l0rd) .
    I think using the built-in systemd support would be the clean solution here.
    Another method I can think of is to move the process to be execed into systemd to the distro root first and then exec systemd there. That's the rationale of me creating the non-systemd layer and putting things there.
    Since the distro namespace is no longer the real root, systemd won't be able to enable the sub-tree control when other processes still present in its cgroup. No matter if they are put in /non-system or / together.

  18. l0rd commented on Sep 28, 2026

    @l0rd

    I think using the built-in systemd support would be the clean solution he

    Feng Wang (@chemwolf6922) agreed

  19. chemwolf6922 commented on Nov 11, 2026

    @chemwolf6922
    Contributor

    The original issue was fixed in 3.0.x released versions.
    The podman desktop issue was fixed in podman v6.2.0.

  20. chemwolf6922 commented on Nov 11, 2026

    @chemwolf6922
    Contributor

    According to this: podman-container-tools/podman#29837. Podman with wsl systemd support will be in 7.x versions.

  21. ptrivedi commented on Nov 11, 2026

    @ptrivedi
    Contributor

    Thanks for closing, Feng Wang (@chemwolf6922)

  22. l0rd commented on Nov 11, 2026

    @l0rd

    According to this: podman-container-tools/podman#29837. Podman with wsl systemd support will be in 7.x versions.

    Podman v6.2.0 has WSL systemd support. We released it yesterday.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    emailed-logsLogs have been emailedexternalIssue exists outside of WSL componentswsl2Issue/feature applies to WSL 2

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions