Skip to content

Linux: ancestor walk fails unconditionally in Docker userns-remap containers (single-uid mapping, / owned by nobody/65534) #1830

New issue
New issue
Closed
Closed
Linux: ancestor walk fails unconditionally in Docker userns-remap containers (single-uid mapping, / owned by nobody/65534)#1830
Labels
bugSomething isn't workingpriority/highNeeds near-term maintainer attention; high-impact bug, regression, safety issue, or release blocker.securitySecurity vulnerabilities, hardeningstability/performanceServer crashes, OOM, hangs, high CPU/memory

Description

@zhaoops
zhaoops
opened on Aug 25, 2026

Linux: ancestor walk fails unconditionally in Docker userns-remap containers (single-uid mapping, / owned by nobody/65534)

Version

Would affect any version from v0.10.0 onward (the daemon coordination layer). Not currently installed — cannot run due to this issue.

Platform

Linux (x64): Docker Desktop 29.x on Windows 11 WSL2, container running Debian 13 (trixie), kernel 6.18.35.2-microsoft-standard-WSL2

Install channel

Binary (attempted previously with v0.10.6; same issue expected on v0.10.8)

What happened, and what did you expect?

Every cbm process exits immediately with secure CLI coordination could not be created (endpoint). The ancestor walk rejects every possible directory path because no ancestor directory from /home/kirocrew to / is owned by uid 0 or uid 1000 — they are all owned by uid 65534 (nobody).

Expected: cbm should be able to run in a standard Docker container that uses user namespace remapping (a single-uid mapping), which is a common production Docker security pattern.

Root cause

Docker's userns-remap feature (or equivalently, Podman's rootless mode) creates a user namespace where only a single uid is mapped:

# /proc/self/uid_map
      1000       1000          1

This means:

  • The process runs as uid 1000 (kirocrew)
  • uid 0 does not exist inside the container namespace
  • All filesystem objects owned by uids outside the mapped range appear as nobody (65534) per kernel overflow behavior

The directory ownership chain:

mode=755  owner=65534:65534  /          ← nobody (unmapped host root)
mode=755  owner=65534:65534  /home      ← nobody
mode=700  owner=1000:1000    /home/kirocrew  ← ✅ current user
mode=1777 owner=65534:65534  /tmp       ← nobody (not uid 0!)

The ancestor walk (posix_directory_parent_secure) requires each ancestor to be owned by either:

  1. The current process euid (1000), OR
  2. uid 0 (root)

Neither condition is satisfiable for /, /home, or /tmp because:

  • uid 0 doesn't exist in the namespace (no mapping)
  • These directories show owner uid 65534 (the kernel's overflow uid for unmapped uids)

The /tmp sticky-bit special case also fails: it requires root-owned (uid 0) + sticky, but /tmp is uid 65534 + sticky.

CBM_RUNTIME_DIR (added in v0.10.5 for #1672) cannot help because any path ultimately traverses / or /home, both of which are uid 65534.

Reproduction

  1. Run any Docker container with a single-uid user namespace mapping: 
    # docker-compose.yml or docker run equivalent
# The container uses userns-remap or is started with --userns=host
# with a USER directive that maps only one uid
  2. Verify the uid_map: cat /proc/self/uid_map → shows 1000 1000 1
  3. Verify / ownership: stat -c '%u' / → shows 65534
  4. Attempt to run any cbm command → fails at ancestor walk

Suggested fix

When the process detects it is running inside a user namespace with a restricted uid_map (single-uid or no uid-0 mapping), the ancestor walk should relax its ownership requirement for ancestors that show the overflow uid (65534/nobody). Specifically:

  1. Detect userns: read /proc/self/uid_map — if uid 0 is not mapped (or maps to an unprivileged host uid), the container has no real root.
  2. Relax ancestor check: in this case, accept ancestors owned by the overflow uid (65534) as equivalent to "owned by the host root that we cannot see". The security guarantee (no local user can swap a path component) is maintained because the container is already isolated — no other uid can exist to attack the path.
  3. Keep strict checks for the private directory itself (CBM_RUNTIME_DIR, CBM_CACHE_DIR) — these must still be owned by the process euid (1000).

Alternative: provide a CBM_SKIP_ANCESTOR_WALK=1 escape hatch for container environments where the operator accepts the risk, similar to how git provides safe.directory for ownership mismatches.

Related issues

Environment details

OS: Debian GNU/Linux 13 (trixie)
Kernel: 6.18.35.2-microsoft-standard-WSL2
Container runtime: Docker Desktop (overlay2)
uid_map: 1000 1000 1
gid_map: 1000 1000 1
Process: uid=1000(kirocrew) gid=1000(kirocrew)

Activity

  1. github-actions
    added
    windowsWindows-specific issues
    on Aug 25, 2026
  2. DeusData
    added
    bugSomething isn't working
    priority/highNeeds near-term maintainer attention; high-impact bug, regression, safety issue, or release blocker.
    securitySecurity vulnerabilities, hardening
    stability/performanceServer crashes, OOM, hangs, high CPU/memory
    and removed
    windowsWindows-specific issues
    on Sep 5, 2026
  3. DeusData
    added a commit that references this issue on Sep 9, 2026

    Merge pull request #2138 from DeusData/fix/decision-bd

    055fbb7

  4. DeusData commented on Sep 9, 2026

    @DeusData
    DeusData
    on Sep 9, 2026
    Owner

    Fixed on main by #2138 (merge 055fbb7). Inside a single-uid user namespace (unshare -U --map-current-user, podman --userns=keep-id) the root-owned ancestors (/, /tmp) appear owned by the kernel overflow uid; the daemon now tolerates an overflow-owned ANCESTOR only when /proc/self/uid_map is a single-uid map (one line, count==1, inside==euid) and /proc/sys/kernel/overflowuid is readable, with no env hatch. The directories that hold data stay strict == geteuid(), so a host-squatted overflow-owned private dir is still refused. Thanks for the report.

  5. DeusData
    closed this as completedon Sep 9, 2026
  6. alcidesmorales
    added a commit that references this issue on Sep 9, 2026

    fix(daemon): accept single-uid userns overflow-owned ancestors (DeusD…

    8116672
  7. DeusData
    added 4 commits that reference this issue on Sep 13, 2026

    fix(test,daemon): make the userns probe revert-sensitive and stop a f…

    f9038b8

    fix(daemon,ci): stop caching the overflow uid, and run the userns gua…

    d21e6cc

    ci(test): report whether unprivileged userns is actually available

    ac8f5b8

    Merge pull request #2195 from DeusData/fix/userns-smoke-spawn

    339b3f4

  8. DeusData commented on Sep 13, 2026

    @DeusData
    DeusData
    on Sep 13, 2026
    Owner

    Follow-up landed in 339b3f40 (PR #2195) — this issue stays closed, but the guard behind it was not doing its job and now is.

    The smoke test could never have passed. It forked a child, entered a user namespace, and called cbm_daemon_ipc_private_directory_secure() directly in that forked child. But the overflow uid was memoised per process via pthread_once, and unshare(CLONE_NEWUSER) is exactly what rewrites /proc/self/uid_map — while pthread_once state survives fork() already marked done. Seven earlier call sites primed the cache, so the child inherited the host answer and refused. The test reported that as a product refusal while the product was correct.

    Three things changed:

    1. The probe now re-execs into a fresh process, so the decision is made by a process that started in the namespace — the only shape production ever takes.
    2. The cache is gone. It was safe only by accident: every caller happens to run in a freshly exec'd process. That made a security decision depend on an invariant nothing enforced, behind a comment claiming /proc state was "immutable". A new test counts derivations and fails by name if a cache ever returns.
    3. The guard now runs on every PR. Its only real-namespace coverage was the ubuntu-22.04 legs, which are broad-matrix only — so it executed in no pull request, on images retiring 2027-04-17, after which it would have become a permanent silent skip. A sysctl on the ubuntu-24.04 legs lifts the AppArmor restriction; CI now prints CBM_USERNS_AVAILABLE=yes as evidence rather than leaving it to inference.

    The tolerance itself is unchanged and still correct. Its justification was overstated, though: the overflow uid is what every unmapped host uid maps to, not only root, so "overflow-owned" does not prove "root-created". The comment now states the real, narrower guarantee — no principal reachable from inside the namespace can create or mutate such an ancestor — and notes the bound: a hostile host-side ancestor owner reaches denial of service and socket-path control, never the leaf, which stays 0700 and euid-rechecked. Narrowing the tolerance was considered and rejected because legitimate root-owned 01777 /tmp is accepted through the same branch, so it would have reverted this fix in practice.

    Verified in a real namespace as a non-root uid: with the tolerance stubbed off the test goes red, with it restored it passes — so the test genuinely binds this decision, not merely the test change.

  9. pull
    added a commit that references this issue on Sep 13, 2026

    fix(test): exec the DeusData#1830 userns probe instead of forking a p…

    469c3dd
  10. daidaiJ
    added 7 commits that reference this issue on Sep 29, 2026

    fix(daemon): accept single-uid userns overflow-owned ancestors (DeusD…

    35fc591

    fix(test): exec the DeusData#1830 userns probe instead of forking a p…

    915cab5

    fix(test,daemon): make the userns probe revert-sensitive and stop a f…

    ecc4969

    fix(daemon,ci): stop caching the overflow uid, and run the userns gua…

    5ea0489

    fix(test): exec the DeusData#1830 userns probe instead of forking a p…

    53f0e84

    fix(test,daemon): make the userns probe revert-sensitive and stop a f…

    842de5d

    fix(daemon,ci): stop caching the overflow uid, and run the userns gua…

    80555e3
  11. DeusData
    added a commit that references this issue on Sep 30, 2026

    fix(daemon): explain foreign-owned cache ancestors and warn on networ…

    6eb3353
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingpriority/highNeeds near-term maintainer attention; high-impact bug, regression, safety issue, or release blocker.securitySecurity vulnerabilities, hardeningstability/performanceServer crashes, OOM, hangs, high CPU/memory

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions