Skip to content

Dockerd rootless: make {/etc,/var/run}/cdi available#48541

Merged
thaJeztah merged 1 commit intomoby:masterfrom
ereslibre:cdi-rootless
Dec 4, 2024
Merged

Dockerd rootless: make {/etc,/var/run}/cdi available#48541
thaJeztah merged 1 commit intomoby:masterfrom
ereslibre:cdi-rootless

Conversation

@ereslibre
Copy link
Copy Markdown
Contributor

When dockerd is executed with the dockerd-rootless.sh script, make /etc/cdi and /var/run/cdi available to the daemon if they exist.

This makes it possible to enable the CDI integration in rootless mode.

Fixes: #47676

Signed-off-by: Rafael Fernández López [email protected]

- What I did

When Dockerd is executed in rootless mode, the Container Device Interface won't be able to find the CDI specs at /etc/cdi nor /var/run/cdi.

The implementation of the CDI code can be found at https://github.com/cncf-tags/container-device-interface. I am focusing on the latest tagged version (v0.8.0).

This implementation uses filepath.Walk (https://pkg.go.dev/path/[email protected]#Walk), which documents not following symlinks.

- How I did it

I modified the contrib/dockerd-rootless.sh script so that the contents of /etc/cdi and /var/run/cdi are made available to the Dockerd daemon, which in turn uses the CDI code to load and interpret the CDI specs, allowing us to do something like the following:

❯ DOCKER_HOST=unix:///run/user/1000/docker.sock docker run --rm --device=nvidia.com/gpu=all -it ubuntu:latest nvidia-smi -L
GPU 0: NVIDIA GeForce RTX 4090 (UUID: GPU-c475e08b-0cc5-f5aa-4326-99699429b449)
GPU 1: NVIDIA GeForce RTX 2080 SUPER (UUID: GPU-5cca1a6f-7cee-b649-40f0-2d3ecb0aa207)

- How to verify it

Note that you will need a valid /etc/cdi or /var/run/cdi definition of some hardware that follows the CDI specification, so that the Docker daemon can load it and make it available to you.

You can use the following script if you have nix installed:

#!/usr/bin/env -S nix shell github:ereslibre/nixities#rootlesskit github:ereslibre/nixities#slirp4netns github:ereslibre/nixities#containerd github:ereslibre/nixities#runc github:ereslibre/nixities#bash -c bash
TMPFILE=$(mktemp '/tmp/dockerd.XXXXXXXXXXXX') || exit 1
cat <<EOF > $TMPFILE
{
  "cdi-spec-dirs": [
    "/etc/cdi/",
    "/var/run/cdi/"
  ],
  "features": {
    "cdi": true
  },
  "userland-proxy": false,
  "rootless": true
}
EOF
DOCKERD=<PATH_TO_MOBY>/moby/moby/bundles/binary/dockerd <PATH_TO_MOBY>/moby/moby/contrib/dockerd-rootless.sh --config-file=$TMPFILE

- Description for the changelog

Dockerd rootless mode loads /etc/cdi and /var/run/cdi as expected by the Container Device Interface (CDI) integration.

- A picture of a cute animal (not mandatory but encouraged)

A cute Vicuña. This is NixOS 24.11 release codename :)

@ereslibre
Copy link
Copy Markdown
Contributor Author

cc/ @elezar (due to #47676)

Comment thread contrib/dockerd-rootless.sh Outdated
@ereslibre ereslibre force-pushed the cdi-rootless branch 3 times, most recently from f2cfa80 to 02cd8d1 Compare September 24, 2024 16:02
Comment thread contrib/dockerd-rootless.sh Outdated
Comment thread contrib/dockerd-rootless.sh Outdated
@elezar
Copy link
Copy Markdown
Contributor

elezar commented Oct 9, 2024

LGTM from a CDI perspective.

Comment thread contrib/dockerd-rootless.sh Outdated
Comment thread contrib/dockerd-rootless.sh Outdated
@stevapple
Copy link
Copy Markdown
Contributor

We have already been applying this patch in production and it saves our day. Is there any possibility that we can merge this PR into the upcoming Docker release so no more patch is required for GPUs in Rootless Docker?

@thaJeztah
Copy link
Copy Markdown
Member

did a quick rebase to have a fresh run of CI

Comment thread contrib/dockerd-rootless.sh Outdated
Copy link
Copy Markdown
Member

@thaJeztah thaJeztah left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@tianon @cpuguy83 want to have a look as well if there's any thing I'm overlooking?

@thaJeztah thaJeztah added this to the 28.0.0 milestone Dec 3, 2024
Comment thread contrib/dockerd-rootless.sh Outdated
Copy link
Copy Markdown
Member

@cpuguy83 cpuguy83 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor change, but otherwise LGTM

Comment thread contrib/dockerd-rootless.sh Outdated
When dockerd is executed with the `dockerd-rootless.sh` script, make
/etc/cdi and /var/run/cdi available to the daemon if they exist.

This makes it possible to enable the CDI integration in rootless mode.

Fixes: moby#47676

Signed-off-by: Rafael Fernández López <[email protected]>
@thaJeztah thaJeztah merged commit 05e91b6 into moby:master Dec 4, 2024
@stevapple
Copy link
Copy Markdown
Contributor

@ereslibre Can you create backports of this PR to 25.x through 27.x? I'm not sure if 25.x backport will actually help anyone, but a 27.x one would be sincerely appreciated.

@ereslibre
Copy link
Copy Markdown
Contributor Author

@ereslibre Can you create backports of this PR to 25.x through 27.x? I'm not sure if 25.x backport will actually help anyone, but a 27.x one would be sincerely appreciated.

@stevapple Yes, I will do that later today!

@thaJeztah
Copy link
Copy Markdown
Member

I was working on some backports, so opened cherry-picks for those branches 👍

@ereslibre
Copy link
Copy Markdown
Contributor Author

I was working on some backports, so opened cherry-picks for those branches 👍

Thanks @thaJeztah!

# mode; otherwise the Docker daemon won't have access to the CDI
# specification files.
mount_directory /etc/cdi
mount_directory /var/run/cdi
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there any reason that we don't scan $XDG_CONFIG_HOME/cdi and XDG_RUNTIME_DIR/cdi ?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No specific reason from my end

# mode; otherwise the Docker daemon won't have access to the CDI
# specification files.
mount_directory /etc/cdi
mount_directory /var/run/cdi
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These mounts are probably not necessary if we can let dockerd ignore EACCES ?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They would not be necessary in that case. However, I wasn’t sure about the implications so I went with the safest approach.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Default Nvidia CDI spec location on rootless kit seems to be unaccessible

6 participants