Repository navigation
feat: glibc extension - #473
Merged
Merged
Conversation
Member
This is really cool and looking forward to actually not having to maintain patches or wrappers. What i still don't understand is about kernel modules, would SideroLabs be still shipping them as extensions (I believe that's the case since only machined in talos can load modules)? |
frezbo
reviewed
Sep 18, 2024
Contributor
Author
I am sending more PRs, but this is one of the major changes from an architecture POV: the gpu-operator would be allowed to load and unload kernel modules, which means enabling module unloading in the kernel (see siderolabs/pkgs#1031) and not removing SYS_MODULE from containers (see siderolabs/talos#9339). Another important note is that only CDI mode (see https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/cdi.html) works on Talos with this patch. The "legacy" runtime hook require more libraries to be present on the system, whereas the CDI hook is a pure Go program that only requires the glibc dynamic loader and |
Member
We're open to moving to using CDI 👍 |
Contributor
Author
|
See siderolabs/talos#9339 for main discussion on loading kernel modules. |
jfroy
force-pushed
the
glibc
branch
3 times, most recently
from
September 21, 2024 06:21
d94cf22 to
b4de00e
Compare
Contributor
Author
|
Patch has been updated to rework the glibc subtree to look like a merged |
frezbo
reviewed
Sep 21, 2024
frezbo
reviewed
Sep 21, 2024
jfroy
added a commit
to jfroy/siderolabs-extensions
that referenced
this pull request
Sep 23, 2024
This patch deprecates the NVIDIA toolkit extension and introduces a new nvidia-driver extension (in production/lts versions and open source/proprietary flavors). The NVIDIA container toolkit must be installed independently, via a future Talos extension, the NVIDIA GPU Operator, or by the cluster administator. The extension depends on the new glibc extension (siderolabs#473) and participates in its filesystem subroot by installing all the NVIDIA components in it. Finally, the extension runs a service that will bind mount this glibc subroot at `/run/nvidia/driver` and run the `nvidia-persistenced` daemon. This careful setup allows the NVIDIA GPU Operator to utilize this extension as if it were a traditional NVIDIA driver container. Signed-off-by: Jean-Francois Roy <[email protected]>
Member
|
I'm going to cut v1.9.0-alpha.0 of Talos, bump the extension validator, and then get back to this PR, thank you for your patience! |
Contributor
Author
No rush, thank you for considering these PRs! |
frezbo
approved these changes
Oct 18, 2024
The extension is mostly copied from the package that is in the nvidia toolkit extensions, but with a few notable changes. - A symbolic link to `ldconfig` is installed at `/sbin/ldconfig`. This allows the nvidia gpu-operator to work without modification. A patch to the extension validation logic is required to allow the new path. - The `/usr/local/glibc` subtree is now structured as a [merged `/usr` root](https://systemd.io/THE_CASE_FOR_THE_USR_MERGE/). This improves compatbility with ld-linux.so.2 for library discovery. Signed-off-by: Jean-Francois Roy <[email protected]> Signed-off-by: Noel Georgi <[email protected]>
Member
|
@jfroy is this good to go, I see the referenced PR for nvidia-container-toolkit is still not merged 🤔 |
Contributor
Author
Yeah I'm working on that internally. The team is just very busy with other work. We are focusing in particular on CDI and DRA. Sidero could package the toolkit in an extension -- the wrappers are created by an installer component that is invoked by the operator and are not an intrinsic part of the toolkit. In any case, this patch is needed for the CDI hook to run, so it's good to pick up no matter what. |
Member
|
/m |
Member
|
I'll wait over the weekend and see if our daily runs for nvidia tests works with this change |
jfroy
added a commit
to jfroy/siderolabs-extensions
that referenced
this pull request
May 27, 2025
This patch deprecates the NVIDIA toolkit extension and introduces a new nvidia-driver extension (in production/lts versions and open source/proprietary flavors). The NVIDIA container toolkit must be installed independently, via a future Talos extension, the NVIDIA GPU Operator, or by the cluster administator. The extension depends on the new glibc extension (siderolabs#473) and participates in its filesystem subroot by installing all the NVIDIA components in it. Finally, the extension runs a service that will bind mount this glibc subroot at `/run/nvidia/driver` and run the `nvidia-persistenced` daemon. This careful setup allows the NVIDIA GPU Operator to utilize this extension as if it were a traditional NVIDIA driver container. Signed-off-by: Jean-Francois Roy <[email protected]>
jfroy
added a commit
to jfroy/siderolabs-extensions
that referenced
this pull request
May 27, 2025
This patch deprecates the NVIDIA toolkit extension and introduces a new nvidia-driver extension (in production/lts versions and open source/proprietary flavors). The NVIDIA container toolkit must be installed independently, via a future Talos extension, the NVIDIA GPU Operator, or by the cluster administator. The extension depends on the new glibc extension (siderolabs#473) and participates in its filesystem subroot by installing all the NVIDIA components in it. Finally, the extension runs a service that will bind mount this glibc subroot at `/run/nvidia/driver` and run the `nvidia-persistenced` daemon. This careful setup allows the NVIDIA GPU Operator to utilize this extension as if it were a traditional NVIDIA driver container. Signed-off-by: Jean-Francois Roy <[email protected]>
jfroy
added a commit
to jfroy/siderolabs-extensions
that referenced
this pull request
Jun 23, 2025
This patch deprecates the NVIDIA toolkit extension and introduces a new nvidia-driver extension (in production/lts versions and open source/proprietary flavors). The NVIDIA container toolkit must be installed independently, via a future Talos extension, the NVIDIA GPU Operator, or by the cluster administator. The extension depends on the new glibc extension (siderolabs#473) and participates in its filesystem subroot by installing all the NVIDIA components in it. Finally, the extension runs a service that will bind mount this glibc subroot at `/run/nvidia/driver` and run the `nvidia-persistenced` daemon. This careful setup allows the NVIDIA GPU Operator to utilize this extension as if it were a traditional NVIDIA driver container. Signed-off-by: Jean-Francois Roy <[email protected]>
jfroy
added a commit
to jfroy/siderolabs-extensions
that referenced
this pull request
Jun 23, 2025
This patch deprecates the NVIDIA toolkit extension and introduces a new nvidia-driver extension (in production/lts versions and open source/proprietary flavors). The NVIDIA container toolkit must be installed independently, via a future Talos extension, the NVIDIA GPU Operator, or by the cluster administator. The extension depends on the new glibc extension (siderolabs#473) and participates in its filesystem subroot by installing all the NVIDIA components in it. Finally, the extension runs a service that will bind mount this glibc subroot at `/run/nvidia/driver` and run the `nvidia-persistenced` daemon. This careful setup allows the NVIDIA GPU Operator to utilize this extension as if it were a traditional NVIDIA driver container. Signed-off-by: Jean-Francois Roy <[email protected]>
jfroy
added a commit
to jfroy/siderolabs-extensions
that referenced
this pull request
Jun 23, 2025
This patch deprecates the NVIDIA toolkit extension and introduces a new nvidia-driver extension (in production/lts versions and open source/proprietary flavors). The NVIDIA container toolkit must be installed independently, via a future Talos extension, the NVIDIA GPU Operator, or by the cluster administator. The extension depends on the new glibc extension (siderolabs#473) and participates in its filesystem subroot by installing all the NVIDIA components in it. Finally, the extension runs a service that will bind mount this glibc subroot at `/run/nvidia/driver` and run the `nvidia-persistenced` daemon. This careful setup allows the NVIDIA GPU Operator to utilize this extension as if it were a traditional NVIDIA driver container. Signed-off-by: Jean-Francois Roy <[email protected]>
jfroy
added a commit
to jfroy/siderolabs-extensions
that referenced
this pull request
Jun 23, 2025
This patch deprecates the NVIDIA toolkit extension and introduces a new nvidia-driver extension (in production/lts versions and open source/proprietary flavors). The NVIDIA container toolkit must be installed independently, via a future Talos extension, the NVIDIA GPU Operator, or by the cluster administator. The extension depends on the new glibc extension (siderolabs#473) and participates in its filesystem subroot by installing all the NVIDIA components in it. Finally, the extension runs a service that will bind mount this glibc subroot at `/run/nvidia/driver` and run the `nvidia-persistenced` daemon. This careful setup allows the NVIDIA GPU Operator to utilize this extension as if it were a traditional NVIDIA driver container. Signed-off-by: Jean-Francois Roy <[email protected]>
jfroy
added a commit
to jfroy/siderolabs-extensions
that referenced
this pull request
Sep 10, 2025
This patch deprecates the NVIDIA toolkit extension and introduces a new nvidia-driver extension (in production/lts versions and open source/proprietary flavors). The NVIDIA container toolkit must be installed independently, via a future Talos extension, the NVIDIA GPU Operator, or by the cluster administator. The extension depends on the new glibc extension (siderolabs#473) and participates in its filesystem subroot by installing all the NVIDIA components in it. Finally, the extension runs a service that will bind mount this glibc subroot at `/run/nvidia/driver` and run the `nvidia-persistenced` daemon. This careful setup allows the NVIDIA GPU Operator to utilize this extension as if it were a traditional NVIDIA driver container. Signed-off-by: Jean-Francois Roy <[email protected]>
jfroy
added a commit
to jfroy/siderolabs-extensions
that referenced
this pull request
Sep 10, 2025
This patch deprecates the NVIDIA toolkit extension and introduces a new nvidia-driver extension (in production/lts versions and open source/proprietary flavors). The NVIDIA container toolkit must be installed independently, via a future Talos extension, the NVIDIA GPU Operator, or by the cluster administator. The extension depends on the new glibc extension (siderolabs#473) and participates in its filesystem subroot by installing all the NVIDIA components in it. Finally, the extension runs a service that will bind mount this glibc subroot at `/run/nvidia/driver` and run the `nvidia-persistenced` daemon. This careful setup allows the NVIDIA GPU Operator to utilize this extension as if it were a traditional NVIDIA driver container. Signed-off-by: Jean-Francois Roy <[email protected]>
jfroy
added a commit
to jfroy/siderolabs-extensions
that referenced
this pull request
Sep 11, 2025
This patch deprecates the NVIDIA toolkit extension and introduces a new nvidia-driver extension (in production/lts versions and open source/proprietary flavors). The NVIDIA container toolkit must be installed independently, via a future Talos extension, the NVIDIA GPU Operator, or by the cluster administator. The extension depends on the new glibc extension (siderolabs#473) and participates in its filesystem subroot by installing all the NVIDIA components in it. Finally, the extension runs a service that will bind mount this glibc subroot at `/run/nvidia/driver` and run the `nvidia-persistenced` daemon. This careful setup allows the NVIDIA GPU Operator to utilize this extension as if it were a traditional NVIDIA driver container. Signed-off-by: Jean-Francois Roy <[email protected]>
jfroy
added a commit
to jfroy/siderolabs-extensions
that referenced
this pull request
Nov 8, 2025
This patch deprecates the NVIDIA toolkit extension and introduces a new nvidia-driver extension (in production/lts versions and open source/proprietary flavors). The NVIDIA container toolkit must be installed independently, via a future Talos extension, the NVIDIA GPU Operator, or by the cluster administator. The extension depends on the new glibc extension (siderolabs#473) and participates in its filesystem subroot by installing all the NVIDIA components in it. Finally, the extension runs a service that will bind mount this glibc subroot at `/run/nvidia/driver` and run the `nvidia-persistenced` daemon. This careful setup allows the NVIDIA GPU Operator to utilize this extension as if it were a traditional NVIDIA driver container. Signed-off-by: Jean-Francois Roy <[email protected]>
jfroy
added a commit
to jfroy/siderolabs-extensions
that referenced
this pull request
Nov 11, 2025
This patch deprecates the NVIDIA toolkit extension and introduces a new nvidia-driver extension (in production/lts versions and open source/proprietary flavors). The NVIDIA container toolkit must be installed independently, via a future Talos extension, the NVIDIA GPU Operator, or by the cluster administator. The extension depends on the new glibc extension (siderolabs#473) and participates in its filesystem subroot by installing all the NVIDIA components in it. Finally, the extension runs a service that will bind mount this glibc subroot at `/run/nvidia/driver` and run the `nvidia-persistenced` daemon. This careful setup allows the NVIDIA GPU Operator to utilize this extension as if it were a traditional NVIDIA driver container. Signed-off-by: Jean-Francois Roy <[email protected]>
jfroy
added a commit
to jfroy/siderolabs-extensions
that referenced
this pull request
Dec 10, 2025
This patch deprecates the NVIDIA toolkit extension and introduces a new nvidia-driver extension (in production/lts versions and open source/proprietary flavors). The NVIDIA container toolkit must be installed independently, via a future Talos extension, the NVIDIA GPU Operator, or by the cluster administator. The extension depends on the new glibc extension (siderolabs#473) and participates in its filesystem subroot by installing all the NVIDIA components in it. Finally, the extension runs a service that will bind mount this glibc subroot at `/run/nvidia/driver` and run the `nvidia-persistenced` daemon. This careful setup allows the NVIDIA GPU Operator to utilize this extension as if it were a traditional NVIDIA driver container. Signed-off-by: Jean-Francois Roy <[email protected]>
jfroy
added a commit
to jfroy/siderolabs-extensions
that referenced
this pull request
Dec 17, 2025
This patch deprecates the NVIDIA toolkit extension and introduces a new nvidia-driver extension (in production/lts versions and open source/proprietary flavors). The NVIDIA container toolkit must be installed independently, via a future Talos extension, the NVIDIA GPU Operator, or by the cluster administator. The extension depends on the new glibc extension (siderolabs#473) and participates in its filesystem subroot by installing all the NVIDIA components in it. Finally, the extension runs a service that will bind mount this glibc subroot at `/run/nvidia/driver` and run the `nvidia-persistenced` daemon. This careful setup allows the NVIDIA GPU Operator to utilize this extension as if it were a traditional NVIDIA driver container. Signed-off-by: Jean-Francois Roy <[email protected]>
jfroy
added a commit
to jfroy/siderolabs-extensions
that referenced
this pull request
Dec 22, 2025
This patch deprecates the NVIDIA toolkit extension and introduces a new nvidia-driver extension (in production/lts versions and open source/proprietary flavors). The NVIDIA container toolkit must be installed independently, via a future Talos extension, the NVIDIA GPU Operator, or by the cluster administator. The extension depends on the new glibc extension (siderolabs#473) and participates in its filesystem subroot by installing all the NVIDIA components in it. Finally, the extension runs a service that will bind mount this glibc subroot at `/run/nvidia/driver` and run the `nvidia-persistenced` daemon. This careful setup allows the NVIDIA GPU Operator to utilize this extension as if it were a traditional NVIDIA driver container. Signed-off-by: Jean-Francois Roy <[email protected]>
jfroy
added a commit
to jfroy/siderolabs-extensions
that referenced
this pull request
Jan 5, 2026
This patch deprecates the NVIDIA toolkit extension and introduces a new nvidia-driver extension (in production/lts versions and open source/proprietary flavors). The NVIDIA container toolkit must be installed independently, via a future Talos extension, the NVIDIA GPU Operator, or by the cluster administator. The extension depends on the new glibc extension (siderolabs#473) and participates in its filesystem subroot by installing all the NVIDIA components in it. Finally, the extension runs a service that will bind mount this glibc subroot at `/run/nvidia/driver` and run the `nvidia-persistenced` daemon. This careful setup allows the NVIDIA GPU Operator to utilize this extension as if it were a traditional NVIDIA driver container. Signed-off-by: Jean-Francois Roy <[email protected]>
jfroy
added a commit
to jfroy/siderolabs-extensions
that referenced
this pull request
Feb 2, 2026
This patch deprecates the NVIDIA toolkit extension and introduces a new nvidia-driver extension (in production/lts versions and open source/proprietary flavors). The NVIDIA container toolkit must be installed independently, via a future Talos extension, the NVIDIA GPU Operator, or by the cluster administator. The extension depends on the new glibc extension (siderolabs#473) and participates in its filesystem subroot by installing all the NVIDIA components in it. Finally, the extension runs a service that will bind mount this glibc subroot at `/run/nvidia/driver` and run the `nvidia-persistenced` daemon. This careful setup allows the NVIDIA GPU Operator to utilize this extension as if it were a traditional NVIDIA driver container. Signed-off-by: Jean-Francois Roy <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds a glibc extension. The intention is to replace the nvidia extensions entirely and only provide the glibc components required by the nvidia gpu-operator and its components (e.g. the nvidia container toolkit).
The extension is pretty much copied from the package that is in the nvidia extensions, with one major modification: a symbolic link to ldconfig is installed at
/sbin/ldconfig. This change allows the nvidia gpu-operator to work without modification*. This change does require a patch to the extension validation logic, which is provided in separate PRs.