You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
container-toolkit fails to start after upgrading to v24.9.0 on k3s cluster #1109
This is effectively a continuation of #1099, but I cannot re-open that issue, so opening a new one.
I am experiencing the same problem while attempting to to upgrade from v24.6.0 to v24.9.0 on a k3s cluster. Perhaps a bad interaction related to this recent commit and the non-standard CONTAINERD paths required for gpu-operator+k3s, specified in my cluster's values as:
It makes sense that the command it is trying to run fails, there is no containerd binary on the host, so chroot /host containerd config dump is expected to fail on a k3s cluster.
Hi @logan2211 thanks for reporting this issue. It is on our radar and we are working on getting a fix out for this. We recently switched to fetching the currently applied container runtime configuration via CLI (e.g. containerd config dump) rather than from a file (see NVIDIA/nvidia-container-toolkit@f477dc0) . This appears to have broke systems where the CLI binary is not in the PATH, like k3s. We are working on using the TOML file as a fallback option in case the CLI binary cannot be found: NVIDIA/nvidia-container-toolkit#777
Thanks, we are trying to upgrade the cluster urgently due to the CVEs. I suppose one possible workaround may be to downgrade gpu-operator to v24.6.2 and override the driver version to 550.127.05?
edit: after testing, proposed workaround downgrading gpu-operator and pinning the driver in values seems to work fine, just noting for anyone else experiencing this issue.
downgrade gpu-operator to v24.6.2 and override the driver version to 550.127.05?
This should work. Or alternatively you can stick to GPU Operator v24.9.0 and downgrade the NVIDIA Container Toolkit to 1.16.2, which does not contain this change.
This issue is stale because it has been open 90 days with no activity. This issue will be closed in 30 days unless new comments are made or the stale label is removed. To skip these checks, apply the "lifecycle/frozen" label.
Activity
logan2211 commentedon Nov 7, 2024
It makes sense that the command it is trying to run fails, there is no
containerdbinary on the host, sochroot /host containerd config dumpis expected to fail on a k3s cluster.logan2211 commentedon Nov 7, 2024
Looks like the fix is in progress here NVIDIA/nvidia-container-toolkit#777
cdesiniotis commentedon Nov 7, 2024
Hi @logan2211 thanks for reporting this issue. It is on our radar and we are working on getting a fix out for this. We recently switched to fetching the currently applied container runtime configuration via CLI (e.g.
containerd config dump) rather than from a file (see NVIDIA/nvidia-container-toolkit@f477dc0) . This appears to have broke systems where the CLI binary is not in the PATH, like k3s. We are working on using the TOML file as a fallback option in case the CLI binary cannot be found: NVIDIA/nvidia-container-toolkit#777logan2211 commentedon Nov 7, 2024
Thanks, we are trying to upgrade the cluster urgently due to the CVEs. I suppose one possible workaround may be to downgrade gpu-operator to v24.6.2 and override the driver version to 550.127.05?
edit: after testing, proposed workaround downgrading gpu-operator and pinning the driver in values seems to work fine, just noting for anyone else experiencing this issue.
cdesiniotis commentedon Nov 7, 2024
This should work. Or alternatively you can stick to GPU Operator v24.9.0 and downgrade the NVIDIA Container Toolkit to 1.16.2, which does not contain this change.
cdesiniotis commentedon Nov 14, 2024
NVIDIA Container Toolkit 1.17.1 is now available and contains a fix for this issue: https://github.com/NVIDIA/nvidia-container-toolkit/releases/tag/v1.17.1
I would recommend overriding the NVIDIA Container Toolkit version to 1.17.1 by configuring
toolkit.versionin Clusterpolicy.elpsyr commentedon Nov 25, 2024
This approach can solve this problem :)
vi /usr/local/bin/containerd
sudo chmod +x /usr/local/bin/containerd
xrwang8 commentedon Dec 18, 2024
environment variables here do not need to be modified,need to modify a few parts.
github-actions commentedon Nov 4, 2025
This issue is stale because it has been open 90 days with no activity. This issue will be closed in 30 days unless new comments are made or the stale label is removed. To skip these checks, apply the "lifecycle/frozen" label.
rajathagasthya commentedon Nov 14, 2025
Closing this one since a fix was provided in Container Toolkit 1.17.1 (see comment).