Repository navigation
Toggling MIG mode fails when nvidia_drm module is loaded #181
Description
mig-parted will fail to perform a GPU reset, and therefore toggle the MIG mode on GPUs where a reset is required,
if the nvidia_drm kernel module is loaded. Below is the error message one might encounter:
$ sudo nvidia-mig-parted apply -f /etc/nvidia-mig-manager/config-default.yaml -k /etc/nvidia-mig-manager/hooks-default.yaml -c all-balanced
ERRO[0003]
The following GPUs could not be reset:
GPU 00000000:01:00.0: In use by another client
GPU 00000000:47:00.0: In use by another client
GPU 00000000:81:00.0: In use by another client
GPU 00000000:C2:00.0: In use by another client
4 devices are currently being used by one or more other processes (e.g., Fabric Manager, CUDA application, graphics application such as an X server, or a monitoring application such as another instance of nvidia-smi). Please first kill all processes using these devices and all compute applications running in the system.
FATA[0008] Error applying MIG configuration with hooks: error resetting all GPUs: exit status 255
A workaround is to unload the nvidia_drm kernel module before applying a MIG configuration and load it after the configuration change has been applied.
$ sudo rmmod nvidia_drm
$ sudo nvidia-mig-parted apply -f /etc/nvidia-mig-manager/config-default.yaml -k /etc/nvidia-mig-manager/hooks-default.yaml -c all-balanced
MIG configuration applied successfully
Activity
bergentruckung commentedon Oct 14, 2025
I'm seeing a similar behaviour on my GPUs as well. Few questions:
nvidia_drmkernel module, since that's for rendering?5-Jeremy commentedon Nov 18, 2025
I found that I needed to go all the way and restart the nvidia module itself before I could change the MIG mode after encountering this error.
(For reference, I did this by running
sudo rmmod nvidia_drm,sudo rmmod nvidia_modeset,sudo rmmod nvidia_uvm, and finallysudo rmmod nvidia; then I re-loaded the modules in reverse order)rajathagasthya commentedon Apr 6, 2026
I'm closing this since this documented as a known issue: https://github.com/NVIDIA/mig-parted?tab=readme-ov-file#known-issues