Repository navigation
[BUG] deepmd-kit v3.2.0 installed with conda cannot freeze fine-tuned model checkpoint #6039
Description
Bug summary
Recently I installed deepmd-kit v3.2.0 via conda, using command
conda create -n deepmd deepmd-kit lammps horovod -c conda-forgeas indicated here in the documentation
However, when attempting to freeze a fine-tuned DPA-4 model checkpoint with:
dp --pt freeze -c model.ckpt.pt -o frozen Deepmd exited with the following error:
nohup: ignoring input
To get the best performance, it is recommended to adjust the number of threads by setting the environment variables OMP_NUM_THREADS, DP_INTRA_OP_PARALLELISM_THREADS, and DP_INTER_OP_PARALLELISM_THREADS. See https://deepmd.rtfd.io/parallelism/ for more information.
/tmp/tmp8hu578aw/cuda_utils.c:1:10: fatal error: cuda.h: No such file or directory
1 | #include "cuda.h"
| ^~~~~~~~
compilation terminated.
To get the best performance, it is recommended to adjust the number of threads by setting the environment variables OMP_NUM_THREADS, DP_INTRA_OP_PARALLELISM_THREADS, and DP_INTER_OP_PARALLELISM_THREADS. See https://deepmd.rtfd.io/parallelism/ for more information.
[2026-09-28 18:07:29,592] DEEPMD INFO DeePMD version: 3.2.0
[2026-09-28 18:07:29,707] DEEPMD INFO Set DP_TRITON_INFER to the desired level (0-3) before freezing; the selected Triton inference kernels are baked into the .pt2 archive.
[2026-09-28 18:07:30,454] DEEPMD INFO Tracing the lower graph on CPU (make_fx)...
[2026-09-28 18:07:38,483] DEEPMD INFO Exporting the traced graph (torch.export)...
[2026-09-28 18:07:53,617] DEEPMD INFO Compiling the AOTInductor package for cuda:0 (the slowest freeze stage; typically several minutes)...
/tmp/tmpo81yk0bq/cuda_utils.c:1:10: fatal error: cuda.h: No such file or directory
1 | #include "cuda.h"
| ^~~~~~~~
compilation terminated.
/tmp/tmp90_pyk_k/cuda_utils.c:1:10: fatal error: cuda.h: No such file or directory
1 | #include "cuda.h"
| ^~~~~~~~
compilation terminated.
/tmp/tmpdsoghzeu/cuda_utils.c:1:10: fatal error: cuda.h: No such file or directory
1 | #include "cuda.h"
| ^~~~~~~~
compilation terminated.
/tmp/tmp94v08ynk/cuda_utils.c:1:10: fatal error: cuda.h: No such file or directory
1 | #include "cuda.h"
| ^~~~~~~~
compilation terminated.
/tmp/tmpup1pulgk/cuda_utils.c:1:10: fatal error: cuda.h: No such file or directory
1 | #include "cuda.h"
| ^~~~~~~~
compilation terminated.
/tmp/tmpop58178f/cuda_utils.c:1:10: fatal error: cuda.h: No such file or directory
1 | #include "cuda.h"
| ^~~~~~~~
compilation terminated.
/tmp/tmpwo0riq74/cuda_utils.c:1:10: fatal error: cuda.h: No such file or directory
1 | #include "cuda.h"
| ^~~~~~~~
compilation terminated.
/tmp/tmp8hhnpogd/cuda_utils.c:1:10: fatal error: cuda.h: No such file or directory
1 | #include "cuda.h"
| ^~~~~~~~
compilation terminated.
/tmp/tmpy8fifl5b/cuda_utils.c:1:10: fatal error: cuda.h: No such file or directory
1 | #include "cuda.h"
| ^~~~~~~~
compilation terminated.
/tmp/tmpqxxsks77/cuda_utils.c:1:10: fatal error: cuda.h: No such file or directory
1 | #include "cuda.h"
| ^~~~~~~~
compilation terminated.
/tmp/tmp3_v4a414/cuda_utils.c:1:10: fatal error: cuda.h: No such file or directory
1 | #include "cuda.h"
| ^~~~~~~~
compilation terminated.
/tmp/tmphrjpqtv0/cuda_utils.c:1:10: fatal error: cuda.h: No such file or directory
1 | #include "cuda.h"
| ^~~~~~~~
compilation terminated.
/tmp/tmp0qroo9wt/cuda_utils.c:1:10: fatal error: cuda.h: No such file or directory
1 | #include "cuda.h"
| ^~~~~~~~
compilation terminated.
/tmp/tmp_d_z6j1k/cuda_utils.c:1:10: fatal error: cuda.h: No such file or directory
1 | #include "cuda.h"
| ^~~~~~~~
compilation terminated.
/tmp/tmpkzwgr00u/cuda_utils.c:1:10: fatal error: cuda.h: No such file or directory
1 | #include "cuda.h"
| ^~~~~~~~
compilation terminated.
/tmp/tmp30q69_ly/cuda_utils.c:1:10: fatal error: cuda.h: No such file or directory
1 | #include "cuda.h"
| ^~~~~~~~
compilation terminated.
/tmp/tmpxttfz19m/cuda_utils.c:1:10: fatal error: cuda.h: No such file or directory
1 | #include "cuda.h"
| ^~~~~~~~
compilation terminated.
/tmp/tmpel86l0vp/cuda_utils.c:1:10: fatal error: cuda.h: No such file or directory
1 | #include "cuda.h"
| ^~~~~~~~
compilation terminated.
/tmp/tmpdbcksoqr/cuda_utils.c:1:10: fatal error: cuda.h: No such file or directory
1 | #include "cuda.h"
| ^~~~~~~~
compilation terminated.
/tmp/tmpyjd7s046/cuda_utils.c:1:10: fatal error: cuda.h: No such file or directory
1 | #include "cuda.h"
| ^~~~~~~~
compilation terminated.
/tmp/tmpm1iruo51/cuda_utils.c:1:10: fatal error: cuda.h: No such file or directory
1 | #include "cuda.h"
| ^~~~~~~~
compilation terminated.
/tmp/tmp4ctd9i1x/cuda_utils.c:1:10: fatal error: cuda.h: No such file or directory
1 | #include "cuda.h"
| ^~~~~~~~
compilation terminated.
/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/copyreg.py:99: FutureWarning: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead.
return cls.__new__(cls, *args)
/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/torch/_inductor/compile_fx.py:320: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance.
warnings.warn(
W0928 18:08:40.499000 2911 site-packages/torch/_inductor/utils.py:1717] Not enough SMs to use max_autotune_gemm mode
/tmp/tmppguck85h/cuda_utils.c:1:10: fatal error: cuda.h: No such file or directory
1 | #include "cuda.h"
| ^~~~~~~~
compilation terminated.
Traceback (most recent call last):
File "/home/fengyu_xie/miniconda3/envs/deepmd4/bin/dp", line 8, in <module>
sys.exit(main())
^^^^^^
File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/deepmd/main.py", line 1158, in main
deepmd_main(args)
File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 367, in wrapper
return f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/deepmd/pt/entrypoints/main.py", line 694, in main
freeze(model=FLAGS.model, output=FLAGS.output, head=FLAGS.head)
File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/deepmd/pt/entrypoints/main.py", line 490, in freeze
freeze_sezm_to_pt2(model, out_pt2, head=head)
File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/deepmd/pt/entrypoints/freeze_pt2.py", line 961, in freeze_sezm_to_pt2
aoti_compile_and_package(exported, package_path=out_path_str)
File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/torch/_inductor/__init__.py", line 152, in aoti_compile_and_package
return aot_inductor_minifier_wrapper(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/torch/_inductor/debug.py", line 1344, in aot_inductor_minifier_wrapper
raise e
File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/torch/_inductor/debug.py", line 1314, in aot_inductor_minifier_wrapper
return func(
^^^^^
File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/torch/_inductor/__init__.py", line 195, in _aoti_compile_and_package_inner
aoti_files = aot_compile(gm, args, kwargs, options=inductor_configs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/torch/_inductor/__init__.py", line 311, in aot_compile
return compile_fx_aot(
^^^^^^^^^^^^^^^
File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/torch/_inductor/compile_fx.py", line 2086, in compile_fx_aot
compiled_artifacts = compile_fx(
^^^^^^^^^^^
File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/torch/_inductor/compile_fx.py", line 2675, in compile_fx
return compile_fx(
^^^^^^^^^^^
File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/torch/_inductor/compile_fx.py", line 2721, in compile_fx
return _maybe_wrap_and_compile_fx_main(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/torch/_inductor/compile_fx.py", line 2815, in _maybe_wrap_and_compile_fx_main
return _compile_fx_main(
^^^^^^^^^^^^^^^^^
File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/torch/_inductor/compile_fx.py", line 3001, in _compile_fx_main
return inference_compiler(unlifted_gm, example_inputs_)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/torch/_functorch/_aot_autograd/schemas.py", line 1421, in __call__
output_code = self.compiler_fn(gm, example_inputs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/torch/_inductor/compile_fx.py", line 2876, in fw_compiler_base
return compile_fx_forward(
^^^^^^^^^^^^^^^^^^^
File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/torch/_inductor/compile_fx.py", line 2504, in compile_fx_forward
result = inner_compile(
^^^^^^^^^^^^^^
File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/torch/_inductor/compile_fx.py", line 827, in compile_fx_inner
return wrap_compiler_debug(_compile_fx_inner, compiler_name="inductor")(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/torch/_dynamo/repro/after_aot.py", line 314, in debug_wrapper
inner_compiled_fn = compiler_fn(gm, example_inputs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/torch/_inductor/compile_fx.py", line 1038, in _compile_fx_inner
raise InductorError(e, currentframe()).with_traceback(
File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/torch/_inductor/compile_fx.py", line 1030, in _compile_fx_inner
mb_compiled_graph = fx_codegen_and_compile(
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/torch/_inductor/compile_fx.py", line 1836, in fx_codegen_and_compile
return scheme.codegen_and_compile(gm, example_inputs, inputs_to_check, graph_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/torch/_inductor/compile_fx.py", line 1556, in codegen_and_compile
wrapper_code, kernel_code = graph.codegen_with_cpp_wrapper()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/torch/_inductor/graph.py", line 2506, in codegen_with_cpp_wrapper
return self.codegen()
^^^^^^^^^^^^^^
File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/torch/_inductor/graph.py", line 2555, in codegen
self.scheduler.codegen()
File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/torch/_inductor/scheduler.py", line 7352, in codegen
else self._codegen(self.nodes)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/torch/_inductor/scheduler.py", line 7641, in _codegen
self.get_backend(device).codegen_node(node)
File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/torch/_inductor/codegen/cuda_combined_scheduling.py", line 152, in codegen_node
return self._triton_scheduling.codegen_node(node)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/torch/_inductor/codegen/simd.py", line 1864, in codegen_node
return self._codegen_nodes(nodes, coalesce_analysis) # type: ignore[arg-type]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/torch/_inductor/codegen/simd.py", line 1837, in _codegen_nodes
return self.codegen_node_schedule(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/torch/_inductor/codegen/simd.py", line 1961, in codegen_node_schedule
src_code = kernel.codegen_kernel()
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/torch/_inductor/codegen/triton.py", line 5760, in codegen_kernel
**self.inductor_meta_common(),
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/torch/_inductor/codegen/triton.py", line 5568, in inductor_meta_common
"backend_hash": torch.utils._triton.triton_hash_with_backend(),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/torch/utils/_triton.py", line 219, in triton_hash_with_backend
backend = triton_backend()
^^^^^^^^^^^^^^^^
File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/torch/utils/_triton.py", line 193, in triton_backend
target = driver.active.get_current_target()
^^^^^^^^^^^^^
File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/triton/runtime/driver.py", line 39, in active
self._active = self.default
^^^^^^^^^^^^
File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/triton/runtime/driver.py", line 33, in default
self._default = _create_driver()
^^^^^^^^^^^^^^^^
File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/triton/runtime/driver.py", line 21, in _create_driver
return active_drivers[0]()
^^^^^^^^^^^^^^^^^^^
File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/triton/backends/nvidia/driver.py", line 336, in __init__
self.utils = CudaUtils() # TODO: make static
^^^^^^^^^^^
File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/triton/backends/nvidia/driver.py", line 66, in __init__
mod = compile_module_from_src(
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/triton/runtime/build.py", line 115, in compile_module_from_src
so = _build(name, src_path, tmpdir, library_dirs or [], include_dirs or [], libraries or [], ccflags or [])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/triton/runtime/build.py", line 70, in _build
subprocess.check_call(cc_cmd, stdout=subprocess.DEVNULL)
File "/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/subprocess.py", line 413, in check_call
raise CalledProcessError(retcode, cmd)
torch._inductor.exc.InductorError: CalledProcessError: Command '['x86_64-conda-linux-gnu-cc', '/tmp/tmppguck85h/cuda_utils.c', '-O3', '-shared', '-fPIC', '-Wno-psabi', '-o', '/tmp/tmppguck85h/cuda_utils.cpython-312-x86_64-linux-gnu.so', '-l:libcuda.so.1', '-L/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/triton/backends/nvidia/lib', '-L/usr/lib/wsl/lib', '-I/home/fengyu_xie/miniconda3/envs/deepmd4/lib/python3.12/site-packages/triton/backends/nvidia/include', '-I/tmp/tmppguck85h', '-I/home/fengyu_xie/miniconda3/envs/deepmd4/include/python3.12', '-I/home/fengyu_xie/miniconda3/envs/deepmd4/targets/x86_64-linux/include']' returned non-zero exit status 1.
Similar issue never happend with Bohrium image "registry.dp.tech/dptech/deepmd-kit:3.2.0-sm89" on 4090_24g nodes
My intial guess is the conda package provided in the conda-forge channel might not have been bundled correctly. Specifically, some issue might exist with the packaged CUDA.
Could you please check this issue?
DeePMD-kit Version
3.2.0
Backend and its version
pytorch==2.12.0
How did you download the software?
conda
Input Files, Running Commands, Error Log, etc.
Steps to Reproduce
- Install deepmd-kit 3.2.0 via default conda command as specified in the documentation
- dp --pt freeze -c model.ckpt.pt -o frozen
Further Information, Files, and Links
No response
Activity
qchempku2017 commentedon Sep 28, 2026
Local hardware is nvidia RTX4060, run in WSL2 conda environment.
njzjz-bot commentedon Sep 29, 2026
Thanks for the detailed traceback. This is consistent with a conda packaging/runtime dependency gap rather than a damaged checkpoint.
The failure happens when Triton/AOTInductor compiles its small NVIDIA driver helper and includes
cuda.h. In the current conda-forgedeepmd-kitrecipe,cuda-driver-dev(andcuda-cudart-dev) are present in the build/host environment, andcuda-driver-devis also present in the test environment, but the Python runtime output does not depend oncuda-driver-dev. That explains why the feedstock can build/test while a clean user environment can reach this missing-header failure duringdp --pt freeze.As a workaround, please try installing the matching development package into the same environment:
and rerun the same freeze command. If the solver asks for a CUDA version choice, please keep it consistent with the CUDA variant selected for PyTorch/DeepMD.
If that fixes the issue, we should update the conda-forge runtime requirements (and add a freeze/AOTI smoke test that actually exercises this path) rather than handling it in the model checkpoint code. If it still fails after installing
cuda-driver-dev, please paste the output ofconda list | grep -E 'deepmd|pytorch|cuda|triton'; no model upload should be necessary at that point.Agent: ChatGPT
Model: GPT-5.6 Sol