model convert from pytorch to onnx met error at onnxruntimedev/miniconda3/lib/python3.9/site-packages/torch/nn/modules/linear.py line 114, in forward
RuntimeError: "addmm_impl_cpu_" not implemented for 'Half'
Traceback (most recent call last):
File "/home/onnxruntimedev/miniconda3/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/home/onnxruntimedev/miniconda3/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/raid/kangwang/llama2/onnx/onnxruntime/onnxruntime/python/tools/transformers/models/llama/convert_to_onnx.py", line 576, in <module>
main()
File "/raid/kangwang/llama2/onnx/onnxruntime/onnxruntime/python/tools/transformers/models/llama/convert_to_onnx.py", line 572, in main
parity_check(parity_cmd)
File "/raid/kangwang/llama2/onnx/onnxruntime/onnxruntime/python/tools/transformers/models/llama/llama_parity.py", line 128, in main
verify_parity(args, config, llama)
File "/raid/kangwang/llama2/onnx/onnxruntime/onnxruntime/python/tools/transformers/models/llama/llama_parity.py", line 28, in verify_parity
pt_outputs = pt_model(**inputs).logits.detach().cpu().numpy()
File "/home/onnxruntimedev/miniconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/onnxruntimedev/miniconda3/lib/python3.9/site-packages/transformers/models/llama/modeling_llama.py", line 1035, in forward
outputs = self.model(
File "/home/onnxruntimedev/miniconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/onnxruntimedev/miniconda3/lib/python3.9/site-packages/transformers/models/llama/modeling_llama.py", line 922, in forward
layer_outputs = decoder_layer(
File "/home/onnxruntimedev/miniconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/onnxruntimedev/miniconda3/lib/python3.9/site-packages/transformers/models/llama/modeling_llama.py", line 632, in forward
hidden_states, self_attn_weights, present_key_value = self.self_attn(
File "/home/onnxruntimedev/miniconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/onnxruntimedev/miniconda3/lib/python3.9/site-packages/transformers/models/llama/modeling_llama.py", line 350, in forward
query_states = self.q_proj(hidden_states)
File "/home/onnxruntimedev/miniconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/onnxruntimedev/miniconda3/lib/python3.9/site-packages/torch/nn/modules/linear.py", line 114, in forward
return F.linear(input, self.weight, self.bias)
RuntimeError: "addmm_impl_cpu_" not implemented for 'Half'
Describe the issue
model convert from pytorch to onnx met error at onnxruntimedev/miniconda3/lib/python3.9/site-packages/torch/nn/modules/linear.py line 114, in forward
RuntimeError: "addmm_impl_cpu_" not implemented for 'Half'
detail error msg below
To reproduce
cd ./onnxruntime/tree/main/onnxruntime/python/tools/transformers/models/llama/
$ python3 -m models.llama.convert_to_onnx -m meta-llama/Llama-2-7b-hf --output llama2-7b-fp16 --precision fp16
Urgency
wish to have a quick fix
Platform
Linux
OS Version
Ubuntu 20.04.6 LTS
ONNX Runtime Installation
Built from Source
ONNX Runtime Version or Commit ID
db558ef
ONNX Runtime API
Python
Architecture
X64
Execution Provider
CUDA
Execution Provider Library Version
cuda_11.8.r11.8