System Info
Copy-and-paste the text below in your GitHub issue and FILL OUT the two last points.
transformers version: 4.36.1- Platform: Linux-6.6.7-arch1-1-x86_64-with-glibc2.38
- Python version: 3.11.6
- Huggingface_hub version: 0.19.4
- Safetensors version: 0.4.1
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.1.2+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: YES (this is the cause of the bug)
- Using distributed or parallel set-up in script?: no, one local GPU
Who can help?
@sanchit-gandhi
Information
Tasks
Reproduction
I'm creating a WhisperProcessor, copy input and model to GPU and run generate() with return_timestamps=True.
processor = WhisperProcessor.from_pretrained("openai/whisper-large-v2")
model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-large-v2")
model = model.to("cuda")
input_features = input_features.to("cuda")
with torch.backends.cuda.sdp_kernel(enable_flash=True, enable_math=True, enable_mem_efficient=True):
output_with_prompt = model.generate(input_features, return_timestamps=True)
result = processor.decode(output_with_prompt[0], skip_special_tokens=True, decode_with_timestamps=False, output_offsets=True)
print(result)Causes the error:
File [/media/DataStore02-12TB/codeRepos/myFasterWhisperTest/.venv/lib/python3.11/site-packages/torch/_tensor.py:1030](https://file+.vscode-resource.vscode-cdn.net/media/DataStore02-12TB/codeRepos/myFasterWhisperTest/.venv/lib/python3.11/site-packages/torch/_tensor.py:1030), in Tensor.__array__(self, dtype)
[1028](https://file+.vscode-resource.vscode-cdn.net/media/DataStore02-12TB/codeRepos/myFasterWhisperTest/.venv/lib/python3.11/site-packages/torch/_tensor.py:1028) return handle_torch_function(Tensor.__array__, (self,), self, dtype=dtype)
[1029](https://file+.vscode-resource.vscode-cdn.net/media/DataStore02-12TB/codeRepos/myFasterWhisperTest/.venv/lib/python3.11/site-packages/torch/_tensor.py:1029) if dtype is None:
-> [1030](https://file+.vscode-resource.vscode-cdn.net/media/DataStore02-12TB/codeRepos/myFasterWhisperTest/.venv/lib/python3.11/site-packages/torch/_tensor.py:1030) return self.numpy()
[1031](https://file+.vscode-resource.vscode-cdn.net/media/DataStore02-12TB/codeRepos/myFasterWhisperTest/.venv/lib/python3.11/site-packages/torch/_tensor.py:1031) else:
[1032](https://file+.vscode-resource.vscode-cdn.net/media/DataStore02-12TB/codeRepos/myFasterWhisperTest/.venv/lib/python3.11/site-packages/torch/_tensor.py:1032) return self.numpy().astype(dtype, copy=False)
TypeError: can't convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.
Copying the output tensor to the CPU before fixes that:
output_with_prompt2=output_with_prompt.cpu()
# timesatmps must be on to skip the initial prompt
result = processor.decode(output_with_prompt2[0], skip_special_tokens=True, decode_with_timestamps=False, output_offsets=True)
print(result)
However, that only happens when output_offsets=True is enabled. When the flag is disabled, the decode works fine on the GPU (but we don't get the timestamps).
I'm also seeing that the decode function is called about 11 times for one processor.decode, is that by design?
Expected behavior
Automatically copy the tensor to CPU for the decode.
System Info
Copy-and-paste the text below in your GitHub issue and FILL OUT the two last points.
transformersversion: 4.36.1Who can help?
@sanchit-gandhi
Information
Tasks
examplesfolder (such as GLUE/SQuAD, ...)Reproduction
I'm creating a WhisperProcessor, copy input and model to GPU and run
generate()withreturn_timestamps=True.Causes the error:
Copying the output tensor to the CPU before fixes that:
However, that only happens when
output_offsets=Trueis enabled. When the flag is disabled, the decode works fine on the GPU (but we don't get the timestamps).I'm also seeing that the decode function is called about 11 times for one
processor.decode, is that by design?Expected behavior
Automatically copy the tensor to CPU for the decode.