System Info
transformers version: 4.42.4- Platform: Linux-6.8.9-arch1-2-x86_64-with-glibc2.39
- Python version: 3.11.9
- Huggingface_hub version: 0.23.4
- Safetensors version: 0.4.3
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.3.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: No
- Using GPU in script?: Yes
- GPU type: NVIDIA GeForce RTX 3090
Who can help?
@Narsil @sanchit-gandhi
Information
Tasks
Reproduction
import numpy as np
import json
from transformers import pipeline, AutoModelForSpeechSeq2Seq, AutoProcessor
import torch
from datasets import load_dataset
device = "cuda"
torch_dtype = torch.bfloat16
# model_id = "openai/whisper-large-v3"
model_id = "distil-whisper/distil-large-v3"
model = AutoModelForSpeechSeq2Seq.from_pretrained(
model_id, torch_dtype=torch_dtype, low_cpu_mem_usage=False, use_safetensors=True
)
model.to(device)
processor = AutoProcessor.from_pretrained(model_id)
pipe = pipeline(
"automatic-speech-recognition",
model=model,
tokenizer=processor.tokenizer,
feature_extractor=processor.feature_extractor,
max_new_tokens=None,
torch_dtype=torch_dtype,
device=device,
)
dataset = load_dataset(
"hf-internal-testing/librispeech_asr_dummy", "clean", split="validation"
)
sample = dataset[0]["audio"]
sample = np.concatenate([sample["array"]] * 10)
results = pipe(
sample,
return_timestamps=True,
generate_kwargs={
"language": "english",
},
)
print(json.dumps(results, indent=4))Output
{
"text": " Mr. Quilter is the Apostle of the Middle Classes, and we are glad to welcome his gospel. Mr. Quilter is the Apostle of the Middle Classes, and we are glad to welcome his gospel. Mr. Quilter is the Apostle of the Middle Classes, and we are glad to welcome His Gospel. Mr. Quilter is the Apostle of the Middle Classes, and we are glad to welcome his gospel. Mr. Quilter is the Apostle of the Middle Classes, and we are glad to welcome his gospel. Mr. Quilter is the Apostle of the Middle Classes, and we are glad to welcome his gospel. Mr. Quilter is the Apostle of the Middle Classes, and we are glad to welcome his gospel. Mr. Quilter is the Apostle of the Middle Classes, and we are glad to welcome his gospel. Mr. Quilter is the Apostle of the Middle Classes, and we are glad to welcome his gospel. Mr. Quilter is the Apostle of the Middle Classes, and we are glad to welcome his gospel.",
"chunks": [
{
"timestamp": [
0.0,
6.5
],
"text": " Mr. Quilter is the Apostle of the Middle Classes, and we are glad to welcome his gospel."
},
{
"timestamp": [
6.5,
12.5
],
"text": " Mr. Quilter is the Apostle of the Middle Classes, and we are glad to welcome his gospel."
},
{
"timestamp": [
12.5,
18.24
],
"text": " Mr. Quilter is the Apostle of the Middle Classes, and we are glad to welcome His Gospel."
},
{
"timestamp": [
18.24,
24.0
],
"text": " Mr. Quilter is the Apostle of the Middle Classes, and we are glad to welcome his gospel."
},
{
"timestamp": [
24.0,
29.84
],
"text": " Mr. Quilter is the Apostle of the Middle Classes, and we are glad to welcome his gospel."
},
{
"timestamp": [
0.0,
4.7
],
"text": " Mr. Quilter is the Apostle of the Middle Classes, and we are glad to welcome his gospel."
},
{
"timestamp": [
5.76,
10.54
],
"text": " Mr. Quilter is the Apostle of the Middle Classes, and we are glad to welcome his gospel."
},
{
"timestamp": [
11.6,
16.4
],
"text": " Mr. Quilter is the Apostle of the Middle Classes, and we are glad to welcome his gospel."
},
{
"timestamp": [
17.46,
22.26
],
"text": " Mr. Quilter is the Apostle of the Middle Classes, and we are glad to welcome his gospel."
},
{
"timestamp": [
23.12,
28.12
],
"text": " Mr. Quilter is the Apostle of the Middle Classes, and we are glad to welcome his gospel."
}
]
}
Expected behavior
Currently the timestamp resets to zero after 30s of audio. I expect the timestamps to increase monotonically.
System Info
transformersversion: 4.42.4Who can help?
@Narsil @sanchit-gandhi
Information
Tasks
examplesfolder (such as GLUE/SQuAD, ...)Reproduction
Output
{ "text": " Mr. Quilter is the Apostle of the Middle Classes, and we are glad to welcome his gospel. Mr. Quilter is the Apostle of the Middle Classes, and we are glad to welcome his gospel. Mr. Quilter is the Apostle of the Middle Classes, and we are glad to welcome His Gospel. Mr. Quilter is the Apostle of the Middle Classes, and we are glad to welcome his gospel. Mr. Quilter is the Apostle of the Middle Classes, and we are glad to welcome his gospel. Mr. Quilter is the Apostle of the Middle Classes, and we are glad to welcome his gospel. Mr. Quilter is the Apostle of the Middle Classes, and we are glad to welcome his gospel. Mr. Quilter is the Apostle of the Middle Classes, and we are glad to welcome his gospel. Mr. Quilter is the Apostle of the Middle Classes, and we are glad to welcome his gospel. Mr. Quilter is the Apostle of the Middle Classes, and we are glad to welcome his gospel.", "chunks": [ { "timestamp": [ 0.0, 6.5 ], "text": " Mr. Quilter is the Apostle of the Middle Classes, and we are glad to welcome his gospel." }, { "timestamp": [ 6.5, 12.5 ], "text": " Mr. Quilter is the Apostle of the Middle Classes, and we are glad to welcome his gospel." }, { "timestamp": [ 12.5, 18.24 ], "text": " Mr. Quilter is the Apostle of the Middle Classes, and we are glad to welcome His Gospel." }, { "timestamp": [ 18.24, 24.0 ], "text": " Mr. Quilter is the Apostle of the Middle Classes, and we are glad to welcome his gospel." }, { "timestamp": [ 24.0, 29.84 ], "text": " Mr. Quilter is the Apostle of the Middle Classes, and we are glad to welcome his gospel." }, { "timestamp": [ 0.0, 4.7 ], "text": " Mr. Quilter is the Apostle of the Middle Classes, and we are glad to welcome his gospel." }, { "timestamp": [ 5.76, 10.54 ], "text": " Mr. Quilter is the Apostle of the Middle Classes, and we are glad to welcome his gospel." }, { "timestamp": [ 11.6, 16.4 ], "text": " Mr. Quilter is the Apostle of the Middle Classes, and we are glad to welcome his gospel." }, { "timestamp": [ 17.46, 22.26 ], "text": " Mr. Quilter is the Apostle of the Middle Classes, and we are glad to welcome his gospel." }, { "timestamp": [ 23.12, 28.12 ], "text": " Mr. Quilter is the Apostle of the Middle Classes, and we are glad to welcome his gospel." } ] }Expected behavior
Currently the timestamp resets to zero after 30s of audio. I expect the timestamps to increase monotonically.