Bug description
Qwen3VLVisionAttention (and Qwen2_5_VLVisionAttention) computes max_seqlen as a 0-d tensor:
# src/transformers/models/qwen3_vl/modeling_qwen3_vl.py, line 221
max_seqlen = (cu_seqlens[1:] - cu_seqlens[:-1]).max()
This is then passed to flash_attn_varlen_func via max_length_q / max_length_k, which expects int. During eager execution this works because PyTorch silently coerces the 0-d tensor. But under torch.compile, Dynamo traces it as a FakeTensor, and the flash_attn C++ op (flash_attn::_flash_attn_varlen_forward) rejects it:
TorchRuntimeError: flash_attn::_flash_attn_varlen_forward() Expected a value of type 'int'
for argument 'max_seqlen_q' but instead found type 'FakeTensor'.
Fix
Add .item() to convert the 0-d tensor to a Python int:
max_seqlen = (cu_seqlens[1:] - cu_seqlens[:-1]).max().item()
This is consistent with how transformers/modeling_flash_attention_utils.py already handles the same issue (line 354):
# This is a limitation of flash attention API, as the function `flash_attn_varlen_func`
# requires `max_length_q`, `max_length_k` to be passed as `int` and not `torch.Tensor`.
max_length_q = max_length_q.item()
Affected models
Qwen3VLVisionAttention (src/transformers/models/qwen3_vl/modeling_qwen3_vl.py, line 221)Qwen2_5_VLVisionAttention (src/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py, line 246)
Reproduction
import torch
torch.set_float32_matmul_precision("high")
from PIL import Image
from transformers import AutoProcessor, Qwen3VLForConditionalGeneration
processor = AutoProcessor.from_pretrained("Qwen/Qwen3-VL-Embedding-2B", trust_remote_code=True)
model = Qwen3VLForConditionalGeneration.from_pretrained(
"Qwen/Qwen3-VL-Embedding-2B",
dtype=torch.bfloat16,
attn_implementation="flash_attention_2",
).cuda().eval()
model = torch.compile(model, mode="max-autotune-no-cudagraphs")
img = Image.new("RGB", (875, 1024), color=(128, 128, 128))
messages = [
{"role": "system", "content": [{"type": "text", "text": "Describe."}]},
{"role": "user", "content": [{"type": "image", "image": img}]},
]
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(text=[text], images=[img], return_tensors="pt", padding=True)
inputs = {k: v.to("cuda") if hasattr(v, "to") else v for k, v in inputs.items()}
with torch.no_grad():
outputs = model(**inputs, output_hidden_states=True) # crashesEnvironment
- transformers: 4.52.4 (also confirmed on main @ 2026-03-24)
- flash-attn: 2.8.3
- torch: 2.7.1+cu128
- GPU: H100
Bug description
Qwen3VLVisionAttention(andQwen2_5_VLVisionAttention) computesmax_seqlenas a 0-d tensor:This is then passed to
flash_attn_varlen_funcviamax_length_q/max_length_k, which expectsint. During eager execution this works because PyTorch silently coerces the 0-d tensor. But undertorch.compile, Dynamo traces it as aFakeTensor, and the flash_attn C++ op (flash_attn::_flash_attn_varlen_forward) rejects it:Fix
Add
.item()to convert the 0-d tensor to a Python int:This is consistent with how
transformers/modeling_flash_attention_utils.pyalready handles the same issue (line 354):Affected models
Qwen3VLVisionAttention(src/transformers/models/qwen3_vl/modeling_qwen3_vl.py, line 221)Qwen2_5_VLVisionAttention(src/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py, line 246)Reproduction
Environment