Repository navigation
[Bugfix] Reject stray <|AUDIO|> placeholder with use_audio_in_video in Qwen2.5-Omni - #7959
Conversation
…n Qwen2.5-Omni With use_audio_in_video=True the Qwen2.5-Omni processor interleaves each audio into its video placeholder and skips standalone audio prompt updates. A separate <|AUDIO|> placeholder in the prompt was therefore left unexpanded with no audio features behind it, and the thinker engine died with an IndexError in get_mrope_input_positions. Reject such prompts in the multimodal processor with a clear ValueError, and drop the stray placeholder from the offline example's use_audio_in_video query, which hit this crash. Signed-off-by: RunguoLi <[email protected]>
|
This PR appears to belong to: docs/design/module/model_integration.md, docs/design/module/ar_runtime.md. Module owners: @tzhouam @fake0fan @Gaohan123 Routing: @tzhouam via module of the changed files, CODEOWNERS; @fake0fan via module of the changed files; @Gaohan123 via module of the changed files @LiRunGuo, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer. Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment. |
Omni ReviewBot triage noteAutomated triage of commit
These are automated triage suggestions only — the final decision belongs to the maintainers. |
|
Self-review: I hit this crash when running the offline example with What I checked:
One thing I did not change: in offline |
linyueqian
left a comment
There was a problem hiding this comment.
Right diagnosis and the right place to fail. In the use_audio_in_video branch the processor drops every standalone audio prompt update and derives the audio placeholders from the video ones, so a separate <|AUDIO|> in the prompt is left unexpanded with no features behind it and get_mrope_input_positions indexes past audio_feature_lengths, taking the whole thinker down. Rejecting that prompt at input processing turns an EngineDeadError into a 400 for the one request, the interleaved placeholder (audio tokens inside the video span) still passes because the check walks the video placeholder ranges, and the offline example is fixed to match the documented prompt shape. The two tests cover the accept and reject cases.
Static read at 6dc51cb4; fork head, no PR code executed. pre-commit and DCO are green and Voyagerroc-Lab has approved; the PR has no ready label so no general lane has run, and it needs one before merge.
…n Qwen2.5-Omni (vllm-project#7959) Signed-off-by: RunguoLi <[email protected]> Signed-off-by: Matthieu Laneuville <[email protected]>
…n Qwen2.5-Omni (vllm-project#7959) Signed-off-by: RunguoLi <[email protected]>
Purpose
Running the Qwen2.5-Omni offline example with
--query-type use_audio_in_videokills the thinker engine on currentmain:Repro (before this PR):
cd examples/offline_inference/qwen2_5_omni python end2end.py --model Qwen/Qwen2.5-Omni-3B --query-type use_audio_in_videoRoot cause. The example prompt contains
<|vision_bos|><|VIDEO|><|vision_eos|>plus a separate<|audio_bos|><|AUDIO|><|audio_eos|>. Withuse_audio_in_video=True,_maybe_apply_prompt_updatesdrops theaudioprompt updates and derives the audio placeholder from the video placeholder (the audio is interleaved into the video span). The standalone<|AUDIO|>token is therefore left unexpanded with no audio item behind it. It passes input processing, reaches the engine, andget_mrope_input_positionsindexes past the end ofaudio_feature_lengths, which takes down the whole stage instead of failing the one request.Fix.
Qwen2_5OmniThinkerMultiModalProcessor: after deriving audio placeholders from video placeholders, reject any<|AUDIO|>token that lies outside a video placeholder with a clearValueErrorat input-processing time, so a malformed prompt fails that request instead of crashing the engine.<|AUDIO|>placeholder from theuse_audio_in_videoquery (this matches the prompt shape described in [Bug]: Qwen2.5-Omni crashes on interleaved audio/video input #7509).The SPDX line change in the example is the repo's
check-spdx-headerhook rewriting the legacy copyright line of a touched file.Test Plan
vLLM Version: 0.29.0 (torch 2.13.0+cu130)
vLLM-Omni Commit: 381b12f + this PR
test_reject_audio_tokens_outside_videos_accepts_interleaved_audioandtest_reject_audio_tokens_outside_videos_rejects_standalone_audio_placeholder.Qwen2.5-Omni-3B, default deploy YAML: the repro command above with--num-prompts 2.<|AUDIO|>,use_audio_in_video=True) through offlineOmni.generate.ruff check,ruff format --check,typos, andtools/pre_commit/{check_test_marks,check_spdx_header,check_forbidden_imports,check_torch_cuda}.py.Test Result
34 passed(32 existing + 2 new).IndexError/EngineDeadErroras above (both requests fail). After:rc=0, both requests return text and audio; the text describes the video and transcribes the baby's speech.ValueError: use_audio_in_video=True interleaves each audio into its video placeholder; remove the separate <|AUDIO|> placeholder from the prompt.and noIndexError/EngineDeadErrorin the log.Not run: the full
pre-commitsuite (mypy matrix, markdownlint, actionlint) and L2+ GPU CI tests; only the checks listed above were run locally.AI assistance: Used Claude Code to help diagnose the crash and draft the fix, tests, and this description. I reviewed the changes and ran the validation above.