Skip to content

[Bugfix] Reject stray <|AUDIO|> placeholder with use_audio_in_video in Qwen2.5-Omni - #7959

Merged
linyueqian merged 1 commit into
vllm-project:mainfrom
LiRunGuo:fix/qwen25-omni-audio-in-video-stray-audio
Sep 22, 2026
Merged

linyueqian merged 1 commit into
vllm-project:mainfrom
LiRunGuo:fix/qwen25-omni-audio-in-video-stray-audio

Conversation

@LiRunGuo

Copy link
Copy Markdown

Purpose

Running the Qwen2.5-Omni offline example with --query-type use_audio_in_video kills the thinker engine on current main:

File ".../qwen2_5_omni/qwen2_5_omni.py", line 489, in get_mrope_input_positions
    audio_seqlen = audio_seqlens[audio_idx]
IndexError: list index out of range
...
vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue.

Repro (before this PR):

cd examples/offline_inference/qwen2_5_omni
python end2end.py --model Qwen/Qwen2.5-Omni-3B --query-type use_audio_in_video

Root cause. The example prompt contains <|vision_bos|><|VIDEO|><|vision_eos|> plus a separate <|audio_bos|><|AUDIO|><|audio_eos|>. With use_audio_in_video=True, _maybe_apply_prompt_updates drops the audio prompt updates and derives the audio placeholder from the video placeholder (the audio is interleaved into the video span). The standalone <|AUDIO|> token is therefore left unexpanded with no audio item behind it. It passes input processing, reaches the engine, and get_mrope_input_positions indexes past the end of audio_feature_lengths, which takes down the whole stage instead of failing the one request.

Fix.

  • Qwen2_5OmniThinkerMultiModalProcessor: after deriving audio placeholders from video placeholders, reject any <|AUDIO|> token that lies outside a video placeholder with a clear ValueError at input-processing time, so a malformed prompt fails that request instead of crashing the engine.
  • Offline example: drop the stray <|AUDIO|> placeholder from the use_audio_in_video query (this matches the prompt shape described in [Bug]: Qwen2.5-Omni crashes on interleaved audio/video input #7509).

The SPDX line change in the example is the repo's check-spdx-header hook rewriting the legacy copyright line of a touched file.

Test Plan

vLLM Version: 0.29.0 (torch 2.13.0+cu130)

vLLM-Omni Commit: 381b12f + this PR

  1. Unit tests (CPU):
    pytest tests/model_executor/models/qwen2_5_omni/test_per_video_audio_mask.py -q
    Adds test_reject_audio_tokens_outside_videos_accepts_interleaved_audio and test_reject_audio_tokens_outside_videos_rejects_standalone_audio_placeholder.
  2. E2E on 2x H200, Qwen2.5-Omni-3B, default deploy YAML: the repro command above with --num-prompts 2.
  3. Malformed prompt check: send the old prompt (video + separate <|AUDIO|>, use_audio_in_video=True) through offline Omni.generate.
  4. Lint on the changed files: ruff check, ruff format --check, typos, and tools/pre_commit/{check_test_marks,check_spdx_header,check_forbidden_imports,check_torch_cuda}.py.

Test Result

  1. 34 passed (32 existing + 2 new).
  2. Before: IndexError / EngineDeadError as above (both requests fail). After: rc=0, both requests return text and audio; the text describes the video and transcribes the baby's speech.
  3. After: ValueError: use_audio_in_video=True interleaves each audio into its video placeholder; remove the separate <|AUDIO|> placeholder from the prompt. and no IndexError / EngineDeadError in the log.
  4. All pass.

Not run: the full pre-commit suite (mypy matrix, markdownlint, actionlint) and L2+ GPU CI tests; only the checks listed above were run locally.

AI assistance: Used Claude Code to help diagnose the crash and draft the fix, tests, and this description. I reviewed the changes and ran the validation above.

…n Qwen2.5-Omni

With use_audio_in_video=True the Qwen2.5-Omni processor interleaves each
audio into its video placeholder and skips standalone audio prompt
updates. A separate <|AUDIO|> placeholder in the prompt was therefore
left unexpanded with no audio features behind it, and the thinker engine
died with an IndexError in get_mrope_input_positions.

Reject such prompts in the multimodal processor with a clear ValueError,
and drop the stray placeholder from the offline example's
use_audio_in_video query, which hit this crash.

Signed-off-by: RunguoLi <[email protected]>
@vllm-omni-review-bot

Copy link
Copy Markdown

This PR appears to belong to: docs/design/module/model_integration.md, docs/design/module/ar_runtime.md.

Module owners: @tzhouam @fake0fan @Gaohan123

Routing: @tzhouam via module of the changed files, CODEOWNERS; @fake0fan via module of the changed files; @Gaohan123 via module of the changed files

@LiRunGuo, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer.

Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment.

@vllm-omni-review-bot

Copy link
Copy Markdown

Omni ReviewBot triage note

Automated triage of commit 6dc51cb44d13 produced:

  • Priority: high. Prompt maintainer attention is suggested.

These are automated triage suggestions only — the final decision belongs to the maintainers.

@LiRunGuo

Copy link
Copy Markdown
Author

Self-review:

I hit this crash when running the offline example with --query-type use_audio_in_video on 2x H200 (Qwen2.5-Omni-3B). The thinker engine died with IndexError in get_mrope_input_positions. After some digging, the reason is the extra <|AUDIO|> placeholder in the example prompt. With use_audio_in_video=True the audio is already put inside the video placeholder, so this extra token has no audio feature behind it.

What I checked:

  • After the fix, the example runs fine (rc=0, both text and audio come back, the text describes the video and the baby's words).
  • If I send the old prompt with the extra <|AUDIO|>, now it fails with a clear ValueError at input processing, and the engine keeps running.
  • Unit tests: 34 passed in test_per_video_audio_mask.py (2 new ones).
  • ruff, typos, SPDX and forbidden-import checks pass on the changed files. pre-commit and build CI are also green now.

One thing I did not change: in offline Omni.generate, any request-level exception still closes the whole instance (self.close() in the except). Maybe worth a separate issue, not sure if it is intended.

@hsliuustc0106 hsliuustc0106 added bug Something isn't working omni code related to omni models labels Sep 22, 2026
@linyueqian linyueqian added the ready label to trigger buildkite CI label Sep 22, 2026

@linyueqian linyueqian left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right diagnosis and the right place to fail. In the use_audio_in_video branch the processor drops every standalone audio prompt update and derives the audio placeholders from the video ones, so a separate <|AUDIO|> in the prompt is left unexpanded with no features behind it and get_mrope_input_positions indexes past audio_feature_lengths, taking the whole thinker down. Rejecting that prompt at input processing turns an EngineDeadError into a 400 for the one request, the interleaved placeholder (audio tokens inside the video span) still passes because the check walks the video placeholder ranges, and the offline example is fixed to match the documented prompt shape. The two tests cover the accept and reject cases.

Static read at 6dc51cb4; fork head, no PR code executed. pre-commit and DCO are green and Voyagerroc-Lab has approved; the PR has no ready label so no general lane has run, and it needs one before merge.

@linyueqian
linyueqian merged commit 6d33967 into vllm-project:main Sep 22, 2026
7 of 9 checks passed
mlaneuville pushed a commit to mlaneuville/vllm-omni that referenced this pull request Sep 22, 2026
…n Qwen2.5-Omni (vllm-project#7959)

Signed-off-by: RunguoLi <[email protected]>
Signed-off-by: Matthieu Laneuville <[email protected]>
khairulkabir1661 pushed a commit to khairulkabir1661/vllm-omni that referenced this pull request Sep 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working omni code related to omni models ready label to trigger buildkite CI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants