[generation] Encode multimodal data only once#45783
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
|
run-slow: aria, aya_vision, blip_2, chameleon, cohere2_vision, deepseek_vl, deepseek_vl_hybrid, emu3, ernie4_5_vl_moe, fast_vlm, florence2, fuyu, gemma3, gemma3n, gemma4, glm46v |
|
run-slow: aria, aya_vision, blip_2, chameleon, cohere2_vision, deepseek_vl, deepseek_vl_hybrid, emu3, ernie4_5_vl_moe, fast_vlm, florence2, fuyu, gemma3, gemma3n, gemma4, glm46v, qwen2_vl, mllama, llava, llava_next_video |
|
This comment contains models: ["models/aria", "models/aya_vision", "models/blip_2", "models/chameleon", "models/cohere2_vision", "models/deepseek_vl", "models/deepseek_vl_hybrid", "models/emu3", "models/ernie4_5_vl_moe", "models/fast_vlm", "models/florence2", "models/fuyu", "models/gemma3", "models/gemma3n", "models/gemma4", "models/glm46v", "models/llava", "models/llava_next_video", "models/mllama", "models/qwen2_vl"] |
CI ResultsCommit Info
Model CI Report❌ 7 new failed tests from this PR 😭
|
|
View the CircleCI Test Summary for this PR: |
|
[For maintainers] Suggested jobs to run (before merge) run-slow: aria, aya_vision, blip_2, chameleon, cohere2_vision, deepseek_vl, deepseek_vl_hybrid, emu3, ernie4_5_vl_moe, exaone4_5, fast_vlm, florence2, fuyu, gemma3, gemma3n, gemma4 |
CI recapDashboard: View test results in Grafana
|
3 participants
What does this PR do?
As per title, moves multimodal encoding out of the loop so it happens only once and on un-expanded pixels. For example, in assisted generation we used to encode twice (once in assistant and once in target model), or in beam search we used to expand all pixel by
x num_beamstimes. This is very inefficient, especially in models where vision encoder requires huge memory like qwen-series (usually extremely long sequence lengths)