You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Improve pre-processed image size for QwenVL #16842
In order for the bbox to be correct, I'm thinking about:
Implement the correct max_pixels / min_pixels from the original config
Pad right/bottom corner of the image if we require to upscale (to multiple of 2*patch_size). This will guarantee that the x/y coordinates stay unchanged. But we also need to check if the original implementation actually use this strategy or not.
I think this might be the key method from the reference implementation:
defsmart_resize(
height: int, width: int, factor: int=28, min_pixels: int=56*56, max_pixels: int=14*14*4*1280
):
"""Rescales the image so that the following conditions are met: 1. Both dimensions (height and width) are divisible by 'factor'. 2. The total number of pixels is within the range ['min_pixels', 'max_pixels']. 3. The aspect ratio of the image is maintained as closely as possible. """ifmax(height, width) /min(height, width) >200:
raiseValueError(
f"absolute aspect ratio must be smaller than 200, got {max(height, width) /min(height, width)}"
)
h_bar=round(height/factor) *factorw_bar=round(width/factor) *factorifh_bar*w_bar>max_pixels:
beta=math.sqrt((height*width) /max_pixels)
h_bar=max(factor, math.floor(height/beta/factor) *factor)
w_bar=max(factor, math.floor(width/beta/factor) *factor)
elifh_bar*w_bar<min_pixels:
beta=math.sqrt(min_pixels/ (height*width))
h_bar=math.ceil(height*beta/factor) *factorw_bar=math.ceil(width*beta/factor) *factorreturnh_bar, w_bar
Note that it will be not just an improvement to QwenVL, but other models that use the max_pixels / min_pixels strategy will also reuse the same code from this fix
Reacted by Piotr Wilkin (ilintar), whyareallnamestakenalready, JJJYmmm and PSK
I'm still doing some tests using Qwen3 VL 30B A3B Q8_0 and on the invoice test it works correct on master but it's wrong on #16878, similar to what @tarruda showed.
Using Qwen2.5 VL 7B Instruct Q8_0, it is incorrect on both branches. Still trying things.
@danielhanchen since a model family often (if not always) reuse the same min/max pixels param for all of it model sizes, I hard-coded the min/max values into the cpp code. Therefore, you don't need to reconvert the GGUF.
also, this conversion script doesn't yet support reading this info from the HF model, but it will be added when we encounter the next model that uses this min/max param.
ref: the cpp code where these values are hard-coded; they are not 100% complete, some TODO are there; feel free to correct them if you want!
for flash attn, I don't have an AMD card to test. can you try these case:
with --no-mmproj-offload, meaning vision encoder is not offloaded to GPU
The results are correct in every test with --no-mmproj-offload:
--- vision hparams ---
load_hparams: image_size: 1540
load_hparams: patch_size: 14
load_hparams: has_llava_proj: 0
load_hparams: minicpmv_version: 0
load_hparams: n_merge: 2
load_hparams: n_wa_pattern: 0
load_hparams: image_min_pixels: 6272
load_hparams: image_max_pixels: 802816
load_hparams: model size: 837.36 MiB
load_hparams: metadata size: 0.08 MiB
alloc_compute_meta: warmup with image size = 448 x 448
alloc_compute_meta: CPU compute buffer size = 62.62 MiB
alloc_compute_meta: graph splits = 1, nodes = 845
warmup: flash attention is enabled
main: loading model: ../../text/models/BroadBit/mistral-small-24b-instruct-2506-q4_k_m.gguf
encoding image slice...
image slice encoded in 40735 ms
decoding image batch 1/1, n_tokens_batch = 998
image decoded (batch 1/1) in 603 ms
The image shows a setup in what appears to be an office or workspace. On a black desk, there are four identical devices lined up in a row. Each device has a sleek, modern design with a prominent white triangle on the front and the word "COLLAIGUE" written below it. The devices appear to be some kind of electronic equipment, possibly speakers or audio devices...
try running the model without any multimodal support (for example, using llama-cli)
The model works correctly without multimodal projection (without --mmproj).
try running everything on CPU with/without flash attention: -ngl 0 -fa on and -ngl 0 -fa off
-ngl 0 -fa on and -ngl 0 -fa off are same, first run buggy, second run OK:
load_hparams: model size: 837.36 MiB
load_hparams: metadata size: 0.08 MiB
alloc_compute_meta: warmup with image size = 448 x 448
alloc_compute_meta: ROCm0 compute buffer size = 62.62 MiB
alloc_compute_meta: CPU compute buffer size = 2.30 MiB
alloc_compute_meta: graph splits = 1, nodes = 845
warmup: flash attention is enabled
main: loading model: ../../text/models/BroadBit/mistral-small-24b-instruct-2506-q4_k_m.gguf
encoding image slice...
image slice encoded in 325 ms
decoding image batch 1/1, n_tokens_batch = 998
image decoded (batch 1/1) in 3479 ms
üksad0 in the Q for in a in ( in ( in here
can and### question about interesting for in
new in the in ( in ( Q in ( Q in ( Q with ( Q in ( . in ( Q of in ( Q in
@broadbit-hu it seems to be a problem specific to ROCm backend with flash attention, but unfortunately I don't have enough knowledge to fix it. feel free to open a dedicated issue so we can assign to ROCm backend maintainer
@broadbit-hu it seems to be a problem specific to ROCm backend with flash attention, but unfortunately I don't have enough knowledge to fix it. feel free to open a dedicated issue so we can assign to ROCm backend maintainer
@ngxson Thanks for you fixes! We'll continue the testing on several AMD GPUs... :)
Activity
pwilkin commentedon Oct 30, 2025
I think this might be the key method from the reference implementation:
ngxson commentedon Oct 30, 2025
Yes, indeed the functions which handles the
max_pixels/min_pixelsis pretty much the same across multiple models: https://github.com/search?q=repo%3Ahuggingface%2Ftransformers%20max_pixels&type=codepwilkin commentedon Oct 30, 2025
@ngxson I checked the code for 2.5 VL and it explicitly references the 2 VL image processor.
ngxson commentedon Oct 30, 2025
Note that it will be not just an improvement to QwenVL, but other models that use the max_pixels / min_pixels strategy will also reuse the same code from this fix
tarruda commentedon Oct 31, 2025
On master branch:
On this branch (I meant to comment in the PR but ended up doing it here):
This is qwen3-vl-8b-instruct UD-Q4_K_XL quant from unsloth (mmproj-F32).
Using this test page, but I changed it in two ways:
Both using these inference parameter:
{ "temperature": 0, "top_k": 1, "max_tokens": 24000, "repeat_penalty": 1, "cache_prompt": false, "samplers": [ "top_k", "temperature" ] }tarruda commentedon Oct 31, 2025
Dummy invoice I used above:
tarruda commentedon Oct 31, 2025
Uploaded my modified test page: https://gist.github.com/tarruda/09dcbc44c2be0cbc96a4b9809942d503
ngxson commentedon Oct 31, 2025
I think your test script have problems. It's better to draw the bbox using HTML canvas as it will be pixel-accurate. Don't trust CSS pixel
Also, try uploading a raw JPG/PNG without modifications. The image you provided was a SVG.
tarruda commentedon Oct 31, 2025
@ngxson did you use my version of the test page? https://gist.github.com/tarruda/09dcbc44c2be0cbc96a4b9809942d503
From your screenshot, it seems like the coordinates were not adjusted on the client.
This is the result with Qwen3-VL-32B-Instruct q8_0 on master branch, seems pretty accurate to me:
Here's the result on #16878
The original dummy invoice was svg, I converted to png using PIL.
Here's the same png file I tested on:
tarruda commentedon Oct 31, 2025
@ngxson also my llama-server script:
ggerganov commentedon Oct 31, 2025
I'm still doing some tests using
Qwen3 VL 30B A3B Q8_0and on the invoice test it works correct onmasterbut it's wrong on #16878, similar to what @tarruda showed.Using
Qwen2.5 VL 7B Instruct Q8_0, it is incorrect on both branches. Still trying things.tarruda commentedon Oct 31, 2025
In the various tests I've ran locally, Qwen3 VL 32B Instruct Q8_0 is the most accurate, so I suggest using it as a baseline.
ngxson commentedon Oct 31, 2025
I can't try the 32B model right now. The screenshot earlier was using your latest html from your gist.
Can you try this html instead? I'm using canvas API so it's more accurate: https://gist.github.com/ngxson/a50158f0f0e32637de347d0fb0f11f07
36 remaining items
ngxson commentedon Nov 4, 2025
@danielhanchen since a model family often (if not always) reuse the same min/max pixels param for all of it model sizes, I hard-coded the min/max values into the cpp code. Therefore, you don't need to reconvert the GGUF.
also, this conversion script doesn't yet support reading this info from the HF model, but it will be added when we encounter the next model that uses this min/max param.
ref: the cpp code where these values are hard-coded; they are not 100% complete, some TODO are there; feel free to correct them if you want!
llama.cpp/tools/mtmd/clip.cpp
Lines 2754 to 2775 in 1f5accb
broadbit-hu commentedon Nov 4, 2025
Yes, the parameters are correct in each test:
broadbit-hu commentedon Nov 4, 2025
The results are correct in every test with
--no-mmproj-offload:The model works correctly without multimodal projection (without
--mmproj).-ngl 0 -fa onand-ngl 0 -fa offare same, first run buggy, second run OK:ngxson commentedon Nov 5, 2025
@broadbit-hu it seems to be a problem specific to ROCm backend with flash attention, but unfortunately I don't have enough knowledge to fix it. feel free to open a dedicated issue so we can assign to ROCm backend maintainer
broadbit-hu commentedon Nov 5, 2025
@ngxson Thanks for you fixes! We'll continue the testing on several AMD GPUs... :)
ggerganov commentedon Nov 5, 2025
@broadbit-hu On the failing machine, run this test and see if it produces any errors:
broadbit-hu commentedon Nov 5, 2025
@ggerganov No errors while
test-backend-opson 7900 XT and 7900 XTX GPUs with ROCm 6.3.4:feat: make image-min-tokens configurable from JS config
fix: add image-min-tokens 1024 to Qwen3.5 VLM perf tests
fix(llm): restore --image-min-tokens 1024 on both vision models
fix(vlm): give Qwen-VL the 1024-token floor it needs to ground
perf(vlm): send the matched crop at native size instead of upscaling it
feat(PROJ-99): parallel=1 + image-min-tokens=1024 in qwen preset
vision ของ Qwen-VL แม่นไม่เต็มที่ เพราะ controller ไม่เคยส่ง --image-…
Plan: the mind hands Fara one goal at a time (report + spec, no code)