Skip to content

Improve pre-processed image size for QwenVL #16842

Closed

Description

@ngxson

Extracted the discussion from:

In order for the bbox to be correct, I'm thinking about:

  • Implement the correct max_pixels / min_pixels from the original config
  • Pad right/bottom corner of the image if we require to upscale (to multiple of 2*patch_size). This will guarantee that the x/y coordinates stay unchanged. But we also need to check if the original implementation actually use this strategy or not.

CC @broadbit-hu @theo77186

Activity

  1. self-assigned this
    on Oct 29, 2025
  2. pwilkin commented on Oct 30, 2025

    @pwilkin
    Member

    I think this might be the key method from the reference implementation:

    def smart_resize(
        height: int, width: int, factor: int = 28, min_pixels: int = 56 * 56, max_pixels: int = 14 * 14 * 4 * 1280
    ):
        """Rescales the image so that the following conditions are met:
    
        1. Both dimensions (height and width) are divisible by 'factor'.
    
        2. The total number of pixels is within the range ['min_pixels', 'max_pixels'].
    
        3. The aspect ratio of the image is maintained as closely as possible.
    
        """
        if max(height, width) / min(height, width) > 200:
            raise ValueError(
                f"absolute aspect ratio must be smaller than 200, got {max(height, width) / min(height, width)}"
            )
        h_bar = round(height / factor) * factor
        w_bar = round(width / factor) * factor
        if h_bar * w_bar > max_pixels:
            beta = math.sqrt((height * width) / max_pixels)
            h_bar = max(factor, math.floor(height / beta / factor) * factor)
            w_bar = max(factor, math.floor(width / beta / factor) * factor)
        elif h_bar * w_bar < min_pixels:
            beta = math.sqrt(min_pixels / (height * width))
            h_bar = math.ceil(height * beta / factor) * factor
            w_bar = math.ceil(width * beta / factor) * factor
        return h_bar, w_bar
  3. ngxson commented on Oct 30, 2025

    @ngxson
    CollaboratorAuthor

    Yes, indeed the functions which handles the max_pixels / min_pixels is pretty much the same across multiple models: https://github.com/search?q=repo%3Ahuggingface%2Ftransformers%20max_pixels&type=code

  4. pwilkin commented on Oct 30, 2025

    @pwilkin
    Member

    @ngxson I checked the code for 2.5 VL and it explicitly references the 2 VL image processor.

  5. ngxson commented on Oct 30, 2025

    @ngxson
    CollaboratorAuthor

    Note that it will be not just an improvement to QwenVL, but other models that use the max_pixels / min_pixels strategy will also reuse the same code from this fix

  6. tarruda commented on Oct 31, 2025

    @tarruda
    Contributor

    On master branch:

    Image

    On this branch (I meant to comment in the PR but ended up doing it here):

    Image

    This is qwen3-vl-8b-instruct UD-Q4_K_XL quant from unsloth (mmproj-F32).

    Using this test page, but I changed it in two ways:

    • Applied this fix
    • Modified the request to return structured output in the format required by the parsing code.

    Both using these inference parameter:

    {
      "temperature": 0,
      "top_k": 1,
      "max_tokens": 24000,
      "repeat_penalty": 1,
      "cache_prompt": false,
      "samplers": [
        "top_k",
        "temperature"
      ]
    }
  7. tarruda commented on Oct 31, 2025

    @tarruda
    Contributor

    Dummy invoice I used above:

    Image

  8. tarruda commented on Oct 31, 2025

    @tarruda
    Contributor
  9. ngxson commented on Oct 31, 2025

    @ngxson
    CollaboratorAuthor

    I think your test script have problems. It's better to draw the bbox using HTML canvas as it will be pixel-accurate. Don't trust CSS pixel

    Also, try uploading a raw JPG/PNG without modifications. The image you provided was a SVG.

    Image
  10. tarruda commented on Oct 31, 2025

    @tarruda
    Contributor

    @ngxson did you use my version of the test page? https://gist.github.com/tarruda/09dcbc44c2be0cbc96a4b9809942d503

    From your screenshot, it seems like the coordinates were not adjusted on the client.

    This is the result with Qwen3-VL-32B-Instruct q8_0 on master branch, seems pretty accurate to me:

    Image

    Here's the result on #16878

    Image

    The original dummy invoice was svg, I converted to png using PIL.
    Here's the same png file I tested on:

    Image
  11. tarruda commented on Oct 31, 2025

    @tarruda
    Contributor

    @ngxson also my llama-server script:

    #!/bin/sh -e
    
    model=Qwen3-VL-32B-Instruct-GGUF/q8_0/Qwen3-VL-32B-Instruct-Q8_0.gguf
    mmproj=Qwen3-VL-32B-Instruct-GGUF/q8_0/mmproj-F32.gguf
    
    llama-server --no-mmap --no-warmup --model $model --mmproj $mmproj --ctx-size 131072 --jinja -fa on --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.0 --host 0.0.0.0
  12. ggerganov commented on Oct 31, 2025

    @ggerganov
    Member

    I'm still doing some tests using Qwen3 VL 30B A3B Q8_0 and on the invoice test it works correct on master but it's wrong on #16878, similar to what @tarruda showed.

    Using Qwen2.5 VL 7B Instruct Q8_0, it is incorrect on both branches. Still trying things.

  13. tarruda commented on Oct 31, 2025

    @tarruda
    Contributor

    I'm still doing some tests using Qwen3 VL 30B A3B Q8_0

    In the various tests I've ran locally, Qwen3 VL 32B Instruct Q8_0 is the most accurate, so I suggest using it as a baseline.

  14. ngxson commented on Oct 31, 2025

    @ngxson
    CollaboratorAuthor

    I can't try the 32B model right now. The screenshot earlier was using your latest html from your gist.

    Can you try this html instead? I'm using canvas API so it's more accurate: https://gist.github.com/ngxson/a50158f0f0e32637de347d0fb0f11f07

  15. 36 remaining items

  16. ngxson commented on Nov 4, 2025

    @ngxson
    CollaboratorAuthor

    @danielhanchen since a model family often (if not always) reuse the same min/max pixels param for all of it model sizes, I hard-coded the min/max values into the cpp code. Therefore, you don't need to reconvert the GGUF.

    also, this conversion script doesn't yet support reading this info from the HF model, but it will be added when we encounter the next model that uses this min/max param.

    ref: the cpp code where these values are hard-coded; they are not 100% complete, some TODO are there; feel free to correct them if you want!

    llama.cpp/tools/mtmd/clip.cpp

    Lines 2754 to 2775 in 1f5accb

    {
    get_u32(KEY_PROJ_SCALE_FACTOR, hparams.n_merge, false);
    // ref: https://huggingface.co/LiquidAI/LFM2-VL-3B/blob/main/preprocessor_config.json
    hparams.set_limit_image_tokens(64, 256);
    } break;
    case PROJECTOR_TYPE_PIXTRAL:
    case PROJECTOR_TYPE_LIGHTONOCR:
    {
    // ref: https://huggingface.co/mistral-community/pixtral-12b/blob/main/preprocessor_config.json
    // TODO: verify the image_min_tokens
    hparams.rope_theta = 10000.0f;
    get_u32(KEY_SPATIAL_MERGE_SIZE, hparams.n_merge, false);
    hparams.set_limit_image_tokens(8, 1024);
    hparams.set_warmup_n_tokens(256); // avoid OOM on warmup
    } break;
    case PROJECTOR_TYPE_KIMIVL:
    {
    hparams.rope_theta = 10000.0f;
    get_u32(KEY_PROJ_SCALE_FACTOR, hparams.n_merge, false);
    // TODO: check kimivl preprocessor for exact values
    hparams.set_limit_image_tokens(8, 1024);
    hparams.set_warmup_n_tokens(256); // avoid OOM on warmup

  17. broadbit-hu commented on Nov 4, 2025

    @broadbit-hu

    @broadbit-hu the random value issue and the flash attn issue are 2 different issue.

    could you firstly confirm if the fix resolves the problem with random value of image_max_pixels ?

    Yes, the parameters are correct in each test:

    --- vision hparams ---
    load_hparams: image_size:         1540
    load_hparams: patch_size:         14
    load_hparams: has_llava_proj:     0
    load_hparams: minicpmv_version:   0
    load_hparams: n_merge:            2
    load_hparams: n_wa_pattern:       0
    load_hparams: image_min_pixels:   6272
    load_hparams: image_max_pixels:   802816
    
  18. broadbit-hu commented on Nov 4, 2025

    @broadbit-hu

    for flash attn, I don't have an AMD card to test. can you try these case:

    • with --no-mmproj-offload, meaning vision encoder is not offloaded to GPU

    The results are correct in every test with --no-mmproj-offload:

    --- vision hparams ---
    load_hparams: image_size:         1540
    load_hparams: patch_size:         14
    load_hparams: has_llava_proj:     0
    load_hparams: minicpmv_version:   0
    load_hparams: n_merge:            2
    load_hparams: n_wa_pattern:       0
    load_hparams: image_min_pixels:   6272
    load_hparams: image_max_pixels:   802816
    
    load_hparams: model size:         837.36 MiB
    load_hparams: metadata size:      0.08 MiB
    alloc_compute_meta: warmup with image size = 448 x 448
    alloc_compute_meta:        CPU compute buffer size =    62.62 MiB
    alloc_compute_meta: graph splits = 1, nodes = 845
    warmup: flash attention is enabled
    main: loading model: ../../text/models/BroadBit/mistral-small-24b-instruct-2506-q4_k_m.gguf
    encoding image slice...
    image slice encoded in 40735 ms
    decoding image batch 1/1, n_tokens_batch = 998
    image decoded (batch 1/1) in 603 ms
    
    The image shows a setup in what appears to be an office or workspace. On a black desk, there are four identical devices lined up in a row. Each device has a sleek, modern design with a prominent white triangle on the front and the word "COLLAIGUE" written below it. The devices appear to be some kind of electronic equipment, possibly speakers or audio devices...
    
    • try running the model without any multimodal support (for example, using llama-cli)

    The model works correctly without multimodal projection (without --mmproj).

    • try running everything on CPU with/without flash attention: -ngl 0 -fa on and -ngl 0 -fa off

    -ngl 0 -fa on and -ngl 0 -fa off are same, first run buggy, second run OK:

    load_hparams: model size:         837.36 MiB
    load_hparams: metadata size:      0.08 MiB
    alloc_compute_meta: warmup with image size = 448 x 448
    alloc_compute_meta:      ROCm0 compute buffer size =    62.62 MiB
    alloc_compute_meta:        CPU compute buffer size =     2.30 MiB
    alloc_compute_meta: graph splits = 1, nodes = 845
    warmup: flash attention is enabled
    main: loading model: ../../text/models/BroadBit/mistral-small-24b-instruct-2506-q4_k_m.gguf
    encoding image slice...
    image slice encoded in 325 ms
    decoding image batch 1/1, n_tokens_batch = 998
    image decoded (batch 1/1) in 3479 ms
    
    üksad0 in the  Q for  in a  in ( in ( in here
     can and### question about interesting for  in
    
    
    
     new in the in ( in (  Q in (  Q in (  Q with (  Q in ( . in (  Q of in (  Q in
    
  19. ngxson commented on Nov 5, 2025

    @ngxson
    CollaboratorAuthor

    @broadbit-hu it seems to be a problem specific to ROCm backend with flash attention, but unfortunately I don't have enough knowledge to fix it. feel free to open a dedicated issue so we can assign to ROCm backend maintainer

  20. broadbit-hu commented on Nov 5, 2025

    @broadbit-hu

    @broadbit-hu it seems to be a problem specific to ROCm backend with flash attention, but unfortunately I don't have enough knowledge to fix it. feel free to open a dedicated issue so we can assign to ROCm backend maintainer

    @ngxson Thanks for you fixes! We'll continue the testing on several AMD GPUs... :)

  21. ggerganov commented on Nov 5, 2025

    @ggerganov
    Member

    @broadbit-hu On the failing machine, run this test and see if it produces any errors:

    test-backend-ops
  22. broadbit-hu commented on Nov 5, 2025

    @broadbit-hu

    @broadbit-hu On the failing machine, run this test and see if it produces any errors:

    test-backend-ops

    @ggerganov No errors while test-backend-ops on 7900 XT and 7900 XTX GPUs with ROCm 6.3.4:

      13185/13185 tests passed
      Backend ROCm0: OK
    Backend 2/2: CPU
      Skipping CPU backend
    2/2 backends passed
    OK
    
    *******                  
    Agent 2                  
    *******                  
      Name:                    gfx1100                            
      Uuid:                    GPU-d6ef127efd298bb5               
      Marketing Name:          Radeon RX 7900 XTX                 
      Vendor Name:             AMD                                
      Feature:                 KERNEL_DISPATCH                    
      Profile:                 BASE_PROFILE                       
      Float Round Mode:        NEAR                               
      Max Queue Number:        128(0x80)                          
      Queue Min Size:          64(0x40)                           
      Queue Max Size:          131072(0x20000)                    
      Queue Type:              MULTI                              
      Node:                    1                                  
      Device Type:             GPU 
    
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions