Skip to content

fix(unlimited-ocr): don't add a second <image> to a prompt that has one - #4822

Merged
jundot merged 1 commit into
jundot:mainfrom
tc3oliver:fix/unlimited-ocr-literal-image-token
Oct 26, 2026
Merged

fix(unlimited-ocr): don't add a second <image> to a prompt that has one#4822
jundot merged 1 commit into
jundot:mainfrom
tc3oliver:fix/unlimited-ocr-literal-image-token

Conversation

@tc3oliver

@tc3oliver tc3oliver commented Oct 24, 2026

Copy link
Copy Markdown
Contributor

Refs #3740

If you send the prompt from the Unlimited-OCR model card, <image>document parsing., with one image, the server returns 500. The prompt patch always prepends <image>, so the processor sees two tokens for one image and its assert fails:

POST /v1/chat/completions → 500 (unhandled): Failed to process inputs with error: The number of image tokens in the prompt does not match the number of images: 2 != 1

This only adds the token when the text doesn't already have one. It doesn't touch the other half of #3740, where extra images get dropped.

I tried it on a local server with mlx-community/Unlimited-OCR-4bit. On main that prompt gives the 500 above. On this branch it gives 200 with 488 prompt tokens and the same output as document parsing. without the token. The new test fails on main with <image><image>document parsing..

Request I used 
import base64, json, urllib.request

img = base64.b64encode(open("page.png", "rb").read()).decode()
body = {
    "model": "Unlimited-OCR-4bit",
    "temperature": 0,
    "max_tokens": 64,
    "messages": [{"role": "user", "content": [
        {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{img}"}},
        {"type": "text", "text": "<image>document parsing."},
    ]}],
}
req = urllib.request.Request(
    "http://127.0.0.1:8000/v1/chat/completions",
    data=json.dumps(body).encode(),
    headers={"Content-Type": "application/json"},
)
print(urllib.request.urlopen(req).status)

@tc3oliver
fix(unlimited-ocr): don't add a second <image> to a prompt that has one
6fb40e8 
The Unlimited-OCR prompt patch prepended <image> to every user message with
images. The model card's documented prompt already starts with <image>, so a
one-image request carried two tokens, failed the processor's token count
assertion and returned HTTP 500.
@tc3oliver

tc3oliver commented Oct 24, 2026

Copy link
Copy Markdown
Contributor Author

The two failing cluster tests pass for me locally. Main's CI on the same commit is red too, on a different timing test: https://github.com/jundot/omlx/actions/runs/40111714156

jundot
jundot approved these changes Oct 26, 2026
View reviewed changes

@jundot jundot left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reproduced the 500 on main with the model card prompt <image>document parsing., and this branch returns 200 with the same output as plain document parsing.. Really appreciate the clear repro and the small fix @tc3oliver. LGTM, merging.

@noonkho this fixes the <image> 500 from #3740. The dropped extra images part is still open.

@jundot
jundot merged commit 285b459 into jundot:main Oct 26, 2026
1 check failed
@tc3oliver
tc3oliver deleted the fix/unlimited-ocr-literal-image-token branch November 13, 2026 05:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jundot jundot jundot approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@tc3oliver @jundot