Skip to content

mtmd: Add DeepSeekOCR Support #17400

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
sfallah wants to merge 109 commits into
base: master
Choose a base branch
from
Open

mtmd: Add DeepSeekOCR Support #17400

sfallah wants to merge 109 commits into from
+1,843 −40

Conversation

@sfallah
Copy link
Contributor

@sfallah sfallah commented Nov 20, 2025 •
edited
Loading

Feature Request: #16676

Make sure to read the contributing guidelines before submitting a PR

GGUF Models

sabafallah/DeepSeek-OCR-GGUF

deepseek-ocr-f32.gguf

mmproj-deepseek-ocr-f32.gguf

Running the Model

Build llama.cpp (Mac)

cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=Release
cmake --build build -j --config Release

Running llama-mtmd-cli

DeepSeekOCR Paper (First page)
build/bin/llama-mtmd-cli \
-m gguf_models/deepseek-ai/deepseek-ocr-f16.gguf \
--mmproj gguf_models/deepseek-ai/mmproj-deepseek-ocr-f16.gguf \
--image tmp/mtmd_test_data/Deepseek-OCR-2510.18234v1_page1.png \
-p "<|grounding|>Convert the document to markdown." \
--chat-template deepseek-ocr --temp 0
Hard Test (Old Newspaper Image)
build/bin/llama-mtmd-cli \
-m gguf_models/deepseek-ai/deepseek-ocr-f16.gguf \
--mmproj gguf_models/deepseek-ai/mmproj-deepseek-ocr-f16.gguf \
--image tools/mtmd/test-1.jpeg \
-p "<|grounding|>Convert the document to markdown." \
--chat-template deepseek-ocr --temp 0

sfallah and others added 22 commits November 14, 2025 12:40
@sfallah
mtmd: llama.cpp DeepSeekOCR support
43a130b 
init commit
@sfallah
loading sam tensors
b6b9f02
@bluebread
mtmd: fix vision model processing
85c7cda
@sfallah
Merge pull request #1 from bluebread/sf/deepseek-ocr
578c8d7 
mtmd: fix vision model processing
@sfallah
deepseek-ocr clip-vit model impl
2aab52e
@bluebread
mtmd: add DeepSeek-OCR LM support with standard attention
eab28ed
@bluebread
mtmd: successfully runs DeepSeek-OCR LM in llama-cli
7630587
@bluebread
mtmd: Fix RoPE type for DeepSeek-OCR LM.
2de3436
@bluebread
Merge branch 'sf/deepseek-ocr' of github.com:sfallah/llama.cpp into s…
e8b2610 
…f/deepseek-ocr
@sfallah
loading LM
97e0907 
testing Vision model loading
@sfallah
Merge branch 'sf/deepseek-ocr' into sf/deepseek-ocr
13dc6fb
@sfallah
Merge pull request #2 from bluebread/sf/deepseek-ocr
b32bb5e 
mtmd: DeepseekOCR Implement DeepSeek3B-MoE-A570M (LM component)
@sfallah
sam warmup working
790bbb9
@sfallah
sam erroneous return corrected
cec9a5c
@sfallah
clip-vit: corrected cls_embd concat
8b3d319
@sfallah
clip-vit: model convert qkv_proj split
1e08157
@sfallah
corrected combining of image encoders' results
331cea8
@bluebread
fix: update callback for ffn_moe_weighted and add callback for attn_o…
6c0715b 
…ut in deepseek2 model
@bluebread
Merge branch 'sf/deepseek-ocr' of github.com:sfallah/llama.cpp into s…
a65ddf5 
…f/deepseek-ocr
@sfallah
concat image_newline and image_seperator tokens
63a042f
@sfallah
visual_model warmup (technically) works
89afda8
@sfallah
window partitioning using standard ggml ops
88032f4
@github-actions github-actions bot added model Model specific examples python python script changes labels Nov 20, 2025
@sfallah sfallah marked this pull request as draft November 20, 2025 09:12
@sfallah sfallah mentioned this pull request Nov 20, 2025
Open
@ngxson ngxson mentioned this pull request Dec 12, 2025
Merged
@ngxson
Copy link
Collaborator

ngxson commented Dec 12, 2025

heads up, sorry for the breaking change but there will be a refactoring (just moving stuff around) in #17965

after finishing with this refactoring (and after you done testing on your side), I'll go back to deepseek-ocr

@sfallah
Copy link
Contributor Author

sfallah commented Dec 13, 2025

@ngxson

heads up, sorry for the breaking change but there will be a refactoring (just moving stuff around) in #17965

after finishing with this refactoring (and after you done testing on your side), I'll go back to deepseek-ocr

Merge with #17965 is done.
I have also added deepseek-ocr to tests.sh.
As far my tests goes, it works, but the python test script is not done yet.
I will finish the test script tomorrow.

@bluebread bluebread mentioned this pull request Dec 14, 2025
Merged
@sfallah
minor formatting fixes
6c36c03
@sfallah
check with fixed expected resutls
dc2066e
@sfallah
Merge pull request #10 from sfallah/sf/deepseek-ocr-test-script
3fc61d4 
python test script for deepseek-ocr
testing OCR on text-1.jpeg newspaper image
checking against expected reference model output for Free-OCR and Markdown
@sfallah
minor formatting
7f8621c
@sfallah
Merge remote-tracking branch 'sfallah/master' into sf/deepseek-ocr
b3bf8cb 
# Conflicts:
#	convert_hf_to_gguf.py
@sfallah
editorconfig-check fix
8ad98ee
@sfallah sfallah requested review from CISC and ngxson December 16, 2025 06:20
@sfallah
Merge branch 'ggml-org:master' into sf/deepseek-ocr
4a4f829
@sfallah
Merge remote-tracking branch 'sfallah/master' into sf/deepseek-ocr
51c3de6 
# Conflicts:
#	gguf-py/gguf/constants.py
#	gguf-py/gguf/tensor_mapping.py
#	tools/mtmd/clip-impl.h
#	tools/mtmd/clip.cpp
#	tools/mtmd/models/models.h
@sfallah
merge with changes from ggml-org#18042
512b2c8
@sfallah
Merge remote-tracking branch 'sfallah/master' into sf/deepseek-ocr
00d2357 
# Conflicts:
#	src/llama-arch.cpp
@sfallah
minor
87e4a00 
- added GLM-4.6V to big tests
- added missing deps for python test
CISC
CISC reviewed Dec 16, 2025
View reviewed changes
CISC
CISC reviewed Dec 17, 2025
View reviewed changes
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@CISC CISC CISC left review comments

@ggerganov ggerganov Awaiting requested review from ggerganov ggerganov is a code owner

@ngxson ngxson Awaiting requested review from ngxson ngxson is a code owner

+3 more reviewers

@bluebread bluebread bluebread left review comments

@rhjdvsgsgks rhjdvsgsgks rhjdvsgsgks left review comments

@kmk142789 kmk142789 kmk142789 approved these changes

Reviewers whose approvals may not affect merge requirements

Requested changes must be addressed to merge this pull request.

Assignees

No one assigned

Labels

examples ggml changes relating to the ggml tensor library for machine learning model Model specific Nvidia GPU Issues specific to Nvidia GPUs python python script changes

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@sfallah @ngxson @bluebread @gitl33 @CISC @rhjdvsgsgks @kmk142789