You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 0fb858c
Browse filesBrowse the repository at this point in the historyBrowse files
authored
Support Gemma 4 12B Unified native multimodal generation (#2685)
## Summary
Continue Justin's original #2286 from an
upstream development branch, as requested by the repository owner. The
original PR was closed on October 7, 2026 in favor of this replacement.
- Recognize/register `gemma4_unified` and wire native image/audio
preprocessing into multimodal generation.
- Preserve padded 6912-dimensional image patches and positions; consume
raw 640-sample audio frames and masks, counting each frame as one audio
soft token.
- Correct INT64-to-INT32 image-position conversion and test native
generation/tensor contracts.
- Reject unsupported image-position dtypes before processing; accept
only INT32 and INT64, with explicit conversion branches.
- Match the Unified dummy speech output to the production rank-two
`[num_valid_frames, hidden_size]` contract and test its native embedding
handoff.
- Stabilize the dummy vision reduction: accumulate in DOUBLE, retain
FLOAT outputs, and match the NumPy float64-then-float32 oracle without
loosening tolerance.
- Make the Unified dummy audio decoder explicitly unlimited, consistent
with the exporter.
- Replace the contributor-fork dependency with the official Microsoft
Extensions URL and immutable landed commit; synchronize its component
manifest.
Current head: `ef8d5a4bccc1865b8b47fc6657024f07f22607d3`.
Companion preprocessing: microsoft/onnxruntime-extensions#1132.
Companion exporter: onnxruntime/mobius#759.
## Latest scoped validation
- Fresh normal CPU dependency build covering the two latest review
fixes, without an Extensions source override: downloaded
`https://github.com/microsoft/onnxruntime-extensions.git` at its landed
squash-merge commit `99fec2db1c0a17de60eb0f1c2353c91f8d5e22ca`,
containing the landed trust-store CI fixmicrosoft/onnxruntime-extensions#1133.
- Imported the freshly rebuilt GenAI package and ran standard, pipeline,
and Unified Gemma4 Python cases: **40 passed, no skips**. This includes
rank-two speech output/embedding handoff, supported INT32/INT64 image
positions, and four unsupported-dtype cases (standard/Unified x
FLOAT/FLOAT16). The latter construct a valid model first and check that
multimodal processor creation rejects the unsupported input dtype.
- Ruff and Black checks passed for the modified Python fixture/tests;
targeted clang-format checks passed for the C++ dtype guard. Dependency
and component-manifest URL/SHA agree exactly.
- Earlier rebase onto official `main` at
`45e972d2e09ba41ca59b377e0a819e013129135f` preserved the new upstream
pipeline fixtures/tests. The PR's existing modality-signal helpers were
given distinct names with matching Unified imports; upstream helper ASTs
and renamed PR helper ASTs were checked against both original conflict
sides.
- Companion Extensions full preprocessing suite: **190 passed, 2
disabled**; seven focused LeakSanitizer tests passed with no leak report
after fixing partial-batch cleanup.
- Companion Mobius exporter suite: **210 passed**.
## Real-model CPU evidence: bounded smoke, not dataset quality
A fresh real `google/gemma-4-12B` FP32 export and native CPU E2E run
were completed on October 8, 2026 at the earlier GenAI head
`976ae3989ad9f13bf623535ef1e12b6584eac329`, against the same official
landed Extensions commit `99fec2db1c0a17de60eb0f1c2353c91f8d5e22ca`. The
clean exporter checkout was tree-equivalent to the landed
onnxruntime/mobius#759. The checkpoint revision was
`023679ed352de9bb66cc873c9009ce3482585c08`.
| CPU FP32 case | Bounded execution result |
|---|---|
| Text only | Eight generated tokens; finite raw logits |
| Image | Native vision -> embedding -> decoder; eight generated tokens;
finite raw logits |
| Two-second audio clip | 50 frames retained, all masks and audio_sizes
checked; eight generated tokens; finite raw logits |
| 31-second synthetic audio with a final marker | All 775 frames and the
final marker retained; eight generated tokens; finite raw logits |
All active provider-bearing node events in these four cases used
`CPUExecutionProvider`. Export/checkpoint/configuration hashes were
checked before and after execution.
**Search-policy qualification:** when the raw greedy top token was
structural end-of-image/end-of-audio ID 258882 or 258883, both IDs were
suppressed after checking the raw logits. This occurred in the text,
image, and long-audio cases, not the short-audio case. These are
therefore **not unmodified-greedy quality results**. Text/image
semantics were not scored; the long synthetic clip generated eight
spaces and is not ASR-quality evidence. Timings include
startup/loading/profiling and are not a performance benchmark.
The two subsequent commits correct the rank-two test fixture and reject
unsupported position dtypes; they do not change generation/search logic.
Their latest-head coverage is the fresh 40-test CPU suite above, not a
claim that this four-case real-model run was repeated at `ef8d5a4`.
Earlier real audio-boundary smoke also exercised CPU FP32 and
CUDA-configured FP16 with a two-second clip and a 31-second synthetic
constant/marker clip, retaining 50/775 frames and finite raw logits.
Those runs preceded the final Extensions ownership-only cleanup, which
is covered separately by LeakSanitizer and full preprocessing tests.
CUDA decoder profiles included CUDA and CPU events; child-session CUDA
execution was unproven, and disabling CPU fallback rejected
construction. Two FP32 decoder constants remained in the FP16 audit.
Pure-CUDA execution and an entirely FP16 graph are not claimed.
## Historical dataset integration and quality evidence
Full original summary, source revisions, and qualifications:
microsoft/onnxruntime-extensions#1091 (comment)
These are **historical GPU dataset measurements**, retained as
supporting integration evidence, not fresh CPU dataset results or a
rerun at the current head. The earlier GenAI head
`eacf6b6121528728108346e847be7350d2d3f535` also passed 23 focused Python
tests, a fresh dependency build, and clean-Mobius real-model
text/image/audio smoke.
| Earlier bounded E2B main/main vs PR/PR comparison, same FP16 ONNX
package | Result |
|---|---|
| First 100 AI2D samples | Identical answers/tokens/tensor digests; 58%
accuracy both |
| First 100 FLEURS samples | Identical transcripts/tokens/tensor
digests; normalized WER 9.4595% both |
The historical **12B Unified base-checkpoint** comparison used 100
samples per dataset:
| Metric | Native preprocessing -> FP16 ONNX | HF preprocessing -> same
ONNX | HF PyTorch BF16 |
|---|---:|---:|---:|
| AI2D strict single-digit accuracy | 50% | 51% | 50% |
| AI2D evaluator-normalized accuracy | 61% | 59% | 59% |
| FLEURS normalized WER (lower is better) | 18.13% | 17.24% | 18.59% |
Qualifications for the historical 12B comparison:
- The harness suppressed structural IDs 258882/258883 uniformly across
all three generation paths. These are controlled quality comparisons
under that shared search policy, not unmodified generation or
exact-parity claims.
- HF prompts explicitly included the media-boundary newlines that the
native processor inserts automatically. Input IDs were not copied or
overwritten to force agreement.
- The BF16 Torch reference used eager attention. An eager-attention mask
issue was found in the separate E2B investigation; the historical 12B
reference still needs a backend cross-check before being treated as
definitive.
- Empty outputs and generation-cap hits were retained. These are fixed
100-sample subsets, not full-dataset release qualification.
- Native/HF image preprocessing values differ; the known image
discrepancy remains tracked in microsoft/onnxruntime-extensions#1131.
**No completed CPU 100-AI2D/100-FLEURS quality result is claimed.** A
newer CPU FP32 dataset attempt was stopped after identifying an
HF/native media-prompt layout mismatch in the harness. Its preliminary
output differences are excluded from the quality/regression conclusions.
We are retaining the historical dataset results and separately labeling
the CPU smoke and latest-head integration tests rather than presenting
them as a new full CPU evaluation.
## Dependency / merge order and CI
The current dependency is the official Microsoft Extensions repository
at the landed-main commit `99fec2db1c0a17de60eb0f1c2353c91f8d5e22ca`.
microsoft/onnxruntime-extensions#1132 was squash-merged at 2026-10-08
16:32:22 UTC. Its landed commit and the previously validated PR head
`6147e57f0def4627fb712bca1850d39480e5382c` have identical Git trees. The
dependency and component manifest were updated together; the original
landed-pin build passed 36 focused tests, and the latest review-fix
build passed 40 against the same final pin.
The Extensions landed-commit dependency gate is satisfied. This PR still
requires its normal reviews and required CI; no merge was performed.
Local x86 results do not establish Windows ARM64 success: the DOUBLE
dummy-vision change targets the observed ARM64 reduction discrepancy,
which remote CI must confirm.
Task-created large model/checkpoint/dataset/cache/build assets are
cleaned up after preserving small reports, configuration snapshots,
hashes, and reproduction evidence. No full-dataset, performance, or
release requalification is claimed.
---------
Signed-off-by: Justin Chu <[email protected]>
Co-authored-by: Justin Chu <[email protected]>
Co-authored-by: Ti-Tai Wang <[email protected]>
Co-authored-by: Copilot <[email protected]>
0 commit comments