Skip to content

Releases

ONNX Runtime GenAI 0.20.0

ONNX Runtime GenAI 0.20.0 Latest
Latest

Choose a tag to compare

View all tags
@baijumeswani baijumeswani released this 14 Nov 21:20
v0.20.0
68745ef

ONNX Runtime GenAI 0.20.0

See more. Stream smarter. Put memory to work.

ONNX Runtime GenAI 0.20.0 brings image-aware Engine requests, timestamped streaming speech, and expanded native speculative decoding. Building on 0.19.0, this release opens up new application scenarios while giving developers more control over large-model deployment.

✨ At a glance

What's new What it enables
Vision inputs in the Engine Image-and-text requests through the Engine's request and scheduling APIs
Native Nemotron Bolt execution Target-and-drafter generation through C++, without a Python inference bridge
Expanded Qwen3.8-Flash-Next support DFlash2, ragged batching, GGUF packaging, and context-aware deployment profiles
GGUF-backed ONNX weights Reuse existing weight files and share eligible tensors across sessions
Streaming ASR timestamps Word- and segment-level timing alongside live transcription
GPU memory budgeting Size caches and retained snapshots against a per-model allowance

🖼️ Bring images into the Engine

The Engine now accepts image-and-text prompts in Python, C++, and C. Compatible packages support multiple images, text continuation, and image-aware prefix reuse, with image preparation managed by the Engine.

This release introduces packed Qwen vision support, Qwen3.8-Flash-Next Engine vision, and Ling-3.0-flash-VL export and image inputs.

Try the new image Q&A sample:

python examples/python/engine/vision-qa.py \
  --model ./models/packed-vision-model \
  --image ./cars.jpg \
  --question "Describe the cars in this image." \
  --follow-up "Which car is closest to the camera?"

Requires an Engine-compatible vision package; ordinary Generator vision exports are not interchangeable.

Related changes: #2865 · #2892 · #2898

⚡ Expand native speculative decoding

Nemotron-3.5 Bolt joins the native runtime. Its C++ execution path combines target verification with a LiLiCorr/DFlash2 drafter through the familiar Model/Generator API and a single-slot Engine adapter. Compatible packages also support larger prompt-prefill chunks and bounded prefix reuse.

Qwen3.8-Flash-Next takes a substantial step forward, with:

  • DFlash2 verification with up to seven draft tokens, plus multi-slot and ragged batching.
  • Checkpoint and GGUF package builders, context-tier selection, and expert-offload profiles.
  • Package configurations up to 256K tokens, including profiles for unified-memory NVIDIA GPUs.
  • Bounded native prefix caching and targeted CUDA-graph warmup and input-preparation improvements.

Performance depends on the model package, hardware, and runtime. Native Bolt is a single-request, greedy CUDA backend -- not a continuously batched execution path.

Related changes: #2761 · #2777 · #2822 · #2820 · #2897 · #2907 · #2937 · #2942 · #2965

📦 Keep the weights. Change the packaging.

ONNX models can now load mapped initializers directly from GGUF weight files, avoiding duplicate .onnx.data packaging for those tensors. Eligible target and drafter sessions can also share a single GPU allocation for common weights.

This feature connects GGUF weight storage with ONNX execution; it does not automatically run arbitrary GGUF models. Matching graphs and initializer mappings are required, and direct CPU-buffer borrowing depends on the ONNX Runtime version.

Guide: GGUF initializers · Related change: #2864

🎙️ Give live transcripts a timeline

Nemotron streaming speech recognition now emits word and segment timestamps, alongside incremental transcript text. Updated Python, C++, and C# samples demonstrate the complete flow, backed by public timestamp APIs in C as well.

Enable timing in the model's genai_config.json:

{
  "model": {
    "timestamp_level": "all",
    "segment_separators": [".", "?", "!"],
    "segment_gap_threshold_seconds": 1.0
  }
}

Choose word, segment, or all for word-level navigation, caption segments, or both. Returned time intervals use integer milliseconds.

Sample: Streaming ASR
Guide: Nemotron speech timestamps
Related changes: #2676 · #2930

More models. More deliberate memory use.

New model capabilities include Ling-3.0-tiny/Bailing MoE V3 export and LiquidAI d1-2B text decisions. The d1 sample answers yes/no, choice, and scoring questions with probabilities rather than generated prose -- useful for classification and routing workflows. d1-2B text decisions are supported on CPU and CUDA. (#2843, #2877)

Per-model GPU memory budgets now guide Engine cache sizing using an application allowance and explicitly configured headroom. Native Qwen and Bolt apply these controls to retained GPU snapshots rather than resizing fixed graph caches. Eligible LFM3 text exports also gain window-sized KV caches for sliding-attention layers. These controls improve memory planning; they are not hard allocator quotas. (#2859, #2937, #2942, #2883)

Under the hood: correctness and compatibility

This release also strengthens constrained-generation length enforcement, LongRoPE consistency, Gemma vision processing, and TRT-RTX shared-KV handling for Qwen. Engine completion metadata now reports which EOS token ended a turn, giving applications clearer model-specific termination information. (#2836, #2846, #2779, #2906, #2683, #2895)

Before deploying: Check the relevant model/package guide for ONNX Runtime, CUDA provider, and custom-operator requirements. Some new execution paths require newer runtime kernels or a freshly exported package.

Full changelog: v0.19.0...rel-0.20.0

Assets 18

v0.19.0

v0.19.0

Choose a tag to compare

View all tags
@baijumeswani baijumeswani released this 04 Nov 05:46
v0.19.0
d01b244 
Version 0.19.0
Loading

v0.18.0

v0.18.0

Choose a tag to compare

View all tags
@baijumeswani baijumeswani released this 04 Nov 05:41
v0.18.0
00d66ef 
Version 0.18.0
Loading

v0.17.1

v0.17.1

Choose a tag to compare

View all tags
@baijumeswani baijumeswani released this 04 Nov 05:38
v0.17.1
83de55a 
Release 0.17.1
Loading

v0.17.0

v0.17.0

Choose a tag to compare

View all tags
@baijumeswani baijumeswani released this 28 Sep 06:15
v0.17.0
1d67be3

Version 0.17.0 makes repeated prompts faster, adds new vision and audio models, and brings more ways to tune the Engine for different GPUs.

Highlights

  • Faster repeated prompts with prefix caching - The dynamic Engine can now reuse matching prompt prefixes across requests. Prefix caching is enabled automatically for supported paged KV-cache layouts.
  • Runtime profiles for different GPUs - A single model package can select Engine settings and model variants based on available GPU memory. New capability APIs also make it easier for applications to discover Engine limits.
  • New LFM2 models - Added support for LFM2-VL and LFM2.5-VL vision models, LFM2-MoE, and LFM2-Audio and LFM2.5-Audio with speech input and output.
  • Moonshine streaming speech recognition - Added Moonshine small and tiny streaming ASR models, designed for fast transcription with low CPU use.
  • Faster DFlash decoding - DFlash now supports CUDA graph capture, prepacked weights, optional projection fusion, sampled speculative decoding, and direct Qwen 3.8 package export.
  • Lower GPU memory pressure - Embedding lookups can be offloaded to CPU while keeping CUDA graph capture enabled for decoding.
  • Less work during generation - Paged decoders can compute logits only for the token rows needed for generation and speculative verification.
  • Safer conversation editing - Engine requests can rewind to the start of an earlier turn and continue from there without creating a new request.
  • A better Model Builder configuration - The new structured configuration organizes target, drafter, speculative decoding, runtime, and search settings in one validated format.
  • More Engine flexibility - Added configurable paged block sizes, improved speculative draft controls, and APIs for maximum request length.
  • Reliability improvements - Includes fixes for WebGPU graph capture, decoder pipeline KV-cache shapes, paged prefill settings, legacy execution providers, and prefill logits.
Loading

v0.16.0

v0.16.0

Choose a tag to compare

View all tags
@baijumeswani baijumeswani released this 22 Sep 19:46
v0.16.0
c458a1e

This is a big release! Version 0.16.0 brings major upgrades for high-throughput serving, faster decoding, and the latest generation of Qwen models.

Highlights

  • A much more capable Engine - Continuous batching now supports chunked prefill, paged attention, multi-turn conversations, mixed workloads, and smarter token scheduling. This improves throughput while making better use of GPU memory.
  • Faster generation with speculative decoding - Added support for draft models, n-gram speculation, MTP self-speculation, and DFlash2 and DSpark block drafting.
  • Tool calling and constrained decoding - The Engine can generate tool calls, continue after tool results without replaying the full prompt, and enforce structured output with guidance.
  • Better multi-turn control - Requests can retain their KV cache between turns. Sampling, token limits, seeds, stop strings, and guidance can be changed for each turn.
  • More Qwen support - The model builder now supports Qwen2.5-VL, Qwen3-VL, and Qwen3.5 dense and MoE models, with additional export and speculative-decoding support for Qwen3.6 and Qwen3.8.
  • More efficient KV caches - Added paged, windowed, sparse, and quantized KV-cache support, including INT4, INT8, and FP8 paths.
  • New quantized model formats - Added native support for NVFP4 and FP8 checkpoints from NVIDIA Model Optimizer and compressed-tensors, plus the quant_auto format.
  • New hardware support - Added the AMDGPU execution provider and WebGPU paged-attention exports.
  • New model support - Added Granite models and improved Marian batching, multimodal prefill, Nemotron audio, and multi-image vision handling.
  • Stronger reliability - This release includes broad validation and recovery improvements across the Engine, speculative decoding, vision and audio inputs, cache management, and process shutdown.
Loading

v0.15.2

v0.15.2

Choose a tag to compare

View all tags
@baijumeswani baijumeswani released this 07 Aug 06:06
v0.15.2
ed5f4e8 
Release 0.15.2
Loading

v0.15.1

v0.15.1

Choose a tag to compare

View all tags
@baijumeswani baijumeswani released this 07 Aug 05:58
v0.15.1
3abf4a8 
Release 0.15.1
Loading

v0.15.0

v0.15.0

Choose a tag to compare

View all tags
@baijumeswani baijumeswani released this 30 Jul 16:21
v0.15.0
dfdc254

What's Changed

Read more

Contributors

skottmckay, qjia7, and 25 other contributors
Loading

v0.14.0

v0.14.0

Choose a tag to compare

View all tags
@baijumeswani baijumeswani released this 29 May 18:06
v0.14.0
b7a6ec3

What's Changed

New Contributors

Full Changelog: v0.13.1...v0.14.0

Contributors

skottmckay, qjia7, and 21 other contributors
Loading