Repository navigation
Releases
ONNX Runtime WebGPU Plugin EP v0.6.0
This release expands model and operator support, improves performance and numerical stability, and adds provider configuration and shader-cache controls.
Highlights
Model and operator support
- Added
Or,Xor,Round,Sign,IsNaN,IsInf,Softsign,LpPool, andGlobalLpPoolkernels. (#33680, #33679) - Added
QuantizeLinearwith signed and unsigned 8-bit and 4-bit outputs, and 4-bitReshape. Supported opsets andprecisionrestrictions apply. (#33683, #33792) - Added FP16 support for
GRUandLSTM, int32 support forRelu, and int8 support forGather. (#33678, #33789, #33790) - Added signed int8 and int64 support for
Abs,Neg,Relu, andSign. Int64 kernels require theenableInt64provider option. (#33958) - Added
MatMulBlockQuantizedFp8Weightsupport for FP16 activations, E4M3FN FP8 weights, and FP32 block scales. (#33074) - Added automatic
MatMulNBitsQLoRAfusion for supported models, using 4-bit block-32 base weights and 8-bit block-32 rank-32 or empty adapters.MatMulNBitsalso supports empty packed weights. (#33370, #33559) - Extended
GatedDeltaNetto support per-key-dimension decay. (#33817) - Added
ConvStateReplayandGatedDeltaNetStateReplayfor reconstructing speculative-decoding state. These operators do not support graph capture;GatedDeltaNetStateReplayuses FP32 state and capsule backing. (#33493, #33213) - Added
PackedSparseAttentionIndexerMergeto combine sparse-attention index sets. (#33224)
Performance and configuration
- Added an Intel-specific dense flash-attention prefill path for unquantized KV caches. (#32702)
- Added subgroup-matrix acceleration for eligible Intel FP16, group-1, channels-last convolutions. Other configurations use existing paths. (#32252)
- Improved
MatMulandMatMulNBitspaths, including NVIDIA and AMD subgroup-matrix kernels and Qualcomm DP4A tuning. (#33639, #32796, #33560) - Reduced transfer overhead through batched input uploads, direct queue writes for eligible small CPU-to-GPU uploads, and batched compatible streamless GPU-to-GPU copies. (#33491, #33551, #33567)
- Extended the opt-in
enableMatmulFp32Accumulationoption to FP16MatMulandGemm. It remains off by default; enabling it can bypass FP16-only subgroup-matrix paths. (#33057, #32630) - Added per-session
enableFeaturesanddisableFeaturesoptions, initially supportingsubgroupMatrix. Disabling it selects existing fallback kernels. (#33481) - Added native
dawnCacheInterfacecallbacks for caching Dawn shader and pipeline blobs, not model weights. Configure the interface before ORT creates the device; it cannot be attached to a host-supplied device. Later sessions may inherit the shared context's cache configuration. (#33936, #33999)
Correctness and packaging
- Improved numerical stability for
LayerNormalization, FP16 reductions,InstanceNormalization, and several elementwise functions. (#33365, #33682) - Fixed graph capture and replay of GPU buffer copies and explicit buffer clears. (#33362, #33509)
- Synchronized Session command recording across execution, graph replay, and CPU I/O operations. Applications must still avoid overlapping I/O binding and
Runon the same Session. (#33763) - Fixed shader-cache collisions across operator variants and a deferred pipeline callback lifetime issue after failed waits. (#33242, #33209)
- Fixed scalar-bias
Gemm, vectorizedPowbroadcasting, three-channelConv2dMMindexing, and empty pooling outputs. (#33664, #33348, #33432, #33668) - Bundled the preview D3D12 Agility SDK in Windows Python wheels, the NuGet package, and Foundry Local zip packages. (#32795)
Runtime Compatibility
The minimum ORT version is 1.24.4. Concurrent independent Sessions are enabled on ORT 1.28.3 and later in the 1.28 series, 1.30.1 and later in the 1.30 series, and 1.31 or later. Other supported versions, including 1.29.x, use serialized compatibility. In serialized mode, callers must serialize all WebGPU operations on the same device, including Session creation/destruction, allocation, I/O binding, transfers, and Run; the plugin does not enforce this serialization. Concurrent mode does not permit overlapping I/O binding and Run on the same Session (#33082).
Known compatibility caveat: ORT 1.26.0 was reported to reject the plugin during kernel creation because of its host-side plugin-version check. Forcing serialized execution does not resolve that restriction (#33082).
Activating the bundled Windows preview Agility SDK requires a compatible Windows version and loader update, Developer Mode, and a compatible GPU driver. See the Agility SDK guidance.
Contributors
Thank you to @aciddelgado, @adrastogi, @apsonawane, @baijumeswani, @daijh, @edgchen1, @GopalakrishnanN, @gramalingam, @gyagp, @jchen10, @Jiawei-Shao, @justinchuby, @keshavv27, @kunal-vaishnavi, @kylo5aby, @mei1127, @mohdmahmodi, @mustjab, @prathikr, @qjia7, @sayanshaw24, @SharmaRithik, @sushraja-msft, @tianleiwu, @xenova, @xhcao, @xiaofeihan1, and @ykhrustalev.
These notes focus on ONNX Runtime WebGPU Plugin EP changes since 0.5.0; contributor acknowledgments also cover commits touching shared WebGPU EP code.
These notes were prepared with AI assistance.
ONNX Runtime CUDA Plugin EP 0.4.0
This release brings CPU/GPU hybrid expert offloading, native GGUF/block-packed weight execution, Tensor-Core
INT2/INT3/INT4 and offline K-packed NVFP4 QMoE kernels, long-context and speculative-decoding acceleration, and
reliability fixes to the ONNX Runtime CUDA Plugin EP. Highlights include session.moe.offload.* expert offloading for
Mixture-of-Experts models that exceed GPU memory, an opt-in persistent MatMulNBits tactic cache, new fused LLM
operators, ONNX opset 28 kernels, and expanded Linux aarch64 CUDA 13 GPU coverage.
Compatibility
The minimum ONNX Runtime core version remains 1.24.4, but meeting that minimum alone does not guarantee
compatibility for contributed operators. Core and the plugin must use the same contributed-operator schemas;
building both from the same ONNX Runtime revision is recommended. Schema mismatches can cause incorrect execution
or crashes and may not be detected when registering the plugin (#32855).
The matched ONNX Runtime version is 1.32.0 for this release.
- Features that require ONNX Runtime 1.32.0 core: CPU/GPU MoE offload (
session.moe.offload.*), which relies on new core session-state support to keep expert initializers on the CPU; pipelined external-weight loading for plugin EPs; and the device memory capacity query (OrtApi::EpDevice_GetMemoryCapacity). Older runtimes ignore the plugin's newOrtEpFactory::GetDeviceMemoryCapacitycallback, so it does not raise the 1.24.4 floor (#33451, #33839, #33447). - New operators and opset 28: Models that use the contributed operators added in this release, or ONNX opset 28 (ONNX 1.23.2), need a core that registers those schemas, such as the matched 1.32.0 release (#33134).
- Not available in the plugin: The opt-in EP-owned runtime fusions added to the bundled CUDA EP in ONNX Runtime 1.32.0 (
ep.cuda.runtime_fusion.unit_swishandep.cuda.runtime_fusion.causal_conv_with_state) are not offered by CUDA plugin builds (#33757, #34021). - Capacity-aware partitioning: With ONNX Runtime 1.32.0 core and
session.resource_cuda_partitioning_settings, the core declines to assignGroupQueryAttentionnodes to the plugin because route-aware workspace estimates cannot yet pass through the plugin accounting path. A warning is logged and those nodes are left for other EPs. Sessions without capacity-aware partitioning are unaffected (#33016).
Highlights
CPU/GPU Hybrid MoE Offload
- Added portable, session-scoped CPU/GPU expert offload for standard
com.microsoft::QMoEnodes, configured entirely throughsession.moe.offload.*session options with no model changes.session.moe.offload.mode="auto"keeps every expert on the GPU when it fits and otherwise enables hybrid offload;"required"fails session creation when a node cannot be admitted. The planner budgets per-layer GPU expert slots (optionally guided by anexpert-profile.binpopularity profile viasession.moe.offload.popularity_file), runs cold experts on the CPU during decode through a doorbell path that works with CUDA Graph replay, streams cold experts to the GPU in waves during prefill, and promotes newly hot experts between steps with the defaultadaptiveplacement (#33451, #33452). - Supported NVFP4 (
weights_prepacked=1), SM80-prepacked INT4 (block_size=32) and INT2, and raw INT3 expert banks, including mixed INT4/INT3/INT2 models, on SM80 and newer GPUs, with Tensor-Core INT3/INT4 offload decode and grouped prefill (#33575, #33622, #33629, #33731, #33787). - Extended hybrid offload to GGUF block-packed
MoEPackedexperts, with a three-stage pipelined prefill and multi-token cold-route execution (#33976, #33973, #34067, #34114). - Added AVX-512 and AVX2+FMA+F16C cold-expert CPU kernels that read the same INT4/INT3/INT2/FP4 weight layout as the GPU without repacking, and memory-mapped expert storage with an optional bounded host cache (
session.moe.offload.host_expert_bytes) for hosts with limited RAM (#33450, #33624, #33785, #33935). - Improved offload on consumer GPUs and Windows with WDDM-aware VRAM budgeting that avoids spilling into shared GPU memory, Windows memory-mapping and pinned-memory accounting fixes, working-set retention for hot experts, ready-first scheduling of cached cold experts, and an opt-in zero-cold doorbell short-circuit (
session.moe.offload.skip_zero_cold_doorbell=1) (#33916, #33935, #33699, #33624, #33629). - Added opt-in approximate routing for offloaded decode: tail-expert pruning below a routing-probability threshold (
session.moe.offload.min_expert_weight; by default only CPU-cold tail experts are pruned) and substitution of cold tail experts with GPU-resident experts (session.moe.offload.tail_substitution_thresholdandsession.moe.offload.tail_substitution_min_rank). Both are disabled by default and change routing results, so validate model accuracy before enabling them (#33916, #33944). - Added
session.moe.offload.gpu_device_mem_limit, a per-session cap on all device allocations made through the session's CUDA allocators (unlikegpu_mem_limit, which caps only the BFC arena), andsession.moe.offload.budget_groupfor sharing one budget between target and draft sessions on the same GPU. The limit is not supported with external CUDA allocators or CUDA memory pools (#33451). - Added
ep.cuda.qmoe_in_warmupandep.cuda.qmoe_prefill_new_turnrun-config entries so applications can exclude warm-up runs from routing statistics and reset prefill state at the start of a conversation turn (#33452, #33622).
Quantized MoE Kernels
- Added support for offline K-packed NVFP4 QMoE weights (
weights_prepacked=1), shared by the GEMV, grouped-GEMM, and dense fallback paths, with a register-decoded decode GEMV, single-CTA fused softmax/top-k routing for 256 and 512 experts, a single-pass finalize kernel, and programmatic dependent launch (PDL) overlap on SM90 and newer. Native E4M3 block-scale conversion is available as an opt-in withORT_NVFP4_GEMV_NATIVE_SCALE=1(#33094, #33354, #33366, #33545, #33323). - Enabled fused block-scaled FP8 QMoE execution by default on SM80 and newer (
ORT_ENABLE_FP8_FUSED=0disables it), and vectorized the block-FP8 decode GEMV and prefill weight dequantization (#33094, #33817). - Added fast INT3 and mixed-width
(FC1, FC2)∈ {2, 3, 4} × {2, 3, 4} QMoE kernels, offline SM80-prepacked INT2/INT4 weights (weights_prepacked=1), and BF16 Tensor-Core decode GEMVs for INT4, INT3, and INT2 experts. The Tensor-Core GEMVs are enabled by default;ep.cuda.disable_moe_gemv_tc=1(INT4/INT2) andep.cuda.disable_qmoe_int3_tc=1(INT3) restore the SIMT kernels, andep.cuda.moe_gemv_tc_min_rowsandep.cuda.qmoe_int3_tc_min_routestune the dispatch thresholds (#33569, #33585, #33689, #33714, #33786). - Added opt-in grouped BF16 INT3 prefill (
ep.cuda.qmoe_int3_grouped_prefill=1), with optional Tensor-Core compatibility rounding (ep.cuda.qmoe_int3_tc_compatibility=1) (#34117). - Extended fused routing to multi-token batches (S ≤ 8) with up to 512 experts, and added profile-free prefill tactic selection for deterministic MoE (
ORT_FORCE_DETERMINISTIC_MOE=1), including INT4 with an Ampere tile fallback (#33596, #33367, #33595). - Added tanh-GELU gating (`activation_type="geglu"...
ONNX Runtime v1.32.0
ONNX Runtime 1.32.0 reintegrates ONNX 1.23.2 with opset 28 support, introduces CPU/GPU hybrid MoE offloading and shared-layout CPU QMoE kernels, expands device-memory, weightless-compilation, and Python model-package APIs, dramatically accelerates session initialization on large graphs, and strengthens runtime and optimizer reliability. These notes cover changes since ONNX Runtime 1.31.1. CUDA and WebGPU kernel updates are summarized here; detailed provider notes are covered by their separate plugin EP releases.
Highlights
- Reintegrated ONNX 1.23.2 with opset 28 operator support (
SpaceToDepth,Mod, signedBitShift, andBFloat16extensions) and updated DLPack to v1.3 across all implemented ORT numeric and boolean tensor types (#33134, #32362, #33848). - Added a session-scoped CPU/GPU hybrid MoE offload planner and execution backend for quantized
QMoE(NVFP4, INT4, INT3, and INT2) and block-packedMoEPackedmodels, backed by shared-layout AVX-512/AVX2 CPUQMoEkernels and memory-mapped expert storage (#33450, #33451, #33452, #33973). - Expanded core APIs with
OrtApi::EpDevice_GetMemoryCapacity, explicit multi-mode weightless compilation (ModelCompilationOptions_SetWeightlessModeandep.enable_weightless_mode), Python bindings for the Model Package APIs, and zero-copy CPU buffer borrowing inAddExternalInitializers(#33447, #32899, #33640, #33877). - Eliminated superlinear bottlenecks in allocation planning, constant sharing, graph I/O mapping, and topological sorting—reducing session initialization on 100k-node constant-heavy graphs from minutes to seconds—and added pipelined pinned-memory staging for external GPU weight loading (#33407, #33939, #34009, #33839).
- Expanded shared generative AI and quantized contrib operators, including Kimi Delta Attention (
GatedDeltaNetper-key-dimension decay), right-padded chunked prefill inGroupQueryAttention, native GGUF/block-packed operators (GatherPacked,MatMulPacked,MoEPacked), and sparse-attention index sharing (#33817, #33503, #33953, #33224).
Announcements & Compatibility
- CUDA Plugin EP v0.4.0 and WebGPU Plugin EP v0.6.0 are matched with ONNX Runtime 1.32.0 (#33214). For contrib-op compatibility, core and the plugin must use the same contributed-operator schemas; building both from the same ONNX Runtime revision is recommended. Schema mismatches can cause incorrect execution or crashes and may not be detected during plugin registration.
- ACL EP removed: Following its deprecation in 1.31.0, the Arm Compute Library (
ACLExecutionProvider) execution provider, its build options, and its standalone C registration API have been removed. Python'sget_all_providers()no longer listsACLExecutionProvider; applications should remove it from explicit provider lists (#33851). - Legacy provider-bridge
NvTensorRTRTXExecutionProviderremoved: In-tree provider-bridgeNvTensorRTRTXExecutionProvidersources have been removed in favor of the standalone NVIDIA TensorRT RTX plugin EP registered viaOrtApi::RegisterExecutionProviderLibrary.NvTensorRtRtxProviderOptionsis retained as a deprecated declaration for source compatibility, and legacyAppendExecutionProvidercalls now report a deprecation error directing callers to the plugin EP (#32276). - ONNX 1.23.2 integration: ONNX has been upgraded to 1.23.2, raising supported opset ceilings to 28 and restoring
ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT6E2M3andONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT6E3M2public element-type mappings andCastconversions (#33134). - DLPack v1.3 boolean contract change: DLPack has been upgraded to v1.3 with native
kDLBoolmetadata and bidirectional support for all implemented ORT numeric and boolean types, including all fivefloat8formats, packedfloat4,int4/uint4, andint2/uint2(float6remains unsupported in DLPack). The low-levelis_bool_tensoroverride was removed, anduint8DLPack metadata now always imports asuint8(#33848). - Weightless compilation API migration:
OrtCompileApi::ModelCompilationOptions_SetWeightlessEnabledand theep.enable_weightless/ep.enable_weightless_ep_context_nodessession options are deprecated in favor ofModelCompilationOptions_SetWeightlessModeandep.enable_weightless_mode("0"=NONE,"1"=EXTERNAL_ONLY,"2"=ALL). EPs can now advertise multi-mode capability viaOrtWeightlessSupport_ALL_OR_EXTERNAL_ONLYandweightless_supported_modes(#32899). - Training CI retirement: Training build and test lanes were retired from active GitHub Actions CI workflows while retaining inference build coverage (#33836).
- This is the last release for CocoaPods, and we will switch to Swift Package Manager.
- We will use combo onnxruntime-gpu python package to reduce storage size from next release. You can test it here
New Features
Core APIs & Runtime
- Added
OrtApi::EpDevice_GetMemoryCapacity(C++EpDevice::GetMemoryCapacity()) andOrtEpFactory::GetDeviceMemoryCapacityto query device-global total memory and estimated currently available memory (DEFAULTorHOST_ACCESSIBLE), with built-in and plugin CUDA EP support (#33447). - Expanded the app–EP weightless contract so EPs advertise all supported weightless modes (
weightless_supported_modes), applications request an explicit mode (ModelCompilationOptions_SetWeightlessMode/ep.enable_weightless_mode), and compiled EPContext models recordweightless_modeandexternal_dependencies(source_model,external_weights,ep_context_binary) in model metadata (#32899). - Extended
session.use_external_initializer_file_buffers_directly="1"toAddExternalInitializersfor zero-copy borrowing of naturally aligned CPU buffers larger than 127 bytes on little-endian hosts, and documented the read-only guarantee for external initializers added viaOrtModelEditorApi::AddInitializerToGraphwithdata_is_external=true(#33877, #33717). - Added lock-free multi-worker pinned-memory staging (
session.external_data_loader.*) to accelerate external initializer loading into GPU memory across built-in and plugin CUDA EPs (#33839). - Added a session-scoped MoE/QMoE CPU/GPU offload planner and memory-budget ledger (
session.moe.offload.*), genericsession.kernel_dispatch_policy(auto,latency,memory,safe) configuration plumbing, Level-1 workspace/prepack memory accounting fallback for nodes absent from memory-profile CSVs, and internal capability-planning protocol types (#33451, #33042, #33311, #33841). - Enabled cross-thread reuse of idle device streams when graph capture is disabled and providers register device-selection callbacks, and enabled compile-session profiling (
<profile_file>_compile) inonnxruntime_perf_testwhen-pis used with--compile_ep_context(#33053, #33694). - Made
onnxruntime_DISABLE_EXCEPTIONS(--disable_exceptions) usable withoutonnxruntime_MINIMAL_BUILDand added theonnxruntime_DISABLE_STREAM=ONCMake option to disable device-stream support in non-minimal builds (#33120, #33482).
ONNX Opset 28 & Shared Contrib Operators
These updates affect core ONNX and contributed operator schemas and cross-provider behavior, not only the separately released GPU plugins.
- Added ONNX opset 28 operator support: native
SpaceToDepth(DCRandCRD) on CPU and CUDA with version-split registrations across DML, JS, WebGPU, and QNN; opset-28 floating floor-modulusModon CPU, CUDA, and DML; opset-28BFloat16support forCompress,OneHot,ReverseSequence, `...
ONNX Runtime v1.30.1
This is a patch release on top of v1.30.0, containing targeted model-validation and memory-safety fixes, improved Plugin EP data transfers, and build and packaging fixes.
Highlights
Bug Fixes
- Hardened contrib-op shape inference with additional rank, attribute, initializer, and overflow validation for
CausalConvWithState,GatedDeltaNet,GroupQueryAttention, andSparseAttention(#32607). - Prevented out-of-bounds reads in
EmbedLayerNormalizationshape inference with scalar beta,ConvTransposeWithDynamicPads, and shape-inference input-data lookup (#32610, #32679, #32681). - Rejected malformed graphs containing inputs to zero-input operator schemas or subgraphs on non-control-flow nodes (#32633, #32641).
- Prevented an out-of-bounds read during session initialization when a
LooporScanbody input is also an initializer, rejecting the invalid graph instead of crashing (#32609). - Fixed a heap buffer overflow caused by reusing packed sub-byte tensor storage for a larger full-byte tensor (#32611).
- Preserved the caller's execution stream in Plugin EP single-tensor data transfers, allowing asynchronous copies to receive the supplied stream while retaining synchronous-copy behavior (#32666).
Build and Packaging
- Fixed Windows build-tools setup to use system-trusted certificates and improved proxy handling (#33142).
- Selected Python 3.12 explicitly before package validation in the combined packaging stage (#33561).
- Restricted FP8 XQA test cases to supported GPU compute capabilities (#32559).
Contributors
Thanks to our 8 contributors for this release!
@adrastogi, @bbernhar, @eserscor, @miaobin, @shiyi9801, @tianleiwu, @wangw-1991, @xiaofeihan1
Full Changelog: v1.30.0...v1.30.1
These release notes were drafted with AI assistance.
ONNX Runtime WebGPU Plugin EP v0.5.0
ONNX Runtime WebGPU Plugin EP 0.5.0 expands attention, mixture-of-experts, and quantized operator support, improves GPU kernel performance, and strengthens runtime integration and correctness.
Highlights
Performance
- Enabled subgroup-matrix kernels on AMD GPUs (#32667).
- Added an FP16 GEMV specialization for single-row decode projections (#33070).
- Improved data movement in
Concat,Slice,Transpose, and pooling, and vectorized NHWC depthwise convolution across channels (#32497, #32498). - Enabled local-window flash attention for
GroupQueryAttention(#32462). - Optimized recurrent and chunked
GatedDeltaNetexecution (#32645, #33143). - Optimized sparse attention indexers and
SparsePagedAttentionsoftmax, and cached sparse-indexer QSA visibility and scores (#32613, #32656, #33017).
Operator and Model Support
- Added native
GatedDeltaNetandVarlenCausalConvWithStatekernels, including subsequent state-update-capacity support forGatedDeltaNet(#32510, #32477, #32767). - Added
SparseAttentionIndexer,DynamicSparseAttention, andSparsePagedAttention, plus a packed sparse attention indexer for continuous batching (#32528, #32529, #32533, #32618). - Extended
PagedAttentionwith full-context and local non-causal attention, and per-head Q/K normalization (#32763, #32792, #32808). - Added unquantized
MoEand expandedQMoEactivation, gating, routing, and packed-input support (#32447). - Added block-scaled FP8
QMoE, INT2 expert weights, and mixed-width integerQMoEexecution (#32794, #32878). - Added Qwen4-Exp Engram support and
VarlenNGramHashMappingfor packed and variable-length batching (#32285, #32358, #32467). - Extended
GatherBlockQuantizedto FP8 and FP4 data. Out-of-range indices now produce zero output slices for both floating-point and existing integer formats (#32480). - Added
GroupNorm,SkipGroupNorm, and 3DConvTransposesupport (#32463, #32688). - Added weightless hyper-connection primitives:
BranchwiseRMSNorm,ScaledSiLU,HyperConnectionPreMix, andHyperConnectionPostMix(#32687). - Added SiLU/Swish/Sigmoid activation modes to
GatedRMSNormand QuickGelu fusion in the im2col-matmul convolution path (#32512, #32560).
Runtime Integration and Configuration
- Added
adapterIndexto select a physical Dawn adapter independently of the ORT WebGPU context identifier,deviceId(#32319). - Added the opt-in
enableMatmulFp32Accumulationprovider option forMatMulNBitsand its fused variants; the default remainsfalse(#29599). - Added concurrent independent-Session execution on eligible ORT hosts, with Session-owned command recordings and separate buffer-reuse tracking; retained serialized compatibility for older hosts (#32587, #32874, #33082).
- Improved external-device integration with context-aware memory identity, host-owned cache coverage, and stateless shared data transfers (#33110, #33208).
Reliability and Correctness
- Fixed out-of-bounds stores in subgroup-matrix
MatMulNBitsand added bounds checks for writes based onseqlens_k(#32582, #32623). - Fixed grouped-convolution underflow and vectorization across group boundaries (#32478, #33129).
- Fixed mixed-type
LayerNormalizationshaders and retained FP32 arithmetic inLayerNormalization(#32629, #32896). - Fixed
DequantizeLinearscalar/vector shader-cache collisions and 1Dblock_sizehandling (#33122, #32734). - Fixed initializer uploads on GPUs without Resizable BAR (#32816).
- Fixed
OrtMemoryInfoABI mismatches across the plugin boundary and bundled Dawn platform cleanup (#33571, #33526).
Build and Native Hosting
- Added a statically linked WebGPU plugin EP build configuration (#32395).
- Isolated the bundled Dawn implementation and added standalone Dawn API package support for native hosts, with external-Dawn plugin CI coverage (#32964, #33006, #33077).
- Updated Dawn and fixed WebGPU builds with LTO enabled and subgroup-configuration header dependencies (#32686, #32813, #32672).
Runtime Compatibility
The minimum ORT version is 1.24.4. Concurrent independent Sessions are enabled on ORT 1.28.3 and later in the 1.28 series, 1.30.1 and later in the 1.30 series, and 1.31 or later. Other supported versions, including 1.29.x, use serialized compatibility. In serialized mode, callers must serialize all WebGPU operations on the same device, including Session creation/destruction, allocation, I/O binding, transfers, and Run; the plugin does not enforce this serialization. Concurrent mode does not permit overlapping I/O binding and Run on the same Session (#33082).
Known compatibility caveat: ORT 1.26.0 was reported to reject the plugin during kernel creation because of its host-side plugin-version check. Forcing serialized execution does not resolve that restriction (#33082).
Contributors
Thanks to everyone who contributed to this release:
@aciddelgado, @adrastogi, @AkZGhost, @apsonawane, @edgchen1, @feich-ms, @gramalingam, @gyagp, @jchen10, @jiafatom, @justinchuby, @kunal-vaishnavi, @kylo5aby, @linzuojian, @mei1127, @mustjab, @prathikr, @qjia7, @sayanshaw24, @SharmaRithik, @skottmckay, @tianleiwu, @xadupre, @xhcao, @xiaofeihan1.
This release covers changes affecting ONNX Runtime WebGPU Plugin EP code and packaging since 0.4.0.
These release notes were prepared with AI assistance from commit history and pull request metadata.
ONNX Runtime v1.31.1
ONNX Runtime 1.31.1 is a patch release that fixes an OpenVINO-enabled Python build regression and corrects WASM Debug external-data tests. These notes cover changes since ONNX Runtime 1.31.0.
Highlights
- Fixed a provider-info namespace mismatch that caused compilation failures when building with both OpenVINO and Python enabled (#33327).
- Fixed WASM Debug test failures by skipping file-backed external-data cases unsupported by the WASM test module, while retaining compile-output checks, buffer-destination coverage, and native test coverage. This is a test-only change and does not change runtime behavior (#33226).
Contributors
Thanks to our 3 human contributors for this release!
@eserscor, @mustjab, @tianleiwu
Full Changelog: rel-1.31.0...rel-1.31.1
Release highlights were prepared with AI assistance.
ONNX Runtime v1.31.0
ONNX Runtime 1.31.0 stabilizes model-package and EPContext data APIs, improves CPU model loading and quantized MoE inference, expands device-based execution-provider selection, and strengthens model-loading and runtime reliability. These notes cover changes since ONNX Runtime 1.30.0. CUDA and WebGPU kernel updates are summarized here; detailed provider notes are covered by their separate plugin EP releases.
Highlights
- Promoted the model-package API and EPContext data callbacks to stable C and C++ APIs, with EPContext callback support added across language bindings (#33166, #32265).
- Reduced CPU session-initialization overhead by parallelizing eligible weight prepacking, and accelerated block-wise INT4/INT8 QMoE experts with MLAS QNBit kernels and grouped expert dispatch (#31691, #32644, #32668).
- Added x86 FP16 LayerNorm/RMSNorm acceleration, Arm KleidiAI SVE2.1 FP16 GEMM support, and additional RISC-V vector kernels (#32715, #32670, #32710).
- Enabled CoreML participation in device-based EP selection, including Apple Neural Engine selection through
PREFER_NPU(#31975). - Added workspace-memory accounting and verification for constrained-memory graph partitioning, alongside broader model, graph, and tensor validation (#31962, #32189).
Announcements & Compatibility
- CUDA Plugin EP v0.3.0 and WebGPU Plugin EP v0.5.0 are matched with ONNX Runtime 1.31.0. For contrib-op compatibility, core and the plugin must use the same contributed-operator schemas; building both from the same ONNX Runtime revision is recommended. Schema mismatches can cause incorrect execution or crashes and may not be detected during plugin registration.
GatherBlockQuantizedbehavior change: out-of-range indices now produce zero output slices across CPU, CUDA, and WebGPU for both integer and floating-point quantized formats. This deliberately differs from ONNXGathererror semantics (#32480).- Experimental API migration: model-package consumers should use
OrtApi::GetModelPackageApiand the stableOrtModelPackageApitable. EPContext callback registration now uses stable APIs; the corresponding experimental lookup entries were removed. The stable native callback contract carries callback/state pointers, with payload limits enforced by application callbacks, bindings, or EPs rather than a native read-options policy (#33166, #32265). - ACL EP is deprecated and will be removed in a future release. Source builds using
--use_aclnow receive a deprecation warning (#32564). - The ONNX dependency remains at 1.22.0. The initial ONNX 1.23 integration was reverted pending upstream fixes; this release does not ship that integration's new opset or FLOAT6 support (#33073).
- Supported native builds now default to 1DS telemetry. Windows TraceLogging remains selectable with
--use_windows_telemetryoronnxruntime_USE_WINDOWS_TELEMETRY=ON; telemetry can be disabled with--no_telemetry(#32384).
New Features
Core APIs & Runtime
- Stabilized the model-package C and C++ APIs for loading packages, selecting compatible model variants, and creating sessions (#33166).
- Stabilized named EPContext data read/write callbacks for application-managed storage. Added EP support negotiation so registered callbacks do not silently fall back to filesystem I/O (#32265).
- Extended the Compile API to write external initializers and EP context data to buffers, enabling in-memory compilation workflows for models larger than 2 GB (#31347).
- Added workspace-memory accounting, source reporting, and post-partition reservation verification. Set
session.strict_workspace_verification=1to fail initialization on workspace declaration overruns or nonzero orphaned reservations; strict verification is not supported for ORT-format model loads (#31962, #32189). - Added the opt-in source-build option
onnxruntime_DISABLE_DEVICE_DISCOVERY=ONto skip platform accelerator probing and report only the CPU device (#32592).
Shared Contrib Operators
These updates affect core operator schemas and cross-provider behavior, not only the separately released GPU plugins.
- Extended
EngramGateandNGramHashMappingfor Qwen4-Exp, and addedVarlenNGramHashMappingfor packed DeepSeek Engram batching with request-local history. Packed hashing also supports Qwen4-Exp head offsets, EOS resets, and segment boundaries (#32285, #32358, #32467). - Added the
activationattribute toGatedRMSNormfor SiLU, its Swish alias, and Sigmoid gating across CPU, CUDA, and WebGPU. SiLU remains the default (#32512). - Added weightless hyper-connection operators
BranchwiseRMSNorm,ScaledSiLU,HyperConnectionPreMix, andHyperConnectionPostMix, with CPU, CUDA, and WebGPU implementations (#32687). - Extended
GatherBlockQuantizedto FP8/FP4 data, full-axis blocks, and scale broadcasting, and improved CPU UINT8 dequantization. See the out-of-range-index behavior change above (#32480, #32621).
Plugin Execution Provider Infrastructure
- Added support for statically linked plugin EPs, with a static WebGPU plugin build configuration (#32395).
- Preserved caller streams in single-tensor plugin data transfers and handled null plugin allocator returns (#32666, #32683).
- Deduplicated AutoEP custom-op domains and corrected EP assignment reporting after NHWC layout transformation (#32711, #33184).
- Added reference guidance for compiled-model compatibility validation, including multi-device scenarios (#29168).
Execution Provider Updates
CUDA EP
CUDA updates include sparse attention and packed continuous batching, expanded paged speculative decoding, INT2 and mixed-width QMoE execution, opt-in MatMul auto-tuning, improved MXFP4 support on Blackwell consumer GPUs, and reliability fixes.
See the CUDA Plugin EP v0.3.0 release notes for detailed features, configuration options, limitations, and fixes. Most CUDA-specific PRs are covered there rather than repeated here. Use the matched core/plugin combination and observe the contrib-op schema compatibility requirements above.
WebGPU EP
- Expanded generative AI support with sparse attention, packed continuous-batching indexers, GatedDeltaNet, and broader quantized MoE support, including block-scaled FP8 and INT2/mixed-width weights (#32528, #32529, #32533, #32618, #32510, #32794, #32878).
- Improved native provider integration and performance, including explicit session-stream concurrency, subgroup-matrix kernels on AMD GPUs, and standalone Dawn API package support (#32587, #32667, #33006).
Detailed WebGPU kernel additions, optimizations, and correctness fixes will be covered in the separate WebGPU Plugin EP v0.5.0 release notes.
CoreML EP
- Exposed CoreML through
OrtEpFactory, device enumeration,SessionOptionsAppendExecutionProvider_V2, and EP selection policies. NPU selection requires iOS 16/macOS 13 or later; GPU selection requires iOS 15/macOS 12 or later. In builds advertising both CoreML and WebGPU for the same Apple GPU,PREFER_GPU/MAX_PERFORMANCEselect CoreML through the existing EP-name tie-breaker; WebGPU remains available through explicit device selection (#31975).
WebNN EP
- Added `...
ONNX Runtime CUDA Plugin EP 0.3.0
This release brings sparse attention and packed continuous-batching support, INT2 and mixed-width QMoE execution,
decode-kernel tuning, and reliability fixes to the ONNX Runtime CUDA Plugin EP. Highlights include new QSA/CSA
attention operators, expanded paged speculative decoding, opt-in MatMul auto-tuning, and improved MXFP4 support
on Blackwell consumer GPUs.
Compatibility
The minimum ONNX Runtime core version remains 1.24.4, but meeting that minimum alone does not guarantee
compatibility for contributed operators. Core and the plugin must use the same contributed-operator schemas;
building both from the same ONNX Runtime revision is recommended. Schema mismatches can cause incorrect execution
or crashes and may not be detected when registering the plugin (#32855).
The matched ONNX Runtime version is 1.31.0 for this release.
Highlights
Plugin Runtime and CUDA Graphs
- Added checked
GroupQueryAttentionworkspace recipes for KV-cache preparation and XQA, FlashAttention, memory-efficient, and unfused routes, plus conservative workspace estimation for resource-constrained graph partitioning. cuDNN workspace is reported as unavailable rather than incorrectly estimated as zero (#32446, #32453, #32454, #32617). - Bounded sliding-window GQA workspace by the resident or staged KV extent rather than the unbounded absolute sequence length, and consolidated runtime scratch sizing and sequence-length-independent route eligibility into shared helpers (#32602, #32945, #33001).
- Replaced eligible decode-path memset operations with small clearing kernels for
fpA_intBsplit-K semaphores, paged XQA semaphores, and compact convolution state updates, reducing CUDA Graph dependency overhead (#32884). - Replaced shared ones-buffer bias broadcasts in
GemmandDecoderAttentionwith direct broadcasts on the consuming CUDA stream, removed the constant-ones cache, and corrected zero-KGemmbias scaling (#32852).
Attention and KV Caching
- Added a cuDNN paged SDPA decode tier to
PagedAttentionfor eligible FP16/BF16 causal workloads with unquantized KV caches. It is selected ahead of FlashAttention when XQA is not the winner, automatically considered on SM90+, and can be disabled withORT_ENABLE_CUDNN_FLASH_ATTENTION=0. Follow-up fixes stabilize default-scale cache keys and backend selection during CUDA Graph execution (#32493, #32624). - Expanded non-causal
PagedAttentionbeyond FlashAttention, including portable paths for small cache pages and supported XQA speculative paths; added coverage for head size 128 with 16/32/64-token pages and ragged batches (#32619). - Added causal paged speculative XQA for FP16 query/output, INT8 KV caches, head size 128, GQA group size 6, and query widths 2-8. Unsupported widths and non-causal H128 workloads retain existing fallbacks (#32740).
- Documented the block-eviction layout contract for sliding-window GQA caches, including how to index resident tokens after multi-token speculative or chunked-prefill steps (#32419).
- Allowed GQA to borrow unquantized past K/V without allocating or copying present caches when new K/V is empty and both present-cache outputs are omitted, and corrected the unfused sliding-window bound to include the current token (#33175).
- Skipped redundant present-cache copies in ONNX
Attentionwhen the outputs alias the external KV-cache inputs (#31150).
Sparse Attention and Continuous Batching
- Added CUDA
SparsePagedAttentionandDynamicSparseAttentionfor externally selected attention indices over paged and contiguous KV caches. These support Qwen4-Exp QSA selected-main-cache attention and DeepSeek V4 CSA joint local/selected-auxiliary attention, with one softmax across participating entries and optional attention sinks (#32524, #32525). - Added
SparseAttentionIndexerfor QSA token selection and CSA compressed-block selection, including rotary preparation, scoring, deterministic TopK, and cache updates (#32526). - Added
PackedSparseAttentionIndexerfor packed, variable-length continuous batching, with per-request metadata, fixed-capacity state, and selection outputs compatible withSparsePagedAttention(#32618). - Optimized dense and packed QSA indexing, added packed QK inputs and broader mask/rotary-cache support, and added optional transactional QSA state-update capture to the packed indexer so speculative decoding can replay an accepted prefix (#32656).
- Parallelized
DynamicSparseAttentionacross selected-token tiles, bounded split workspace, replaced quadratic duplicate validation with a GPU bitmap, and added single-split fast paths for large-context inference (#32671).
New Operator and Model Support
- Extended
EngramGateandNGramHashMappingfor Qwen4-Exp, including normalized gated-value output, autoregressive history, per-head offsets, and EOS/segment resets (#32285). - Added
VarlenNGramHashMappingfor packed DeepSeek Engram batching, preserving request-local history and preventing n-gram mixing across request boundaries, then extended it with Qwen4-Exp hashing semantics (#32358, #32467). - Added an
activationattribute toGatedRMSNormfor SiLU, its Swish alias, and Sigmoid gating. SiLU remains the default (#32512). - Added the weightless hyper-connection fusion operators
BranchwiseRMSNorm,ScaledSiLU,HyperConnectionPreMix, andHyperConnectionPostMix(#32687). - Extended
GatherBlockQuantizedto FP8 and FP4 data, including full-axis blocks and scale broadcasting. Behavior change: out-of-range indices now produce zero output slices for both integer and floating-point quantized formats, rather than following ONNXGathererror semantics (#32480).
Quantized MatMul and MoE
- Added per-projection QMoE weight-width attributes and CUDA execution for uniform INT2 and mixed INT2/INT4 FC1/FC2 combinations
(2,2),(2,4), and(4,2), with packed decode paths and a scratch-bounded dequantization fallback for supported configurations (#32697, #32743, #32761). - Enabled eligible packed INT2/mixed-width QMoE prefill by default on SM80+ with FP16/BF16 activations, symmetric quantization, and interleaved fused SwiGLU. Support covers block sizes 32/64/128 and the three FC1/FC2 combinations above, subject to alignment, expert-count, and workspace restrictions (#32963, #33045, #33005, #33092).
- Added block-scaled FP8 QMoE support with routed-expert compaction, alongside the existing per-expert global-scale mode (#32887).
- Made NVFP4 QMoE decode consume schema-native packed weights and scales directly, avoiding decode-only repacking while retaining grouped-GEMM paths for larger workloads (#32325).
- Fixed MXFP4 W4A16 QMoE session creation and dispatch on SM120/SM121 and other supported SM80+ configurations, corrected the plugin adapter's
FLOAT8E8M0type mapping, and accelerated FP4 conversion in decode and grouped GEMM (#33067). - Added BF16 routing support to CUDA MoE and documented/tested packed token inputs for MoE and QMoE (#32659, #32522).
- Added opt-in FP16/BF16
MatMulauto-tuning viaep.cuda.enable_gemm_auto_tune=1orORT_CUDA_GEMM_AUTO_TUNE=1, choosing among cuBLAS, small-N GEMV, and eligible tinygemm2 kernels on SM90+. cuBLAS remains the default. Graph-enabled sessions validate candidates with CUDA Graph replay before caching the winner; warm-up inference is required before capture (#32876, #32901, #33085). - Improved
MatMulNBitssmall-M GEMV tiling, biased near-tie tactic selection toward GEMV for small decode batches, and enabl...
ONNX Runtime v1.28.3
This is a patch release on top of v1.28.2, containing memory-safety and model-validation fixes, a plugin execution fix, and packaging infrastructure updates.
Highlights
Bug Fixes
- Fixed an out-of-bounds read in
EmbedLayerNormalizationshape inference when beta is a scalar (#32610). - Fixed a heap buffer overflow when reusing a packed sub-byte buffer for a full-byte tensor (#32611).
- Fixed an out-of-bounds read when a Loop or Scan body input is also an initializer (#32609).
- Hardened shape inference for applicable contrib operators, including GroupQueryAttention, SparseAttention, and CausalConvWithState (#32607).
- Fixed out-of-bounds reads in ConvTransposeWithDynamicPads and
InferenceContextImpl::getInputData(#32679, #32681). - Rejected invalid graph structures: inputs to a zero-input operator schema and subgraphs on non-control-flow nodes (#32633, #32641).
- Preserved streams during single-tensor data transfers in the plugin EP (#32666).
Build and Test Reliability
- Unblocked packaging with Linux image prerequisites and an authenticated ARM64 pip index, mirrored Dawn checkouts, authenticated Android pip and Gradle feeds, and authenticated NuGet feeds for mobile workloads (#31724, #29743, #32400, #32005).
- Installed
uuid-devfor the Linux minimal CoreML build and removed the Hugging Face download dependency from a quantization test (#32491, #32443).
Contributors
Thanks to our 10 contributors for this release!
@wangw-1991, @bbernhar, @shiyi9801, @tianleiwu, @adrastogi, @eserscor, @jiafatom, @miaobin, @sanaa-hamel-microsoft, @xiaofeihan1
Full Changelog: v1.28.2...v1.28.3
These release notes were drafted with assistance from GitHub Copilot.
ONNX Runtime CUDA Plugin EP 0.2.0
This release brings expanded quantized inference and attention support, kernel performance improvements, and
reliability fixes to the ONNX Runtime CUDA Plugin EP. Highlights include 2-bit MatMulNBits, INT4 paged KV caches,
speculative decoding improvements, compact recurrent-state updates, and safer CUDA Graph execution across sessions.
Please refer to the CUDA Plugin EP Quick Start for usage and installation guidance. The matched ONNX Runtime version is 1.30.0.
Highlights
Plugin Runtime and CUDA Graphs
- Gave each plugin EP session its own device arena, preventing cross-session reuse of CUDA Graph buffers and honoring per-session arena options (#32807).
- Added workspace-memory accounting and reporting for resource-constrained graph partitioning, including packed attention workspace recipes and estimates, and optional-input-aware shape handling (#31962, #32283, #32312, #32321).
- Made
GatherNDcapture-safe and keptCudaAsyncBufferstaging buffers alive across graph replay (#32120, #32121). - Fixed shared-cache scratch lifetime in CUDA MHA, reset plugin EP stream chunks before release, and corrected the user-stream test lifetime (#31968, #31983, #32449).
- Fixed CUDA plugin device discovery on WSL, with improved hardware-identity matching and handling of devices missing from platform discovery (#32517).
Attention and KV Caching
- Added bidirectional
GroupQueryAttentionsupport and anis_causalattribute toPagedAttention(#31704, #32225). - Added INT4 paged KV caches with per-channel scales, including FP16 XQA decode and speculative-decode paths. Packed INT4 cache storage uses half the bytes of INT8 cache storage; availability is controlled by the
onnxruntime_USE_INT4_KV_CACHEbuild option, which defaults to enabled (#32515). - Enabled split-KV for paged FlashAttention decode and improved
PagedAttentiondispatch diagnostics (#32102, #32099). - Expanded paged XQA with group-size-6 support, head size 256, FP16-cache decode at head size 256, and native block tables for 128-token pages (#32108, #32229, #32263, #32127).
- Added paged XQA speculative verification for metadata-bounded query groups of 2-8 tokens at head size 256 and group size 6, including ragged batches and INT8, FP8, and native FP16/BF16 KV caches (#32340).
- Added the
session.gqa_value_layoutsession option for applications using a BNHSGroupQueryAttentionValue cache, with graph-inserted layout conversions; BNSH remains the default (#32139).
New Operator and Model Support
- Added compact
GatedDeltaNetwith transition capture for replaying accepted speculative prefixes, plus BF16 support (#32282, #32307). - Added
VarlenCausalConvWithStatefor packed, variable-length continuous batching, and compact state updates that avoid duplicating full convolution-state checkpoints for speculative prefixes (#32168, #32290). - Added CUDA kernels for the DeepSeek Engram contrib operators
EngramGateandNGramHashMapping(#32268). - Registered BF16
ReduceMeankernels (#32326).
Quantized MatMul and MoE
- Added 2-bit
MatMulNBitssupport, including dequantization and specialized GEMV/batched paths, and fused floating-point-activation/integer-weight (fpA_intB) GEMM/GEMV kernels for eligible FP16/BF16 workloads (#32693, #32699). - Made compact
fpA_intBkernels the default build configuration, extended compactMatMulNBitsto BF16, and refined GEMV eligibility checks. The complete kernel matrix remains available throughonnxruntime_USE_FPA_INTB_GEMM_FULL(#32324, #32721, #32338). - Enabled FP4 QMoE kernels by default in CUDA builds and added Windows support for Blackwell SM120 (#32096, #32163).
- Added an opt-in FP8 DeepGEMM decode path for eligible SM90 QMoE workloads, controlled by
ORT_QMOE_FP4_DEEPGEMMand disabled by default. This path is disabled on Windows because of upstream header requirements (#32122, #32485). - Bounded QMoE workspace with configurable row tiling and bounded FP8 weight-dequantization scratch by tiling over N (#32097, #32129).
- Vectorized NVFP4 weight dequantization for prefill, tuned NVFP4 GEMV tiling for Qwen multi-token prediction, and extended speculative-decode GEMVs to 64 rows (#32128, #32140, #32289).
- Tuned FP4/FP8 GEMV scheduling for 48-SM SM121 GPUs and improved FP8 GEMV residency for grids just past two blocks per SM (#32408, #32409, #32433).
- Reduced expensive
fpA_intBtactic-profiling work and added optional, size-gated M-row chunking for largeMatMulNBitsworkloads. Chunking is disabled by default and can be configured withep.cuda.matmul_nbits_m_chunk_sizeorORT_MATMULNBITS_M_CHUNK_SIZE(#32758, #32810).
General Kernel Performance
- Parallelized
ArgMax/ArgMinover wide last axes, optimized wide-last-axisTopK, and accelerated low-lane INT64CumSum(#32092, #32404, #32238). - Added a single-memcpy
Slicefast path for contiguous subregions (#28902). - Passed
Concatper-input metadata by value and avoided pinned buffers in CUDASplitandConcatfast paths (#32119, #32410).
Reliability and Correctness
- Fixed
ScatterElementsreduction dispatch by element type andAbssigned-zero handling (#29879, #31477). - Handled zero-element inputs/bias in
BiasGeluandFastGeluas no-ops and zero-sized outputs in CUDA random generator kernels (#31698, #31997). - Added bounds and shape validation for per-element
Splitsizes, 8-bitMatMulNBitsg_idx,GatherElementscounts, andScatterNDindex depth (#29461, #31643, #32030, #32034). - Validated CUDA NMS mask sizes, QDQ element counts, per-channel
ImageScalerbias, andCropinputs (#32014, #32029, #32002, #32157). - Bounded
RemovePaddingsequence-token counts and Whisper beam-search cross-QK layer/head indices (#31994, #31998). - Hardened integer arithmetic in
RotaryEmbedding,SparseAttention, CUDA reduction scans, and softmax offsets (#31995, #31996, #32137, #32330).