Skip to content

Releases

ONNX Runtime WebGPU Plugin EP v0.6.0

Choose a tag to compare

@edgchen1 edgchen1 released this 16 Nov 22:44
2b68ec0

This release expands model and operator support, improves performance and numerical stability, and adds provider configuration and shader-cache controls.

Highlights

Model and operator support

  • Added Or, Xor, Round, Sign, IsNaN, IsInf, Softsign, LpPool, and GlobalLpPool kernels. (#33680, #33679)
  • Added QuantizeLinear with signed and unsigned 8-bit and 4-bit outputs, and 4-bit Reshape. Supported opsets and precision restrictions apply. (#33683, #33792)
  • Added FP16 support for GRU and LSTM, int32 support for Relu, and int8 support for Gather. (#33678, #33789, #33790)
  • Added signed int8 and int64 support for Abs, Neg, Relu, and Sign. Int64 kernels require the enableInt64 provider option. (#33958)
  • Added MatMulBlockQuantizedFp8Weight support for FP16 activations, E4M3FN FP8 weights, and FP32 block scales. (#33074)
  • Added automatic MatMulNBitsQLoRA fusion for supported models, using 4-bit block-32 base weights and 8-bit block-32 rank-32 or empty adapters. MatMulNBits also supports empty packed weights. (#33370, #33559)
  • Extended GatedDeltaNet to support per-key-dimension decay. (#33817)
  • Added ConvStateReplay and GatedDeltaNetStateReplay for reconstructing speculative-decoding state. These operators do not support graph capture; GatedDeltaNetStateReplay uses FP32 state and capsule backing. (#33493, #33213)
  • Added PackedSparseAttentionIndexerMerge to combine sparse-attention index sets. (#33224)

Performance and configuration

  • Added an Intel-specific dense flash-attention prefill path for unquantized KV caches. (#32702)
  • Added subgroup-matrix acceleration for eligible Intel FP16, group-1, channels-last convolutions. Other configurations use existing paths. (#32252)
  • Improved MatMul and MatMulNBits paths, including NVIDIA and AMD subgroup-matrix kernels and Qualcomm DP4A tuning. (#33639, #32796, #33560)
  • Reduced transfer overhead through batched input uploads, direct queue writes for eligible small CPU-to-GPU uploads, and batched compatible streamless GPU-to-GPU copies. (#33491, #33551, #33567)
  • Extended the opt-in enableMatmulFp32Accumulation option to FP16 MatMul and Gemm. It remains off by default; enabling it can bypass FP16-only subgroup-matrix paths. (#33057, #32630)
  • Added per-session enableFeatures and disableFeatures options, initially supporting subgroupMatrix. Disabling it selects existing fallback kernels. (#33481)
  • Added native dawnCacheInterface callbacks for caching Dawn shader and pipeline blobs, not model weights. Configure the interface before ORT creates the device; it cannot be attached to a host-supplied device. Later sessions may inherit the shared context's cache configuration. (#33936, #33999)

Correctness and packaging

  • Improved numerical stability for LayerNormalization, FP16 reductions, InstanceNormalization, and several elementwise functions. (#33365, #33682)
  • Fixed graph capture and replay of GPU buffer copies and explicit buffer clears. (#33362, #33509)
  • Synchronized Session command recording across execution, graph replay, and CPU I/O operations. Applications must still avoid overlapping I/O binding and Run on the same Session. (#33763)
  • Fixed shader-cache collisions across operator variants and a deferred pipeline callback lifetime issue after failed waits. (#33242, #33209)
  • Fixed scalar-bias Gemm, vectorized Pow broadcasting, three-channel Conv2dMM indexing, and empty pooling outputs. (#33664, #33348, #33432, #33668)
  • Bundled the preview D3D12 Agility SDK in Windows Python wheels, the NuGet package, and Foundry Local zip packages. (#32795)

Runtime Compatibility

The minimum ORT version is 1.24.4. Concurrent independent Sessions are enabled on ORT 1.28.3 and later in the 1.28 series, 1.30.1 and later in the 1.30 series, and 1.31 or later. Other supported versions, including 1.29.x, use serialized compatibility. In serialized mode, callers must serialize all WebGPU operations on the same device, including Session creation/destruction, allocation, I/O binding, transfers, and Run; the plugin does not enforce this serialization. Concurrent mode does not permit overlapping I/O binding and Run on the same Session (#33082).

Known compatibility caveat: ORT 1.26.0 was reported to reject the plugin during kernel creation because of its host-side plugin-version check. Forcing serialized execution does not resolve that restriction (#33082).

Activating the bundled Windows preview Agility SDK requires a compatible Windows version and loader update, Developer Mode, and a compatible GPU driver. See the Agility SDK guidance.

Contributors

Thank you to @aciddelgado, @adrastogi, @apsonawane, @baijumeswani, @daijh, @edgchen1, @GopalakrishnanN, @gramalingam, @gyagp, @jchen10, @Jiawei-Shao, @justinchuby, @keshavv27, @kunal-vaishnavi, @kylo5aby, @mei1127, @mohdmahmodi, @mustjab, @prathikr, @qjia7, @sayanshaw24, @SharmaRithik, @sushraja-msft, @tianleiwu, @xenova, @xhcao, @xiaofeihan1, and @ykhrustalev.

These notes focus on ONNX Runtime WebGPU Plugin EP changes since 0.5.0; contributor acknowledgments also cover commits touching shared WebGPU EP code.

These notes were prepared with AI assistance.

ONNX Runtime CUDA Plugin EP 0.4.0

Choose a tag to compare

@tianleiwu tianleiwu released this 14 Nov 06:34

This release brings CPU/GPU hybrid expert offloading, native GGUF/block-packed weight execution, Tensor-Core
INT2/INT3/INT4 and offline K-packed NVFP4 QMoE kernels, long-context and speculative-decoding acceleration, and
reliability fixes to the ONNX Runtime CUDA Plugin EP. Highlights include session.moe.offload.* expert offloading for
Mixture-of-Experts models that exceed GPU memory, an opt-in persistent MatMulNBits tactic cache, new fused LLM
operators, ONNX opset 28 kernels, and expanded Linux aarch64 CUDA 13 GPU coverage.

Compatibility

The minimum ONNX Runtime core version remains 1.24.4, but meeting that minimum alone does not guarantee
compatibility for contributed operators. Core and the plugin must use the same contributed-operator schemas;
building both from the same ONNX Runtime revision is recommended. Schema mismatches can cause incorrect execution
or crashes and may not be detected when registering the plugin (#32855).

The matched ONNX Runtime version is 1.32.0 for this release.

  • Features that require ONNX Runtime 1.32.0 core: CPU/GPU MoE offload (session.moe.offload.*), which relies on new core session-state support to keep expert initializers on the CPU; pipelined external-weight loading for plugin EPs; and the device memory capacity query (OrtApi::EpDevice_GetMemoryCapacity). Older runtimes ignore the plugin's new OrtEpFactory::GetDeviceMemoryCapacity callback, so it does not raise the 1.24.4 floor (#33451, #33839, #33447).
  • New operators and opset 28: Models that use the contributed operators added in this release, or ONNX opset 28 (ONNX 1.23.2), need a core that registers those schemas, such as the matched 1.32.0 release (#33134).
  • Not available in the plugin: The opt-in EP-owned runtime fusions added to the bundled CUDA EP in ONNX Runtime 1.32.0 (ep.cuda.runtime_fusion.unit_swish and ep.cuda.runtime_fusion.causal_conv_with_state) are not offered by CUDA plugin builds (#33757, #34021).
  • Capacity-aware partitioning: With ONNX Runtime 1.32.0 core and session.resource_cuda_partitioning_settings, the core declines to assign GroupQueryAttention nodes to the plugin because route-aware workspace estimates cannot yet pass through the plugin accounting path. A warning is logged and those nodes are left for other EPs. Sessions without capacity-aware partitioning are unaffected (#33016).

Highlights

CPU/GPU Hybrid MoE Offload

  • Added portable, session-scoped CPU/GPU expert offload for standard com.microsoft::QMoE nodes, configured entirely through session.moe.offload.* session options with no model changes. session.moe.offload.mode="auto" keeps every expert on the GPU when it fits and otherwise enables hybrid offload; "required" fails session creation when a node cannot be admitted. The planner budgets per-layer GPU expert slots (optionally guided by an expert-profile.bin popularity profile via session.moe.offload.popularity_file), runs cold experts on the CPU during decode through a doorbell path that works with CUDA Graph replay, streams cold experts to the GPU in waves during prefill, and promotes newly hot experts between steps with the default adaptive placement (#33451, #33452).
  • Supported NVFP4 (weights_prepacked=1), SM80-prepacked INT4 (block_size=32) and INT2, and raw INT3 expert banks, including mixed INT4/INT3/INT2 models, on SM80 and newer GPUs, with Tensor-Core INT3/INT4 offload decode and grouped prefill (#33575, #33622, #33629, #33731, #33787).
  • Extended hybrid offload to GGUF block-packed MoEPacked experts, with a three-stage pipelined prefill and multi-token cold-route execution (#33976, #33973, #34067, #34114).
  • Added AVX-512 and AVX2+FMA+F16C cold-expert CPU kernels that read the same INT4/INT3/INT2/FP4 weight layout as the GPU without repacking, and memory-mapped expert storage with an optional bounded host cache (session.moe.offload.host_expert_bytes) for hosts with limited RAM (#33450, #33624, #33785, #33935).
  • Improved offload on consumer GPUs and Windows with WDDM-aware VRAM budgeting that avoids spilling into shared GPU memory, Windows memory-mapping and pinned-memory accounting fixes, working-set retention for hot experts, ready-first scheduling of cached cold experts, and an opt-in zero-cold doorbell short-circuit (session.moe.offload.skip_zero_cold_doorbell=1) (#33916, #33935, #33699, #33624, #33629).
  • Added opt-in approximate routing for offloaded decode: tail-expert pruning below a routing-probability threshold (session.moe.offload.min_expert_weight; by default only CPU-cold tail experts are pruned) and substitution of cold tail experts with GPU-resident experts (session.moe.offload.tail_substitution_threshold and session.moe.offload.tail_substitution_min_rank). Both are disabled by default and change routing results, so validate model accuracy before enabling them (#33916, #33944).
  • Added session.moe.offload.gpu_device_mem_limit, a per-session cap on all device allocations made through the session's CUDA allocators (unlike gpu_mem_limit, which caps only the BFC arena), and session.moe.offload.budget_group for sharing one budget between target and draft sessions on the same GPU. The limit is not supported with external CUDA allocators or CUDA memory pools (#33451).
  • Added ep.cuda.qmoe_in_warmup and ep.cuda.qmoe_prefill_new_turn run-config entries so applications can exclude warm-up runs from routing statistics and reset prefill state at the start of a conversation turn (#33452, #33622).

Quantized MoE Kernels

  • Added support for offline K-packed NVFP4 QMoE weights (weights_prepacked=1), shared by the GEMV, grouped-GEMM, and dense fallback paths, with a register-decoded decode GEMV, single-CTA fused softmax/top-k routing for 256 and 512 experts, a single-pass finalize kernel, and programmatic dependent launch (PDL) overlap on SM90 and newer. Native E4M3 block-scale conversion is available as an opt-in with ORT_NVFP4_GEMV_NATIVE_SCALE=1 (#33094, #33354, #33366, #33545, #33323).
  • Enabled fused block-scaled FP8 QMoE execution by default on SM80 and newer (ORT_ENABLE_FP8_FUSED=0 disables it), and vectorized the block-FP8 decode GEMV and prefill weight dequantization (#33094, #33817).
  • Added fast INT3 and mixed-width (FC1, FC2) ∈ {2, 3, 4} × {2, 3, 4} QMoE kernels, offline SM80-prepacked INT2/INT4 weights (weights_prepacked=1), and BF16 Tensor-Core decode GEMVs for INT4, INT3, and INT2 experts. The Tensor-Core GEMVs are enabled by default; ep.cuda.disable_moe_gemv_tc=1 (INT4/INT2) and ep.cuda.disable_qmoe_int3_tc=1 (INT3) restore the SIMT kernels, and ep.cuda.moe_gemv_tc_min_rows and ep.cuda.qmoe_int3_tc_min_routes tune the dispatch thresholds (#33569, #33585, #33689, #33714, #33786).
  • Added opt-in grouped BF16 INT3 prefill (ep.cuda.qmoe_int3_grouped_prefill=1), with optional Tensor-Core compatibility rounding (ep.cuda.qmoe_int3_tc_compatibility=1) (#34117).
  • Extended fused routing to multi-token batches (S ≤ 8) with up to 512 experts, and added profile-free prefill tactic selection for deterministic MoE (ORT_FORCE_DETERMINISTIC_MOE=1), including INT4 with an Ampere tile fallback (#33596, #33367, #33595).
  • Added tanh-GELU gating (`activation_type="geglu"...
Read more

ONNX Runtime v1.32.0

Choose a tag to compare

@tianleiwu tianleiwu released this 13 Nov 09:36
fab30a4

ONNX Runtime 1.32.0 reintegrates ONNX 1.23.2 with opset 28 support, introduces CPU/GPU hybrid MoE offloading and shared-layout CPU QMoE kernels, expands device-memory, weightless-compilation, and Python model-package APIs, dramatically accelerates session initialization on large graphs, and strengthens runtime and optimizer reliability. These notes cover changes since ONNX Runtime 1.31.1. CUDA and WebGPU kernel updates are summarized here; detailed provider notes are covered by their separate plugin EP releases.

Highlights

  • Reintegrated ONNX 1.23.2 with opset 28 operator support (SpaceToDepth, Mod, signed BitShift, and BFloat16 extensions) and updated DLPack to v1.3 across all implemented ORT numeric and boolean tensor types (#33134, #32362, #33848).
  • Added a session-scoped CPU/GPU hybrid MoE offload planner and execution backend for quantized QMoE (NVFP4, INT4, INT3, and INT2) and block-packed MoEPacked models, backed by shared-layout AVX-512/AVX2 CPU QMoE kernels and memory-mapped expert storage (#33450, #33451, #33452, #33973).
  • Expanded core APIs with OrtApi::EpDevice_GetMemoryCapacity, explicit multi-mode weightless compilation (ModelCompilationOptions_SetWeightlessMode and ep.enable_weightless_mode), Python bindings for the Model Package APIs, and zero-copy CPU buffer borrowing in AddExternalInitializers (#33447, #32899, #33640, #33877).
  • Eliminated superlinear bottlenecks in allocation planning, constant sharing, graph I/O mapping, and topological sorting—reducing session initialization on 100k-node constant-heavy graphs from minutes to seconds—and added pipelined pinned-memory staging for external GPU weight loading (#33407, #33939, #34009, #33839).
  • Expanded shared generative AI and quantized contrib operators, including Kimi Delta Attention (GatedDeltaNet per-key-dimension decay), right-padded chunked prefill in GroupQueryAttention, native GGUF/block-packed operators (GatherPacked, MatMulPacked, MoEPacked), and sparse-attention index sharing (#33817, #33503, #33953, #33224).

Announcements & Compatibility

  • CUDA Plugin EP v0.4.0 and WebGPU Plugin EP v0.6.0 are matched with ONNX Runtime 1.32.0 (#33214). For contrib-op compatibility, core and the plugin must use the same contributed-operator schemas; building both from the same ONNX Runtime revision is recommended. Schema mismatches can cause incorrect execution or crashes and may not be detected during plugin registration.
  • ACL EP removed: Following its deprecation in 1.31.0, the Arm Compute Library (ACLExecutionProvider) execution provider, its build options, and its standalone C registration API have been removed. Python's get_all_providers() no longer lists ACLExecutionProvider; applications should remove it from explicit provider lists (#33851).
  • Legacy provider-bridge NvTensorRTRTXExecutionProvider removed: In-tree provider-bridge NvTensorRTRTXExecutionProvider sources have been removed in favor of the standalone NVIDIA TensorRT RTX plugin EP registered via OrtApi::RegisterExecutionProviderLibrary. NvTensorRtRtxProviderOptions is retained as a deprecated declaration for source compatibility, and legacy AppendExecutionProvider calls now report a deprecation error directing callers to the plugin EP (#32276).
  • ONNX 1.23.2 integration: ONNX has been upgraded to 1.23.2, raising supported opset ceilings to 28 and restoring ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT6E2M3 and ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT6E3M2 public element-type mappings and Cast conversions (#33134).
  • DLPack v1.3 boolean contract change: DLPack has been upgraded to v1.3 with native kDLBool metadata and bidirectional support for all implemented ORT numeric and boolean types, including all five float8 formats, packed float4, int4/uint4, and int2/uint2 (float6 remains unsupported in DLPack). The low-level is_bool_tensor override was removed, and uint8 DLPack metadata now always imports as uint8 (#33848).
  • Weightless compilation API migration: OrtCompileApi::ModelCompilationOptions_SetWeightlessEnabled and the ep.enable_weightless / ep.enable_weightless_ep_context_nodes session options are deprecated in favor of ModelCompilationOptions_SetWeightlessMode and ep.enable_weightless_mode ("0" = NONE, "1" = EXTERNAL_ONLY, "2" = ALL). EPs can now advertise multi-mode capability via OrtWeightlessSupport_ALL_OR_EXTERNAL_ONLY and weightless_supported_modes (#32899).
  • Training CI retirement: Training build and test lanes were retired from active GitHub Actions CI workflows while retaining inference build coverage (#33836).
  • This is the last release for CocoaPods, and we will switch to Swift Package Manager.
  • We will use combo onnxruntime-gpu python package to reduce storage size from next release. You can test it here

New Features

Core APIs & Runtime

  • Added OrtApi::EpDevice_GetMemoryCapacity (C++ EpDevice::GetMemoryCapacity()) and OrtEpFactory::GetDeviceMemoryCapacity to query device-global total memory and estimated currently available memory (DEFAULT or HOST_ACCESSIBLE), with built-in and plugin CUDA EP support (#33447).
  • Expanded the app–EP weightless contract so EPs advertise all supported weightless modes (weightless_supported_modes), applications request an explicit mode (ModelCompilationOptions_SetWeightlessMode / ep.enable_weightless_mode), and compiled EPContext models record weightless_mode and external_dependencies (source_model, external_weights, ep_context_binary) in model metadata (#32899).
  • Extended session.use_external_initializer_file_buffers_directly="1" to AddExternalInitializers for zero-copy borrowing of naturally aligned CPU buffers larger than 127 bytes on little-endian hosts, and documented the read-only guarantee for external initializers added via OrtModelEditorApi::AddInitializerToGraph with data_is_external=true (#33877, #33717).
  • Added lock-free multi-worker pinned-memory staging (session.external_data_loader.*) to accelerate external initializer loading into GPU memory across built-in and plugin CUDA EPs (#33839).
  • Added a session-scoped MoE/QMoE CPU/GPU offload planner and memory-budget ledger (session.moe.offload.*), generic session.kernel_dispatch_policy (auto, latency, memory, safe) configuration plumbing, Level-1 workspace/prepack memory accounting fallback for nodes absent from memory-profile CSVs, and internal capability-planning protocol types (#33451, #33042, #33311, #33841).
  • Enabled cross-thread reuse of idle device streams when graph capture is disabled and providers register device-selection callbacks, and enabled compile-session profiling (<profile_file>_compile) in onnxruntime_perf_test when -p is used with --compile_ep_context (#33053, #33694).
  • Made onnxruntime_DISABLE_EXCEPTIONS (--disable_exceptions) usable without onnxruntime_MINIMAL_BUILD and added the onnxruntime_DISABLE_STREAM=ON CMake option to disable device-stream support in non-minimal builds (#33120, #33482).

ONNX Opset 28 & Shared Contrib Operators

These updates affect core ONNX and contributed operator schemas and cross-provider behavior, not only the separately released GPU plugins.

  • Added ONNX opset 28 operator support: native SpaceToDepth (DCR and CRD) on CPU and CUDA with version-split registrations across DML, JS, WebGPU, and QNN; opset-28 floating floor-modulus Mod on CPU, CUDA, and DML; opset-28 BFloat16 support for Compress, OneHot, ReverseSequence, `...
Read more

ONNX Runtime v1.30.1

Choose a tag to compare

@adrastogi adrastogi released this 29 Oct 22:01
411c706

This is a patch release on top of v1.30.0, containing targeted model-validation and memory-safety fixes, improved Plugin EP data transfers, and build and packaging fixes.

Highlights

Bug Fixes

  • Hardened contrib-op shape inference with additional rank, attribute, initializer, and overflow validation for CausalConvWithState, GatedDeltaNet, GroupQueryAttention, and SparseAttention (#32607).
  • Prevented out-of-bounds reads in EmbedLayerNormalization shape inference with scalar beta, ConvTransposeWithDynamicPads, and shape-inference input-data lookup (#32610, #32679, #32681).
  • Rejected malformed graphs containing inputs to zero-input operator schemas or subgraphs on non-control-flow nodes (#32633, #32641).
  • Prevented an out-of-bounds read during session initialization when a Loop or Scan body input is also an initializer, rejecting the invalid graph instead of crashing (#32609).
  • Fixed a heap buffer overflow caused by reusing packed sub-byte tensor storage for a larger full-byte tensor (#32611).
  • Preserved the caller's execution stream in Plugin EP single-tensor data transfers, allowing asynchronous copies to receive the supplied stream while retaining synchronous-copy behavior (#32666).

Build and Packaging

  • Fixed Windows build-tools setup to use system-trusted certificates and improved proxy handling (#33142).
  • Selected Python 3.12 explicitly before package validation in the combined packaging stage (#33561).
  • Restricted FP8 XQA test cases to supported GPU compute capabilities (#32559).

Contributors

Thanks to our 8 contributors for this release!

@adrastogi, @bbernhar, @eserscor, @miaobin, @shiyi9801, @tianleiwu, @wangw-1991, @xiaofeihan1

Full Changelog: v1.30.0...v1.30.1

These release notes were drafted with AI assistance.

ONNX Runtime WebGPU Plugin EP v0.5.0

Choose a tag to compare

@edgchen1 edgchen1 released this 29 Oct 21:30
c44523c

ONNX Runtime WebGPU Plugin EP 0.5.0 expands attention, mixture-of-experts, and quantized operator support, improves GPU kernel performance, and strengthens runtime integration and correctness.

Highlights

Performance

  • Enabled subgroup-matrix kernels on AMD GPUs (#32667).
  • Added an FP16 GEMV specialization for single-row decode projections (#33070).
  • Improved data movement in Concat, Slice, Transpose, and pooling, and vectorized NHWC depthwise convolution across channels (#32497, #32498).
  • Enabled local-window flash attention for GroupQueryAttention (#32462).
  • Optimized recurrent and chunked GatedDeltaNet execution (#32645, #33143).
  • Optimized sparse attention indexers and SparsePagedAttention softmax, and cached sparse-indexer QSA visibility and scores (#32613, #32656, #33017).

Operator and Model Support

  • Added native GatedDeltaNet and VarlenCausalConvWithState kernels, including subsequent state-update-capacity support for GatedDeltaNet (#32510, #32477, #32767).
  • Added SparseAttentionIndexer, DynamicSparseAttention, and SparsePagedAttention, plus a packed sparse attention indexer for continuous batching (#32528, #32529, #32533, #32618).
  • Extended PagedAttention with full-context and local non-causal attention, and per-head Q/K normalization (#32763, #32792, #32808).
  • Added unquantized MoE and expanded QMoE activation, gating, routing, and packed-input support (#32447).
  • Added block-scaled FP8 QMoE, INT2 expert weights, and mixed-width integer QMoE execution (#32794, #32878).
  • Added Qwen4-Exp Engram support and VarlenNGramHashMapping for packed and variable-length batching (#32285, #32358, #32467).
  • Extended GatherBlockQuantized to FP8 and FP4 data. Out-of-range indices now produce zero output slices for both floating-point and existing integer formats (#32480).
  • Added GroupNorm, SkipGroupNorm, and 3D ConvTranspose support (#32463, #32688).
  • Added weightless hyper-connection primitives: BranchwiseRMSNorm, ScaledSiLU, HyperConnectionPreMix, and HyperConnectionPostMix (#32687).
  • Added SiLU/Swish/Sigmoid activation modes to GatedRMSNorm and QuickGelu fusion in the im2col-matmul convolution path (#32512, #32560).

Runtime Integration and Configuration

  • Added adapterIndex to select a physical Dawn adapter independently of the ORT WebGPU context identifier, deviceId (#32319).
  • Added the opt-in enableMatmulFp32Accumulation provider option for MatMulNBits and its fused variants; the default remains false (#29599).
  • Added concurrent independent-Session execution on eligible ORT hosts, with Session-owned command recordings and separate buffer-reuse tracking; retained serialized compatibility for older hosts (#32587, #32874, #33082).
  • Improved external-device integration with context-aware memory identity, host-owned cache coverage, and stateless shared data transfers (#33110, #33208).

Reliability and Correctness

  • Fixed out-of-bounds stores in subgroup-matrix MatMulNBits and added bounds checks for writes based on seqlens_k (#32582, #32623).
  • Fixed grouped-convolution underflow and vectorization across group boundaries (#32478, #33129).
  • Fixed mixed-type LayerNormalization shaders and retained FP32 arithmetic in LayerNormalization (#32629, #32896).
  • Fixed DequantizeLinear scalar/vector shader-cache collisions and 1D block_size handling (#33122, #32734).
  • Fixed initializer uploads on GPUs without Resizable BAR (#32816).
  • Fixed OrtMemoryInfo ABI mismatches across the plugin boundary and bundled Dawn platform cleanup (#33571, #33526).

Build and Native Hosting

  • Added a statically linked WebGPU plugin EP build configuration (#32395).
  • Isolated the bundled Dawn implementation and added standalone Dawn API package support for native hosts, with external-Dawn plugin CI coverage (#32964, #33006, #33077).
  • Updated Dawn and fixed WebGPU builds with LTO enabled and subgroup-configuration header dependencies (#32686, #32813, #32672).

Runtime Compatibility

The minimum ORT version is 1.24.4. Concurrent independent Sessions are enabled on ORT 1.28.3 and later in the 1.28 series, 1.30.1 and later in the 1.30 series, and 1.31 or later. Other supported versions, including 1.29.x, use serialized compatibility. In serialized mode, callers must serialize all WebGPU operations on the same device, including Session creation/destruction, allocation, I/O binding, transfers, and Run; the plugin does not enforce this serialization. Concurrent mode does not permit overlapping I/O binding and Run on the same Session (#33082).

Known compatibility caveat: ORT 1.26.0 was reported to reject the plugin during kernel creation because of its host-side plugin-version check. Forcing serialized execution does not resolve that restriction (#33082).

Contributors

Thanks to everyone who contributed to this release:

@aciddelgado, @adrastogi, @AkZGhost, @apsonawane, @edgchen1, @feich-ms, @gramalingam, @gyagp, @jchen10, @jiafatom, @justinchuby, @kunal-vaishnavi, @kylo5aby, @linzuojian, @mei1127, @mustjab, @prathikr, @qjia7, @sayanshaw24, @SharmaRithik, @skottmckay, @tianleiwu, @xadupre, @xhcao, @xiaofeihan1.


This release covers changes affecting ONNX Runtime WebGPU Plugin EP code and packaging since 0.4.0.

These release notes were prepared with AI assistance from commit history and pull request metadata.

ONNX Runtime v1.31.1

Choose a tag to compare

@eserscor eserscor released this 26 Oct 19:29
6af1a93

ONNX Runtime 1.31.1 is a patch release that fixes an OpenVINO-enabled Python build regression and corrects WASM Debug external-data tests. These notes cover changes since ONNX Runtime 1.31.0.

Highlights

  • Fixed a provider-info namespace mismatch that caused compilation failures when building with both OpenVINO and Python enabled (#33327).
  • Fixed WASM Debug test failures by skipping file-backed external-data cases unsupported by the WASM test module, while retaining compile-output checks, buffer-destination coverage, and native test coverage. This is a test-only change and does not change runtime behavior (#33226).

Contributors

Thanks to our 3 human contributors for this release!

@eserscor, @mustjab, @tianleiwu

Full Changelog: rel-1.31.0...rel-1.31.1

Release highlights were prepared with AI assistance.

ONNX Runtime v1.31.0

Choose a tag to compare

@tianleiwu tianleiwu released this 09 Oct 04:51
402979a

ONNX Runtime 1.31.0 stabilizes model-package and EPContext data APIs, improves CPU model loading and quantized MoE inference, expands device-based execution-provider selection, and strengthens model-loading and runtime reliability. These notes cover changes since ONNX Runtime 1.30.0. CUDA and WebGPU kernel updates are summarized here; detailed provider notes are covered by their separate plugin EP releases.

Highlights

  • Promoted the model-package API and EPContext data callbacks to stable C and C++ APIs, with EPContext callback support added across language bindings (#33166, #32265).
  • Reduced CPU session-initialization overhead by parallelizing eligible weight prepacking, and accelerated block-wise INT4/INT8 QMoE experts with MLAS QNBit kernels and grouped expert dispatch (#31691, #32644, #32668).
  • Added x86 FP16 LayerNorm/RMSNorm acceleration, Arm KleidiAI SVE2.1 FP16 GEMM support, and additional RISC-V vector kernels (#32715, #32670, #32710).
  • Enabled CoreML participation in device-based EP selection, including Apple Neural Engine selection through PREFER_NPU (#31975).
  • Added workspace-memory accounting and verification for constrained-memory graph partitioning, alongside broader model, graph, and tensor validation (#31962, #32189).

Announcements & Compatibility

  • CUDA Plugin EP v0.3.0 and WebGPU Plugin EP v0.5.0 are matched with ONNX Runtime 1.31.0. For contrib-op compatibility, core and the plugin must use the same contributed-operator schemas; building both from the same ONNX Runtime revision is recommended. Schema mismatches can cause incorrect execution or crashes and may not be detected during plugin registration.
  • GatherBlockQuantized behavior change: out-of-range indices now produce zero output slices across CPU, CUDA, and WebGPU for both integer and floating-point quantized formats. This deliberately differs from ONNX Gather error semantics (#32480).
  • Experimental API migration: model-package consumers should use OrtApi::GetModelPackageApi and the stable OrtModelPackageApi table. EPContext callback registration now uses stable APIs; the corresponding experimental lookup entries were removed. The stable native callback contract carries callback/state pointers, with payload limits enforced by application callbacks, bindings, or EPs rather than a native read-options policy (#33166, #32265).
  • ACL EP is deprecated and will be removed in a future release. Source builds using --use_acl now receive a deprecation warning (#32564).
  • The ONNX dependency remains at 1.22.0. The initial ONNX 1.23 integration was reverted pending upstream fixes; this release does not ship that integration's new opset or FLOAT6 support (#33073).
  • Supported native builds now default to 1DS telemetry. Windows TraceLogging remains selectable with --use_windows_telemetry or onnxruntime_USE_WINDOWS_TELEMETRY=ON; telemetry can be disabled with --no_telemetry (#32384).

New Features

Core APIs & Runtime

  • Stabilized the model-package C and C++ APIs for loading packages, selecting compatible model variants, and creating sessions (#33166).
  • Stabilized named EPContext data read/write callbacks for application-managed storage. Added EP support negotiation so registered callbacks do not silently fall back to filesystem I/O (#32265).
  • Extended the Compile API to write external initializers and EP context data to buffers, enabling in-memory compilation workflows for models larger than 2 GB (#31347).
  • Added workspace-memory accounting, source reporting, and post-partition reservation verification. Set session.strict_workspace_verification=1 to fail initialization on workspace declaration overruns or nonzero orphaned reservations; strict verification is not supported for ORT-format model loads (#31962, #32189).
  • Added the opt-in source-build option onnxruntime_DISABLE_DEVICE_DISCOVERY=ON to skip platform accelerator probing and report only the CPU device (#32592).

Shared Contrib Operators

These updates affect core operator schemas and cross-provider behavior, not only the separately released GPU plugins.

  • Extended EngramGate and NGramHashMapping for Qwen4-Exp, and added VarlenNGramHashMapping for packed DeepSeek Engram batching with request-local history. Packed hashing also supports Qwen4-Exp head offsets, EOS resets, and segment boundaries (#32285, #32358, #32467).
  • Added the activation attribute to GatedRMSNorm for SiLU, its Swish alias, and Sigmoid gating across CPU, CUDA, and WebGPU. SiLU remains the default (#32512).
  • Added weightless hyper-connection operators BranchwiseRMSNorm, ScaledSiLU, HyperConnectionPreMix, and HyperConnectionPostMix, with CPU, CUDA, and WebGPU implementations (#32687).
  • Extended GatherBlockQuantized to FP8/FP4 data, full-axis blocks, and scale broadcasting, and improved CPU UINT8 dequantization. See the out-of-range-index behavior change above (#32480, #32621).

Plugin Execution Provider Infrastructure

  • Added support for statically linked plugin EPs, with a static WebGPU plugin build configuration (#32395).
  • Preserved caller streams in single-tensor plugin data transfers and handled null plugin allocator returns (#32666, #32683).
  • Deduplicated AutoEP custom-op domains and corrected EP assignment reporting after NHWC layout transformation (#32711, #33184).
  • Added reference guidance for compiled-model compatibility validation, including multi-device scenarios (#29168).

Execution Provider Updates

CUDA EP

CUDA updates include sparse attention and packed continuous batching, expanded paged speculative decoding, INT2 and mixed-width QMoE execution, opt-in MatMul auto-tuning, improved MXFP4 support on Blackwell consumer GPUs, and reliability fixes.

See the CUDA Plugin EP v0.3.0 release notes for detailed features, configuration options, limitations, and fixes. Most CUDA-specific PRs are covered there rather than repeated here. Use the matched core/plugin combination and observe the contrib-op schema compatibility requirements above.

WebGPU EP

  • Expanded generative AI support with sparse attention, packed continuous-batching indexers, GatedDeltaNet, and broader quantized MoE support, including block-scaled FP8 and INT2/mixed-width weights (#32528, #32529, #32533, #32618, #32510, #32794, #32878).
  • Improved native provider integration and performance, including explicit session-stream concurrency, subgroup-matrix kernels on AMD GPUs, and standalone Dawn API package support (#32587, #32667, #33006).

Detailed WebGPU kernel additions, optimizations, and correctness fixes will be covered in the separate WebGPU Plugin EP v0.5.0 release notes.

CoreML EP

  • Exposed CoreML through OrtEpFactory, device enumeration, SessionOptionsAppendExecutionProvider_V2, and EP selection policies. NPU selection requires iOS 16/macOS 13 or later; GPU selection requires iOS 15/macOS 12 or later. In builds advertising both CoreML and WebGPU for the same Apple GPU, PREFER_GPU/MAX_PERFORMANCE select CoreML through the existing EP-name tie-breaker; WebGPU remains available through explicit device selection (#31975).

WebNN EP

  • Added `...
Read more

ONNX Runtime CUDA Plugin EP 0.3.0

Choose a tag to compare

@tianleiwu tianleiwu released this 09 Oct 00:49
402979a

This release brings sparse attention and packed continuous-batching support, INT2 and mixed-width QMoE execution,
decode-kernel tuning, and reliability fixes to the ONNX Runtime CUDA Plugin EP. Highlights include new QSA/CSA
attention operators, expanded paged speculative decoding, opt-in MatMul auto-tuning, and improved MXFP4 support
on Blackwell consumer GPUs.

Compatibility

The minimum ONNX Runtime core version remains 1.24.4, but meeting that minimum alone does not guarantee
compatibility for contributed operators. Core and the plugin must use the same contributed-operator schemas;
building both from the same ONNX Runtime revision is recommended. Schema mismatches can cause incorrect execution
or crashes and may not be detected when registering the plugin (#32855).

The matched ONNX Runtime version is 1.31.0 for this release.

Highlights

Plugin Runtime and CUDA Graphs

  • Added checked GroupQueryAttention workspace recipes for KV-cache preparation and XQA, FlashAttention, memory-efficient, and unfused routes, plus conservative workspace estimation for resource-constrained graph partitioning. cuDNN workspace is reported as unavailable rather than incorrectly estimated as zero (#32446, #32453, #32454, #32617).
  • Bounded sliding-window GQA workspace by the resident or staged KV extent rather than the unbounded absolute sequence length, and consolidated runtime scratch sizing and sequence-length-independent route eligibility into shared helpers (#32602, #32945, #33001).
  • Replaced eligible decode-path memset operations with small clearing kernels for fpA_intB split-K semaphores, paged XQA semaphores, and compact convolution state updates, reducing CUDA Graph dependency overhead (#32884).
  • Replaced shared ones-buffer bias broadcasts in Gemm and DecoderAttention with direct broadcasts on the consuming CUDA stream, removed the constant-ones cache, and corrected zero-K Gemm bias scaling (#32852).

Attention and KV Caching

  • Added a cuDNN paged SDPA decode tier to PagedAttention for eligible FP16/BF16 causal workloads with unquantized KV caches. It is selected ahead of FlashAttention when XQA is not the winner, automatically considered on SM90+, and can be disabled with ORT_ENABLE_CUDNN_FLASH_ATTENTION=0. Follow-up fixes stabilize default-scale cache keys and backend selection during CUDA Graph execution (#32493, #32624).
  • Expanded non-causal PagedAttention beyond FlashAttention, including portable paths for small cache pages and supported XQA speculative paths; added coverage for head size 128 with 16/32/64-token pages and ragged batches (#32619).
  • Added causal paged speculative XQA for FP16 query/output, INT8 KV caches, head size 128, GQA group size 6, and query widths 2-8. Unsupported widths and non-causal H128 workloads retain existing fallbacks (#32740).
  • Documented the block-eviction layout contract for sliding-window GQA caches, including how to index resident tokens after multi-token speculative or chunked-prefill steps (#32419).
  • Allowed GQA to borrow unquantized past K/V without allocating or copying present caches when new K/V is empty and both present-cache outputs are omitted, and corrected the unfused sliding-window bound to include the current token (#33175).
  • Skipped redundant present-cache copies in ONNX Attention when the outputs alias the external KV-cache inputs (#31150).

Sparse Attention and Continuous Batching

  • Added CUDA SparsePagedAttention and DynamicSparseAttention for externally selected attention indices over paged and contiguous KV caches. These support Qwen4-Exp QSA selected-main-cache attention and DeepSeek V4 CSA joint local/selected-auxiliary attention, with one softmax across participating entries and optional attention sinks (#32524, #32525).
  • Added SparseAttentionIndexer for QSA token selection and CSA compressed-block selection, including rotary preparation, scoring, deterministic TopK, and cache updates (#32526).
  • Added PackedSparseAttentionIndexer for packed, variable-length continuous batching, with per-request metadata, fixed-capacity state, and selection outputs compatible with SparsePagedAttention (#32618).
  • Optimized dense and packed QSA indexing, added packed QK inputs and broader mask/rotary-cache support, and added optional transactional QSA state-update capture to the packed indexer so speculative decoding can replay an accepted prefix (#32656).
  • Parallelized DynamicSparseAttention across selected-token tiles, bounded split workspace, replaced quadratic duplicate validation with a GPU bitmap, and added single-split fast paths for large-context inference (#32671).

New Operator and Model Support

  • Extended EngramGate and NGramHashMapping for Qwen4-Exp, including normalized gated-value output, autoregressive history, per-head offsets, and EOS/segment resets (#32285).
  • Added VarlenNGramHashMapping for packed DeepSeek Engram batching, preserving request-local history and preventing n-gram mixing across request boundaries, then extended it with Qwen4-Exp hashing semantics (#32358, #32467).
  • Added an activation attribute to GatedRMSNorm for SiLU, its Swish alias, and Sigmoid gating. SiLU remains the default (#32512).
  • Added the weightless hyper-connection fusion operators BranchwiseRMSNorm, ScaledSiLU, HyperConnectionPreMix, and HyperConnectionPostMix (#32687).
  • Extended GatherBlockQuantized to FP8 and FP4 data, including full-axis blocks and scale broadcasting. Behavior change: out-of-range indices now produce zero output slices for both integer and floating-point quantized formats, rather than following ONNX Gather error semantics (#32480).

Quantized MatMul and MoE

  • Added per-projection QMoE weight-width attributes and CUDA execution for uniform INT2 and mixed INT2/INT4 FC1/FC2 combinations (2,2), (2,4), and (4,2), with packed decode paths and a scratch-bounded dequantization fallback for supported configurations (#32697, #32743, #32761).
  • Enabled eligible packed INT2/mixed-width QMoE prefill by default on SM80+ with FP16/BF16 activations, symmetric quantization, and interleaved fused SwiGLU. Support covers block sizes 32/64/128 and the three FC1/FC2 combinations above, subject to alignment, expert-count, and workspace restrictions (#32963, #33045, #33005, #33092).
  • Added block-scaled FP8 QMoE support with routed-expert compaction, alongside the existing per-expert global-scale mode (#32887).
  • Made NVFP4 QMoE decode consume schema-native packed weights and scales directly, avoiding decode-only repacking while retaining grouped-GEMM paths for larger workloads (#32325).
  • Fixed MXFP4 W4A16 QMoE session creation and dispatch on SM120/SM121 and other supported SM80+ configurations, corrected the plugin adapter's FLOAT8E8M0 type mapping, and accelerated FP4 conversion in decode and grouped GEMM (#33067).
  • Added BF16 routing support to CUDA MoE and documented/tested packed token inputs for MoE and QMoE (#32659, #32522).
  • Added opt-in FP16/BF16 MatMul auto-tuning via ep.cuda.enable_gemm_auto_tune=1 or ORT_CUDA_GEMM_AUTO_TUNE=1, choosing among cuBLAS, small-N GEMV, and eligible tinygemm2 kernels on SM90+. cuBLAS remains the default. Graph-enabled sessions validate candidates with CUDA Graph replay before caching the winner; warm-up inference is required before capture (#32876, #32901, #33085).
  • Improved MatMulNBits small-M GEMV tiling, biased near-tie tactic selection toward GEMV for small decode batches, and enabl...
Read more

ONNX Runtime v1.28.3

Choose a tag to compare

@adrastogi adrastogi released this 02 Oct 20:48
0d68ff6

This is a patch release on top of v1.28.2, containing memory-safety and model-validation fixes, a plugin execution fix, and packaging infrastructure updates.

Highlights

Bug Fixes

  • Fixed an out-of-bounds read in EmbedLayerNormalization shape inference when beta is a scalar (#32610).
  • Fixed a heap buffer overflow when reusing a packed sub-byte buffer for a full-byte tensor (#32611).
  • Fixed an out-of-bounds read when a Loop or Scan body input is also an initializer (#32609).
  • Hardened shape inference for applicable contrib operators, including GroupQueryAttention, SparseAttention, and CausalConvWithState (#32607).
  • Fixed out-of-bounds reads in ConvTransposeWithDynamicPads and InferenceContextImpl::getInputData (#32679, #32681).
  • Rejected invalid graph structures: inputs to a zero-input operator schema and subgraphs on non-control-flow nodes (#32633, #32641).
  • Preserved streams during single-tensor data transfers in the plugin EP (#32666).

Build and Test Reliability

  • Unblocked packaging with Linux image prerequisites and an authenticated ARM64 pip index, mirrored Dawn checkouts, authenticated Android pip and Gradle feeds, and authenticated NuGet feeds for mobile workloads (#31724, #29743, #32400, #32005).
  • Installed uuid-dev for the Linux minimal CoreML build and removed the Hugging Face download dependency from a quantization test (#32491, #32443).

Contributors

Thanks to our 10 contributors for this release!

@wangw-1991, @bbernhar, @shiyi9801, @tianleiwu, @adrastogi, @eserscor, @jiafatom, @miaobin, @sanaa-hamel-microsoft, @xiaofeihan1

Full Changelog: v1.28.2...v1.28.3

These release notes were drafted with assistance from GitHub Copilot.

ONNX Runtime CUDA Plugin EP 0.2.0

Choose a tag to compare

@tianleiwu tianleiwu released this 08 Oct 22:11

This release brings expanded quantized inference and attention support, kernel performance improvements, and
reliability fixes to the ONNX Runtime CUDA Plugin EP. Highlights include 2-bit MatMulNBits, INT4 paged KV caches,
speculative decoding improvements, compact recurrent-state updates, and safer CUDA Graph execution across sessions.

Please refer to the CUDA Plugin EP Quick Start for usage and installation guidance. The matched ONNX Runtime version is 1.30.0.

Highlights

Plugin Runtime and CUDA Graphs

  • Gave each plugin EP session its own device arena, preventing cross-session reuse of CUDA Graph buffers and honoring per-session arena options (#32807).
  • Added workspace-memory accounting and reporting for resource-constrained graph partitioning, including packed attention workspace recipes and estimates, and optional-input-aware shape handling (#31962, #32283, #32312, #32321).
  • Made GatherND capture-safe and kept CudaAsyncBuffer staging buffers alive across graph replay (#32120, #32121).
  • Fixed shared-cache scratch lifetime in CUDA MHA, reset plugin EP stream chunks before release, and corrected the user-stream test lifetime (#31968, #31983, #32449).
  • Fixed CUDA plugin device discovery on WSL, with improved hardware-identity matching and handling of devices missing from platform discovery (#32517).

Attention and KV Caching

  • Added bidirectional GroupQueryAttention support and an is_causal attribute to PagedAttention (#31704, #32225).
  • Added INT4 paged KV caches with per-channel scales, including FP16 XQA decode and speculative-decode paths. Packed INT4 cache storage uses half the bytes of INT8 cache storage; availability is controlled by the onnxruntime_USE_INT4_KV_CACHE build option, which defaults to enabled (#32515).
  • Enabled split-KV for paged FlashAttention decode and improved PagedAttention dispatch diagnostics (#32102, #32099).
  • Expanded paged XQA with group-size-6 support, head size 256, FP16-cache decode at head size 256, and native block tables for 128-token pages (#32108, #32229, #32263, #32127).
  • Added paged XQA speculative verification for metadata-bounded query groups of 2-8 tokens at head size 256 and group size 6, including ragged batches and INT8, FP8, and native FP16/BF16 KV caches (#32340).
  • Added the session.gqa_value_layout session option for applications using a BNHS GroupQueryAttention Value cache, with graph-inserted layout conversions; BNSH remains the default (#32139).

New Operator and Model Support

  • Added compact GatedDeltaNet with transition capture for replaying accepted speculative prefixes, plus BF16 support (#32282, #32307).
  • Added VarlenCausalConvWithState for packed, variable-length continuous batching, and compact state updates that avoid duplicating full convolution-state checkpoints for speculative prefixes (#32168, #32290).
  • Added CUDA kernels for the DeepSeek Engram contrib operators EngramGate and NGramHashMapping (#32268).
  • Registered BF16 ReduceMean kernels (#32326).

Quantized MatMul and MoE

  • Added 2-bit MatMulNBits support, including dequantization and specialized GEMV/batched paths, and fused floating-point-activation/integer-weight (fpA_intB) GEMM/GEMV kernels for eligible FP16/BF16 workloads (#32693, #32699).
  • Made compact fpA_intB kernels the default build configuration, extended compact MatMulNBits to BF16, and refined GEMV eligibility checks. The complete kernel matrix remains available through onnxruntime_USE_FPA_INTB_GEMM_FULL (#32324, #32721, #32338).
  • Enabled FP4 QMoE kernels by default in CUDA builds and added Windows support for Blackwell SM120 (#32096, #32163).
  • Added an opt-in FP8 DeepGEMM decode path for eligible SM90 QMoE workloads, controlled by ORT_QMOE_FP4_DEEPGEMM and disabled by default. This path is disabled on Windows because of upstream header requirements (#32122, #32485).
  • Bounded QMoE workspace with configurable row tiling and bounded FP8 weight-dequantization scratch by tiling over N (#32097, #32129).
  • Vectorized NVFP4 weight dequantization for prefill, tuned NVFP4 GEMV tiling for Qwen multi-token prediction, and extended speculative-decode GEMVs to 64 rows (#32128, #32140, #32289).
  • Tuned FP4/FP8 GEMV scheduling for 48-SM SM121 GPUs and improved FP8 GEMV residency for grids just past two blocks per SM (#32408, #32409, #32433).
  • Reduced expensive fpA_intB tactic-profiling work and added optional, size-gated M-row chunking for large MatMulNBits workloads. Chunking is disabled by default and can be configured with ep.cuda.matmul_nbits_m_chunk_size or ORT_MATMULNBITS_M_CHUNK_SIZE (#32758, #32810).

General Kernel Performance

  • Parallelized ArgMax/ArgMin over wide last axes, optimized wide-last-axis TopK, and accelerated low-lane INT64 CumSum (#32092, #32404, #32238).
  • Added a single-memcpy Slice fast path for contiguous subregions (#28902).
  • Passed Concat per-input metadata by value and avoided pinned buffers in CUDA Split and Concat fast paths (#32119, #32410).

Reliability and Correctness

  • Fixed ScatterElements reduction dispatch by element type and Abs signed-zero handling (#29879, #31477).
  • Handled zero-element inputs/bias in BiasGelu and FastGelu as no-ops and zero-sized outputs in CUDA random generator kernels (#31698, #31997).
  • Added bounds and shape validation for per-element Split sizes, 8-bit MatMulNBits g_idx, GatherElements counts, and ScatterND index depth (#29461, #31643, #32030, #32034).
  • Validated CUDA NMS mask sizes, QDQ element counts, per-channel ImageScaler bias, and Crop inputs (#32014, #32029, #32002, #32157).
  • Bounded RemovePadding sequence-token counts and Whisper beam-search cross-QK layer/head indices (#31994, #31998).
  • Hardened integer arithmetic in RotaryEmbedding, SparseAttention, CUDA reduction scans, and softmax offsets (#31995, #31996, #32137, #32330).

...

Read more