Repository navigation
Conversation
ONNX Runtime's MatMulAddFusion turns each Linear layer into a Gemm, whose CUDA kernel broadcasts the bias with an extra rank-1 GEMM on a non-tensor-core cuBLAS kernel (162 per pair). A separate broadcast Add is faster. Adds a disabled_optimizers argument to ONNXModel.
This was referenced Oct 28, 2026
Member
Author
|
Closing this one. The gain is real but small (−1.3 ms/pair, −2.4% at 2k, −1.8% at 8k), measured only on an A100 with CUDA 12 and ONNX Runtime 1.23/1.30. The risks outweigh it:
The finding is documented in #4908. The better fix is upstream in ONNX Runtime's CUDA #4915 and #4916 are re-stacked directly on #4908, and their numbers are updated. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #4908 (FlexGlue matcher) and #4907.
Summary
ONNX Runtime's
MatMulAddFusionturns each Linear layer of the FlexGlue matcher (MatMul+ biasAdd) into aGemm. Its CUDA kernel first broadcasts the bias into the output with an extra rank-1 GEMM (k = 1), which cuBLAS runs on a slow non-tensor-core kernel (ampere_fp16_sgemm_fp16_128x128_nn, 162 launches per pair, 2.2 ms of GPU time), and only then runs the real tensor-core GEMM withbeta = 1.This PR disables the fusion for the FlexGlue matcher on CUDA. The Linear layers then run as a tensor-core
MatMul(beta = 0) plus one fused broadcastAdd.ONNXModelgets adisabled_optimizersargument, which sets ONNX Runtime'soptimization.disable_specified_optimizerssession config entry.{"MatMulAddFusion"}when the matcher runs on CUDA. Nothing changes on CPU or CoreML.Can the bias be folded into the GEMM epilogue instead?
Not from the model side:
FusedMatMulhas no bias input, andGemmFastGelufuses only the tanh-approximated GELU.LayerNorm, the attention (after the q/kLayerNorm), and residual adds.The separate broadcast
Addthat remains costs 1.36 ms per pair.Results
Kernel profile: fp16 matcher, N = 2048, A100, ONNX Runtime 1.23 (CUPTI). GPU time: 30.42 → 28.91 ms per pair.
Addkernels (1.36 ms).beta = 0: 5.24 → 4.85 ms.COLMAP matching speed:
delivery_area+terrains+office(2132 pairs), with the keypoints of the evaluation, on an idle A100.Matches: identical up to floating-point rounding; 669 instead of 668 of the 2132 pairs have matches.
Tests:
feature/andcontrollers/tests pass.