Skip to content

Latest commit

 

History

History
29 lines (15 loc) · 2.39 KB

mlx-quantized-matmul.md

File metadata and controls

29 lines (15 loc) · 2.39 KB

Fixing quantized matrix multiplication in Apple MLX

Outcome: MLX PR #4202 was merged on August 12, 2026. The final change corrected scale-pointer stepping in two Metal loaders, disabled the affected non-transposed NAX dispatch, and expanded regression coverage.

The problem

A non-transposed quantized matrix multiplication could return corrupt values or NaNs when using a quantization group size of 32. A single reduction tile worked; adding another tile exposed the failure.

That distinction was the useful clue. The arithmetic inside a tile was not enough to explain the behavior. I compared the single-tile control (K=64) with multi-tile inputs (K=128 and K=256) and traced how the loader moved between tiles.

What I owned

I authored the PR and investigated the scale and bias addressing. In the affected branch, the loader advanced by n_groups * group_stride, although group_stride already represented the required step. Multiplying again moved the pointer too far. I corrected the corresponding path in both the affine and floating-point quantized loaders.

The decision

Correcting the loader and deciding whether to dispatch to that kernel were separate decisions. The merged patch also required transpose before entering the NAX path, keeping the affected non-transposed route disabled. Describing the outcome as simply enabling a repaired fast path would miss that part of the final review.

Verification

The final test changes cover multiple matrix shapes, quantization modes, dtypes, batching, partial tiles, and the group-size-32 boundary. The expanded non-transposed sweep is explicitly skipped under the CI environment variable because of its runtime cost.

My PR description records a local Metal run with 37 tests and 3,106 subtests passing, following 48 failing subtests before the fix. Those are the results recorded during the contribution, not a fresh rerun for this case study.

What this demonstrates

The small control case was essential: it isolated the transition between tiles. The final work combined a root-cause correction, broader regression coverage, and a conservative dispatch choice agreed through upstream review.

PR and discussion · Final merged commit · All case studies