First MLX / Apple Silicon native implementation of QJL (arXiv:2406.03482) and TurboQuant (arXiv:2504.19874).
TurboQuant is Google Research's ultra-low-precision KV cache quantization algorithm (PolarQuant + QJL). The official implementation targets JAX + CUDA. This project ports it to Apple Silicon via MLX, with a PyTorch reference implementation for parity verification.
Two paper-defined modes are supported:
- TURBOQUANTmse (
mode="mse"): all bits to PolarQuant. Biased but sufficient for LLM inference. Recommended default. Confirmed by community implementations (TheTom/turboquant_plus). - TURBOQUANTprod (
mode="prod"): PolarQuant (b−1 bits) + independent QJL residual (1 bit). Unbiased inner-product estimator per Theorem 2 of the TurboQuant paper. For research / theoretical validation.
- PyTorch reference implementations for QJL, PolarQuant, and TurboQuant (mse + prod)
- MLX backend for TurboQuant on Apple Silicon, parity-checked against PyTorch (max abs score diff
1.79e-7across all 4 mode×codebook configs) - Lloyd-Max optimized PolarQuant codebooks (PolarQuant paper, Section 3)
- At-scale validation harness on 1000 random unit vectors (
tests/test_at_scale.py) - Phase-by-phase handoff documents under
docs/
| Phase | Description | Status |
|---|---|---|
| Phase 1 | QJL PyTorch reference + validation harness | Complete |
| Phase 2 | PolarQuant PyTorch reference + Lloyd-Max codebook | Complete |
| Phase 3 | Full TurboQuant (mse + prod) + MLX backend | Complete |
| Phase 4 | Downstream integration | Superseded by gpt2-from-scratch Phase 8/11 — TurboQuant integrated into GPT-2 attention, Δ PPL < 1.05, 9.14× KV cache memory compression on real LLM inference |
Reproduced via tests/test_at_scale.py (1000 random unit vectors, dim=1024, 50 queries) and recorded in results/at_scale_validation.json:
| Method | bits/dim | Recall@5 (toy, 15 docs) | Recall@5 (1000 docs) | Pearson r |
|---|---|---|---|---|
| QJL 1-bit | 1.0 | 88% | 33.6% | 0.796 |
| PolarQuant 2-bit | 2.03 | 88% | 46.4% | 0.890 |
| TURBOQUANTmse 3-bit | 3.03 | 92% | 71.6% | 0.971 |
| TURBOQUANTprod 3-bit | 3.03 | 88% | 48.8% | 0.903 |
Additional checks:
- MLX vs PyTorch max abs score diff:
0.000000(all 4 mode × codebook configs) - TURBOQUANTprod unbiasedness: bias =
0.001(< 0.05 threshold) - PolarQuant codebooks use Lloyd-Max optimization (arXiv:2502.02617, Section 3)
Requires Python 3.10+ and Apple Silicon for the MLX backend (CPU/MPS otherwise).
pip install -r requirements.txtrequirements.txt declares torch, sentence-transformers, numpy, and mlx.
Run the full validation suite:
python tests/test_qjl.py # Phase 1 — QJL PyTorch baseline
python tests/test_polar.py # Phase 2 — PolarQuant + Lloyd-Max codebook
python tests/test_turbo.py # Phase 3 — TurboQuant mse + prod (PyTorch)
python tests/test_turbo_mlx.py # Phase 3 — MLX backend parity check
python tests/test_at_scale.py # Phase 3 — 1000-document validationEach test writes a JSON report under results/.
qjl-mlx/
├── src/
│ ├── qjl_torch.py # QJL reference (Phase 1)
│ ├── polar_quant.py # PolarQuant + Lloyd-Max codebooks (Phase 2)
│ ├── turbo_quant.py # TurboQuant mse + prod (PyTorch, Phase 3)
│ └── turbo_quant_mlx.py # TurboQuant MLX backend (Phase 3)
├── tests/
│ ├── test_qjl.py
│ ├── test_polar.py
│ ├── test_turbo.py
│ ├── test_turbo_mlx.py
│ └── test_at_scale.py
├── docs/
│ ├── phase1-handoff.md
│ ├── phase2-handoff.md
│ ├── phase3-handoff.md
│ └── qjl-mlx-fix-plan.md
├── results/ # JSON validation outputs
├── AGENTS.md # Repository working principles
├── requirements.txt
└── README.md
The library has no environment variables or config files — everything is passed at compressor construction. Key parameters (see src/turbo_quant.py and src/turbo_quant_mlx.py):
| Parameter | Where | Meaning |
|---|---|---|
dim |
TurboQuantCompressor, TurboQuantMLXCompressor |
Embedding dimension (e.g. 1024) |
total_bits |
same | Total per-dim bit budget; mse gives all bits to PolarQuant, prod reserves 1 bit for the QJL residual |
mode |
same | "mse" (recommended default) or "prod" (unbiased, for research) |
polar_codebook |
same | "analytic" (Lloyd-Max optimized) or "uniform" |
seed |
same | RNG seed; prod mode uses seed+1 for the independent QJL rotation |
device |
PyTorch only | "cpu" / "mps" / "cuda" |
The repository has no dedicated linter or CI configuration — validation is the test suite. Each phase commit is expected to pass its own tests/test_*.py script before being marked complete (see AGENTS.md). When extending the work, write a new test under tests/, update or add a handoff document under docs/, and record metrics under results/.
This is a research / paper-port repository whose roadmap is complete (Phase 4 superseded by gpt2-from-scratch). PRs are welcome but not actively solicited; if you open one, follow the rules in AGENTS.md:
- New behavior must come with a corresponding
tests/test_*.pyscript. - Update or add a handoff document under
docs/for the affected phase. - Keep
results/*.jsonregenerated alongside any algorithmic change. - Do not declare a phase "complete" unless code, tests, and handoff doc are all updated in the same change.
- Authoritative sources:
docs/phase*-handoff.mdfor completed phases,docs/qjl-mlx-fix-plan.mdfor the v2 fix log, andAGENTS.mdfor repository working principles. - Real downstream consumer: gpt2-from-scratch Phase 8/11 — verifies TurboQuant on GPT-2 KV cache. Update its links if API changes here.
- No GitHub Actions / CI: changes are validated locally by re-running the test scripts above. Keep
results/*.jsonin sync with code changes. - Per
AGENTS.md: a phase is not "complete" until code, validation, and the matching handoff document underdocs/are all updated together.
- QJL: arXiv:2406.03482
- PolarQuant: arXiv:2502.02617
- TurboQuant: arXiv:2504.19874
- MLX: ml-explore.github.io/mlx
- Downstream consumer: gpt2-from-scratch
MIT
TurboQuant 是 Google Research 提出的超低精度 KV cache 量化算法(PolarQuant + QJL)。官方实现基于 JAX + CUDA,本项目通过 MLX 将其移植到 Apple Silicon,并提供 PyTorch 参考实现用于一致性比对。
支持论文定义的两种模式:
- TURBOQUANTmse(
mode="mse"):所有 bit 给 PolarQuant。有偏,但在 LLM 推理场景下足够。推荐默认值,社区多个独立实现已验证(TheTom/turboquant_plus)。 - TURBOQUANTprod(
mode="prod"):PolarQuant(b−1 bit)+ 独立 QJL 残差(1 bit)。按 TurboQuant 论文 Theorem 2 是无偏内积估计器,仅用于研究 / 理论验证。
- QJL、PolarQuant、TurboQuant(mse + prod)的 PyTorch 参考实现
- TurboQuant 在 Apple Silicon 上的 MLX 后端,与 PyTorch 在 4 种 mode × codebook 配置下的最大得分绝对差为
1.79e-7 - PolarQuant 使用 Lloyd-Max 最优码本(PolarQuant 论文 Section 3)
- 1000 个随机单位向量上的中等规模验证(
tests/test_at_scale.py) docs/下按阶段归档的 handoff 文档
| 阶段 | 内容 | 状态 |
|---|---|---|
| Phase 1 | QJL PyTorch 参考实现 + 验证脚本 | 已完成 |
| Phase 2 | PolarQuant PyTorch 参考 + Lloyd-Max 码本 | 已完成 |
| Phase 3 | TurboQuant 完整两模式 + MLX 后端 | 已完成 |
| Phase 4 | 下游集成 | 已被 gpt2-from-scratch Phase 8/11 替代——TurboQuant 已接入 GPT-2 attention,真实 LLM 推理 Δ PPL < 1.05,KV cache 内存压缩 9.14× |
由 tests/test_at_scale.py 复现(1000 个随机单位向量,dim=1024,50 个查询),记录在 results/at_scale_validation.json:
| 方法 | bits/dim | Recall@5(toy,15 文档) | Recall@5(1000 文档) | Pearson r |
|---|---|---|---|---|
| QJL 1-bit | 1.0 | 88% | 33.6% | 0.796 |
| PolarQuant 2-bit | 2.03 | 88% | 46.4% | 0.890 |
| TURBOQUANTmse 3-bit | 3.03 | 92% | 71.6% | 0.971 |
| TURBOQUANTprod 3-bit | 3.03 | 88% | 48.8% | 0.903 |
补充验证:
- MLX vs PyTorch 最大得分绝对差:
0.000000(4 个配置全部) - TURBOQUANTprod 无偏性:bias =
0.001(< 0.05 阈值) - PolarQuant 使用 Lloyd-Max 优化码本(arXiv:2502.02617 Section 3)
需要 Python 3.10+;MLX 后端需要 Apple Silicon,否则只能跑 PyTorch 路径(CPU/MPS)。
pip install -r requirements.txtrequirements.txt 包含 torch、sentence-transformers、numpy、mlx。
运行完整验证套件:
python tests/test_qjl.py # Phase 1 —— QJL PyTorch 基线
python tests/test_polar.py # Phase 2 —— PolarQuant + Lloyd-Max 码本
python tests/test_turbo.py # Phase 3 —— TurboQuant mse + prod(PyTorch)
python tests/test_turbo_mlx.py # Phase 3 —— MLX 后端 parity 验证
python tests/test_at_scale.py # Phase 3 —— 1000 文档验证每个测试都会在 results/ 下写入对应的 JSON 报告。
qjl-mlx/
├── src/
│ ├── qjl_torch.py # QJL 参考实现(Phase 1)
│ ├── polar_quant.py # PolarQuant + Lloyd-Max 码本(Phase 2)
│ ├── turbo_quant.py # TurboQuant mse + prod(PyTorch,Phase 3)
│ └── turbo_quant_mlx.py # TurboQuant MLX 后端(Phase 3)
├── tests/
│ ├── test_qjl.py
│ ├── test_polar.py
│ ├── test_turbo.py
│ ├── test_turbo_mlx.py
│ └── test_at_scale.py
├── docs/
│ ├── phase1-handoff.md
│ ├── phase2-handoff.md
│ ├── phase3-handoff.md
│ └── qjl-mlx-fix-plan.md
├── results/ # 各阶段 JSON 验证输出
├── AGENTS.md # 仓库工作准则
├── requirements.txt
└── README.md
本库没有环境变量和配置文件——所有参数都在构造 compressor 时传入。关键参数(详见 src/turbo_quant.py 和 src/turbo_quant_mlx.py):
| 参数 | 位置 | 含义 |
|---|---|---|
dim |
TurboQuantCompressor、TurboQuantMLXCompressor |
向量维度(如 1024) |
total_bits |
同上 | 每维总 bit 预算;mse 全给 PolarQuant,prod 留 1 bit 给 QJL 残差 |
mode |
同上 | "mse"(推荐默认值)或 "prod"(无偏,研究用) |
polar_codebook |
同上 | "analytic"(Lloyd-Max 最优)或 "uniform" |
seed |
同上 | 随机种子;prod 模式用 seed+1 生成独立的 QJL 旋转矩阵 |
device |
仅 PyTorch | "cpu" / "mps" / "cuda" |
仓库没有专门的 linter / CI 配置——验证 = 跑测试套件。按 AGENTS.md 的约定:每个阶段的提交在标记为完成前,必须通过对应的 tests/test_*.py。继续扩展时:在 tests/ 下加测试、在 docs/ 下更新或新增 handoff 文档、在 results/ 下保留指标 JSON。
本仓库是论文移植 / 研究性项目,路线图已完成(Phase 4 被 gpt2-from-scratch 替代)。欢迎 PR,但不主动征集;如果要提 PR,请遵守 AGENTS.md 的约定:
- 新行为必须配套
tests/test_*.py验证脚本。 - 在
docs/下更新或新增对应阶段的 handoff 文档。 - 算法变动时同步重跑并更新
results/*.json。 - 一个阶段只有在代码、测试、handoff 文档同时更新后才能称为"完成"。
- 权威信息源:已完成阶段看
docs/phase*-handoff.md,v2 修复历程看docs/qjl-mlx-fix-plan.md,仓库工作准则看AGENTS.md。 - 真实下游消费者:gpt2-from-scratch 的 Phase 8/11——验证了 TurboQuant 在 GPT-2 KV cache 上的表现。如果本仓库 API 变了,记得同步那边的链接。
- 无 GitHub Actions / CI:所有变更靠本地重跑上面的测试脚本验证;改代码时记得同步
results/*.json。 - 遵循
AGENTS.md:一个阶段只有在代码、验证、docs/下对应的 handoff 文档同时更新后才能称为"完成"。
- QJL:arXiv:2406.03482
- PolarQuant:arXiv:2502.02617
- TurboQuant:arXiv:2504.19874
- MLX:ml-explore.github.io/mlx
- 下游消费者:gpt2-from-scratch
MIT