Skip to content
lingengyuanPublic

About

QJL and TurboQuant KV-cache quantization with MLX and PyTorch reference checks.

Resources

Stars

3 stars

Watchers

0 watching

Forks

Latest commit

 

History

7 Commits

Folders and files

Repository files navigation

qjl-mlx

First MLX / Apple Silicon native implementation of QJL (arXiv:2406.03482) and TurboQuant (arXiv:2504.19874).


English

Project Description

TurboQuant is Google Research's ultra-low-precision KV cache quantization algorithm (PolarQuant + QJL). The official implementation targets JAX + CUDA. This project ports it to Apple Silicon via MLX, with a PyTorch reference implementation for parity verification.

Two paper-defined modes are supported:

  • TURBOQUANTmse (mode="mse"): all bits to PolarQuant. Biased but sufficient for LLM inference. Recommended default. Confirmed by community implementations (TheTom/turboquant_plus).
  • TURBOQUANTprod (mode="prod"): PolarQuant (b−1 bits) + independent QJL residual (1 bit). Unbiased inner-product estimator per Theorem 2 of the TurboQuant paper. For research / theoretical validation.

Features

  • PyTorch reference implementations for QJL, PolarQuant, and TurboQuant (mse + prod)
  • MLX backend for TurboQuant on Apple Silicon, parity-checked against PyTorch (max abs score diff 1.79e-7 across all 4 mode×codebook configs)
  • Lloyd-Max optimized PolarQuant codebooks (PolarQuant paper, Section 3)
  • At-scale validation harness on 1000 random unit vectors (tests/test_at_scale.py)
  • Phase-by-phase handoff documents under docs/

Roadmap

Phase Description Status
Phase 1 QJL PyTorch reference + validation harness Complete
Phase 2 PolarQuant PyTorch reference + Lloyd-Max codebook Complete
Phase 3 Full TurboQuant (mse + prod) + MLX backend Complete
Phase 4 Downstream integration Superseded by gpt2-from-scratch Phase 8/11 — TurboQuant integrated into GPT-2 attention, Δ PPL < 1.05, 9.14× KV cache memory compression on real LLM inference

Key Metrics

Reproduced via tests/test_at_scale.py (1000 random unit vectors, dim=1024, 50 queries) and recorded in results/at_scale_validation.json:

Method bits/dim Recall@5 (toy, 15 docs) Recall@5 (1000 docs) Pearson r
QJL 1-bit 1.0 88% 33.6% 0.796
PolarQuant 2-bit 2.03 88% 46.4% 0.890
TURBOQUANTmse 3-bit 3.03 92% 71.6% 0.971
TURBOQUANTprod 3-bit 3.03 88% 48.8% 0.903

Additional checks:

  • MLX vs PyTorch max abs score diff: 0.000000 (all 4 mode × codebook configs)
  • TURBOQUANTprod unbiasedness: bias = 0.001 (< 0.05 threshold)
  • PolarQuant codebooks use Lloyd-Max optimization (arXiv:2502.02617, Section 3)

Installation

Requires Python 3.10+ and Apple Silicon for the MLX backend (CPU/MPS otherwise).

pip install -r requirements.txt

requirements.txt declares torch, sentence-transformers, numpy, and mlx.

Quick Start

Run the full validation suite:

python tests/test_qjl.py        # Phase 1 — QJL PyTorch baseline
python tests/test_polar.py      # Phase 2 — PolarQuant + Lloyd-Max codebook
python tests/test_turbo.py      # Phase 3 — TurboQuant mse + prod (PyTorch)
python tests/test_turbo_mlx.py  # Phase 3 — MLX backend parity check
python tests/test_at_scale.py   # Phase 3 — 1000-document validation

Each test writes a JSON report under results/.

Project Structure

qjl-mlx/
├── src/
│   ├── qjl_torch.py          # QJL reference (Phase 1)
│   ├── polar_quant.py        # PolarQuant + Lloyd-Max codebooks (Phase 2)
│   ├── turbo_quant.py        # TurboQuant mse + prod (PyTorch, Phase 3)
│   └── turbo_quant_mlx.py    # TurboQuant MLX backend (Phase 3)
├── tests/
│   ├── test_qjl.py
│   ├── test_polar.py
│   ├── test_turbo.py
│   ├── test_turbo_mlx.py
│   └── test_at_scale.py
├── docs/
│   ├── phase1-handoff.md
│   ├── phase2-handoff.md
│   ├── phase3-handoff.md
│   └── qjl-mlx-fix-plan.md
├── results/                   # JSON validation outputs
├── AGENTS.md                  # Repository working principles
├── requirements.txt
└── README.md

Configuration

The library has no environment variables or config files — everything is passed at compressor construction. Key parameters (see src/turbo_quant.py and src/turbo_quant_mlx.py):

Parameter Where Meaning
dim TurboQuantCompressor, TurboQuantMLXCompressor Embedding dimension (e.g. 1024)
total_bits same Total per-dim bit budget; mse gives all bits to PolarQuant, prod reserves 1 bit for the QJL residual
mode same "mse" (recommended default) or "prod" (unbiased, for research)
polar_codebook same "analytic" (Lloyd-Max optimized) or "uniform"
seed same RNG seed; prod mode uses seed+1 for the independent QJL rotation
device PyTorch only "cpu" / "mps" / "cuda"

Development and Testing

The repository has no dedicated linter or CI configuration — validation is the test suite. Each phase commit is expected to pass its own tests/test_*.py script before being marked complete (see AGENTS.md). When extending the work, write a new test under tests/, update or add a handoff document under docs/, and record metrics under results/.

Contributing

This is a research / paper-port repository whose roadmap is complete (Phase 4 superseded by gpt2-from-scratch). PRs are welcome but not actively solicited; if you open one, follow the rules in AGENTS.md:

  1. New behavior must come with a corresponding tests/test_*.py script.
  2. Update or add a handoff document under docs/ for the affected phase.
  3. Keep results/*.json regenerated alongside any algorithmic change.
  4. Do not declare a phase "complete" unless code, tests, and handoff doc are all updated in the same change.

Maintenance Guide

  • Authoritative sources: docs/phase*-handoff.md for completed phases, docs/qjl-mlx-fix-plan.md for the v2 fix log, and AGENTS.md for repository working principles.
  • Real downstream consumer: gpt2-from-scratch Phase 8/11 — verifies TurboQuant on GPT-2 KV cache. Update its links if API changes here.
  • No GitHub Actions / CI: changes are validated locally by re-running the test scripts above. Keep results/*.json in sync with code changes.
  • Per AGENTS.md: a phase is not "complete" until code, validation, and the matching handoff document under docs/ are all updated together.

References

License

MIT


简体中文

项目说明

TurboQuant 是 Google Research 提出的超低精度 KV cache 量化算法(PolarQuant + QJL)。官方实现基于 JAX + CUDA,本项目通过 MLX 将其移植到 Apple Silicon,并提供 PyTorch 参考实现用于一致性比对。

支持论文定义的两种模式:

  • TURBOQUANTmse(mode="mse"):所有 bit 给 PolarQuant。有偏,但在 LLM 推理场景下足够。推荐默认值,社区多个独立实现已验证(TheTom/turboquant_plus)。
  • TURBOQUANTprod(mode="prod"):PolarQuant(b−1 bit)+ 独立 QJL 残差(1 bit)。按 TurboQuant 论文 Theorem 2 是无偏内积估计器,仅用于研究 / 理论验证。

功能特性

  • QJL、PolarQuant、TurboQuant(mse + prod)的 PyTorch 参考实现
  • TurboQuant 在 Apple Silicon 上的 MLX 后端,与 PyTorch 在 4 种 mode × codebook 配置下的最大得分绝对差为 1.79e-7
  • PolarQuant 使用 Lloyd-Max 最优码本(PolarQuant 论文 Section 3)
  • 1000 个随机单位向量上的中等规模验证(tests/test_at_scale.py)
  • docs/ 下按阶段归档的 handoff 文档

路线图

阶段 内容 状态
Phase 1 QJL PyTorch 参考实现 + 验证脚本 已完成
Phase 2 PolarQuant PyTorch 参考 + Lloyd-Max 码本 已完成
Phase 3 TurboQuant 完整两模式 + MLX 后端 已完成
Phase 4 下游集成 已被 gpt2-from-scratch Phase 8/11 替代——TurboQuant 已接入 GPT-2 attention,真实 LLM 推理 Δ PPL < 1.05,KV cache 内存压缩 9.14×

关键指标

由 tests/test_at_scale.py 复现(1000 个随机单位向量,dim=1024,50 个查询),记录在 results/at_scale_validation.json:

方法 bits/dim Recall@5(toy,15 文档) Recall@5(1000 文档) Pearson r
QJL 1-bit 1.0 88% 33.6% 0.796
PolarQuant 2-bit 2.03 88% 46.4% 0.890
TURBOQUANTmse 3-bit 3.03 92% 71.6% 0.971
TURBOQUANTprod 3-bit 3.03 88% 48.8% 0.903

补充验证:

  • MLX vs PyTorch 最大得分绝对差:0.000000(4 个配置全部)
  • TURBOQUANTprod 无偏性:bias = 0.001(< 0.05 阈值)
  • PolarQuant 使用 Lloyd-Max 优化码本(arXiv:2502.02617 Section 3)

安装

需要 Python 3.10+;MLX 后端需要 Apple Silicon,否则只能跑 PyTorch 路径(CPU/MPS)。

pip install -r requirements.txt

requirements.txt 包含 torch、sentence-transformers、numpy、mlx。

快速开始

运行完整验证套件:

python tests/test_qjl.py        # Phase 1 —— QJL PyTorch 基线
python tests/test_polar.py      # Phase 2 —— PolarQuant + Lloyd-Max 码本
python tests/test_turbo.py      # Phase 3 —— TurboQuant mse + prod(PyTorch)
python tests/test_turbo_mlx.py  # Phase 3 —— MLX 后端 parity 验证
python tests/test_at_scale.py   # Phase 3 —— 1000 文档验证

每个测试都会在 results/ 下写入对应的 JSON 报告。

项目结构

qjl-mlx/
├── src/
│   ├── qjl_torch.py          # QJL 参考实现(Phase 1)
│   ├── polar_quant.py        # PolarQuant + Lloyd-Max 码本(Phase 2)
│   ├── turbo_quant.py        # TurboQuant mse + prod(PyTorch,Phase 3)
│   └── turbo_quant_mlx.py    # TurboQuant MLX 后端(Phase 3)
├── tests/
│   ├── test_qjl.py
│   ├── test_polar.py
│   ├── test_turbo.py
│   ├── test_turbo_mlx.py
│   └── test_at_scale.py
├── docs/
│   ├── phase1-handoff.md
│   ├── phase2-handoff.md
│   ├── phase3-handoff.md
│   └── qjl-mlx-fix-plan.md
├── results/                   # 各阶段 JSON 验证输出
├── AGENTS.md                  # 仓库工作准则
├── requirements.txt
└── README.md

配置

本库没有环境变量和配置文件——所有参数都在构造 compressor 时传入。关键参数(详见 src/turbo_quant.py 和 src/turbo_quant_mlx.py):

参数 位置 含义
dim TurboQuantCompressor、TurboQuantMLXCompressor 向量维度(如 1024)
total_bits 同上 每维总 bit 预算;mse 全给 PolarQuant,prod 留 1 bit 给 QJL 残差
mode 同上 "mse"(推荐默认值)或 "prod"(无偏,研究用)
polar_codebook 同上 "analytic"(Lloyd-Max 最优)或 "uniform"
seed 同上 随机种子;prod 模式用 seed+1 生成独立的 QJL 旋转矩阵
device 仅 PyTorch "cpu" / "mps" / "cuda"

开发与测试

仓库没有专门的 linter / CI 配置——验证 = 跑测试套件。按 AGENTS.md 的约定:每个阶段的提交在标记为完成前,必须通过对应的 tests/test_*.py。继续扩展时:在 tests/ 下加测试、在 docs/ 下更新或新增 handoff 文档、在 results/ 下保留指标 JSON。

贡献指南

本仓库是论文移植 / 研究性项目,路线图已完成(Phase 4 被 gpt2-from-scratch 替代)。欢迎 PR,但不主动征集;如果要提 PR,请遵守 AGENTS.md 的约定:

  1. 新行为必须配套 tests/test_*.py 验证脚本。
  2. 在 docs/ 下更新或新增对应阶段的 handoff 文档。
  3. 算法变动时同步重跑并更新 results/*.json。
  4. 一个阶段只有在代码、测试、handoff 文档同时更新后才能称为"完成"。

维护指南

  • 权威信息源:已完成阶段看 docs/phase*-handoff.md,v2 修复历程看 docs/qjl-mlx-fix-plan.md,仓库工作准则看 AGENTS.md。
  • 真实下游消费者:gpt2-from-scratch 的 Phase 8/11——验证了 TurboQuant 在 GPT-2 KV cache 上的表现。如果本仓库 API 变了,记得同步那边的链接。
  • 无 GitHub Actions / CI:所有变更靠本地重跑上面的测试脚本验证;改代码时记得同步 results/*.json。
  • 遵循 AGENTS.md:一个阶段只有在代码、验证、docs/ 下对应的 handoff 文档同时更新后才能称为"完成"。

参考资料

许可证

MIT

About

QJL and TurboQuant KV-cache quantization with MLX and PyTorch reference checks.

Resources

Stars

3 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages