I rebuilt GPT-2 Small (124M) end-to-end on a MacBook Air M5 / 32GB: data prep, pre-training, LoRA, evaluation, KV cache quantization, MLX port, and a 1B-token FineWeb run comparing AdamW vs Muon. No cloud GPUs, no API keys, no shortcuts.
This repo is the full reproducible artifact.
- Muon lost to AdamW on a real 1B-token run. Despite the nanogpt-speedrun results, on MLX/M5 Muon ended at val_loss 5.43 vs AdamW 4.41 (PPL 228 vs 82) across 10K identical steps. The optimizer is hardware- and config-sensitive in ways the original paper doesn't advertise.
- TurboQuant compresses the KV cache 9.14× with Δ PPL < 1.05. I ported QJL + TurboQuant from JAX/CUDA to MLX (qjl-mlx) and validated it inside this GPT-2's attention. Quality loss is effectively zero.
- MLX hits 98% wall-clock efficiency on M5. Going from PyTorch MPS → MLX Metal took throughput from ~597 tok/s (with sleep/GC pauses) to 2,032 tok/s — a 3.4× speedup.
caffeinate -i -sandrenice -20matter more than people think. - HellaSwag at 124M is honestly ~24%. Don't expect emergent behavior. The point isn't SOTA, it's a working end-to-end pipeline you can actually read.
Most "GPT-2 from scratch" tutorials stop at pre-training on a toy dataset. Real LLM engineering is the other 80%: tokenization edge cases, MPS pauses, optimizer instability at scale, KV cache that doesn't fit in 32GB, and the moment your loss goes NaN at step 6,200.
I went through all of it on consumer hardware so you can copy the parts that work.
| AdamW | Muon | |
|---|---|---|
| Best val loss | 4.408 @ 8,300 | 5.431 @ 8,300 |
| Best PPL | 82 | 228 |
| Final val loss | 4.717 | 5.873 |
| Avg throughput | 2,079 tok/s | 1,946 tok/s |
| Wall clock | 22h 12m | 23h 43m |
Full per-step .jsonl logs in logs/.
| seq_len | Exact PPL | TurboQuant PPL | Δ PPL | Memory ratio |
|---|---|---|---|---|
| 256 | 243.04 | 244.04 | +1.00 | 9.14× |
| 512 | 221.50 | 222.54 | +1.04 | 9.14× |
| 1024 | 193.10 | 193.91 | +0.81 | 9.14× |
| Backend | tok/s | Efficiency |
|---|---|---|
| PyTorch MPS (sleep/GC pauses) | ~597 | ~40% |
PyTorch MPS (caffeinate -i) |
~1,800 | ~92% |
| MLX Metal | 2,032 | 98.3% |
| # | Topic | Framework |
|---|---|---|
| 1 | BPE tokenization + sliding-window shards | PyTorch |
| 2 | GPT-2 Small from 5 classes | PyTorch |
| 3 | Pre-training with AdamW (WikiText-2) | PyTorch / MPS |
| 4 | Hand-written LoRA (0.35% trainable params, no PEFT) | PyTorch |
| 5 | HuggingFace safetensors export | PyTorch |
| 6 | Inference + sampling strategies | PyTorch |
| 7 | Evaluation — PPL + HellaSwag (lm-eval) | — |
| 8 | KV cache quantization via TurboQuant | PyTorch |
| 9 | MLX port + parity vs PyTorch | MLX |
| 10 | FineWeb 1B training — AdamW vs Muon | MLX |
| 11 | TurboQuant inference benchmark | PyTorch |
Each phase has a handoff doc in docs/ and structured JSON results in results/.
pip install torch tiktoken transformers datasets huggingface_hub mlx mlx-lm lm-evalPhase 9 — MLX validation (2K steps, ~30min):
caffeinate -i -s python mlx_train.py --dataset wikitext2 --optimizer adamw --max_steps 2000
# expect: val_loss ≈ 5.61, ~2,032 tok/s, ≥97% efficiencyPhase 10 — FineWeb 1B run (requires HF token, ~22h each):
HF_TOKEN=