Skip to content

About

GPT-2 Small (124M) training, LoRA, evaluation, and inference experiments on Apple Silicon.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

GPT-2 from scratch on a MacBook Air — 1B tokens, $0 cloud cost

I rebuilt GPT-2 Small (124M) end-to-end on a MacBook Air M5 / 32GB: data prep, pre-training, LoRA, evaluation, KV cache quantization, MLX port, and a 1B-token FineWeb run comparing AdamW vs Muon. No cloud GPUs, no API keys, no shortcuts.

This repo is the full reproducible artifact.

TL;DR — what surprised me

  1. Muon lost to AdamW on a real 1B-token run. Despite the nanogpt-speedrun results, on MLX/M5 Muon ended at val_loss 5.43 vs AdamW 4.41 (PPL 228 vs 82) across 10K identical steps. The optimizer is hardware- and config-sensitive in ways the original paper doesn't advertise.
  2. TurboQuant compresses the KV cache 9.14× with Δ PPL < 1.05. I ported QJL + TurboQuant from JAX/CUDA to MLX (qjl-mlx) and validated it inside this GPT-2's attention. Quality loss is effectively zero.
  3. MLX hits 98% wall-clock efficiency on M5. Going from PyTorch MPS → MLX Metal took throughput from ~597 tok/s (with sleep/GC pauses) to 2,032 tok/s — a 3.4× speedup. caffeinate -i -s and renice -20 matter more than people think.
  4. HellaSwag at 124M is honestly ~24%. Don't expect emergent behavior. The point isn't SOTA, it's a working end-to-end pipeline you can actually read.

Why this exists

Most "GPT-2 from scratch" tutorials stop at pre-training on a toy dataset. Real LLM engineering is the other 80%: tokenization edge cases, MPS pauses, optimizer instability at scale, KV cache that doesn't fit in 32GB, and the moment your loss goes NaN at step 6,200.

I went through all of it on consumer hardware so you can copy the parts that work.

Headline numbers

FineWeb 1B-token run — AdamW vs Muon (10K steps each, identical config)

AdamW Muon
Best val loss 4.408 @ 8,300 5.431 @ 8,300
Best PPL 82 228
Final val loss 4.717 5.873
Avg throughput 2,079 tok/s 1,946 tok/s
Wall clock 22h 12m 23h 43m

Full per-step .jsonl logs in logs/.

TurboQuant KV cache compression (3-bit MSE, self-authored)

seq_len Exact PPL TurboQuant PPL Δ PPL Memory ratio
256 243.04 244.04 +1.00 9.14×
512 221.50 222.54 +1.04 9.14×
1024 193.10 193.91 +0.81 9.14×

MLX vs PyTorch MPS

Backend tok/s Efficiency
PyTorch MPS (sleep/GC pauses) ~597 ~40%
PyTorch MPS (caffeinate -i) ~1,800 ~92%
MLX Metal 2,032 98.3%

The 11 phases

# Topic Framework
1 BPE tokenization + sliding-window shards PyTorch
2 GPT-2 Small from 5 classes PyTorch
3 Pre-training with AdamW (WikiText-2) PyTorch / MPS
4 Hand-written LoRA (0.35% trainable params, no PEFT) PyTorch
5 HuggingFace safetensors export PyTorch
6 Inference + sampling strategies PyTorch
7 Evaluation — PPL + HellaSwag (lm-eval) —
8 KV cache quantization via TurboQuant PyTorch
9 MLX port + parity vs PyTorch MLX
10 FineWeb 1B training — AdamW vs Muon MLX
11 TurboQuant inference benchmark PyTorch

Each phase has a handoff doc in docs/ and structured JSON results in results/.

Reproduce

pip install torch tiktoken transformers datasets huggingface_hub mlx mlx-lm lm-eval

Phase 9 — MLX validation (2K steps, ~30min):

caffeinate -i -s python mlx_train.py --dataset wikitext2 --optimizer adamw --max_steps 2000
# expect: val_loss ≈ 5.61, ~2,032 tok/s, ≥97% efficiency

Phase 10 — FineWeb 1B run (requires HF token, ~22h each):

HF_TOKEN=

About

GPT-2 Small (124M) training, LoRA, evaluation, and inference experiments on Apple Silicon.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages