Skip to content

About

Kimi K3 training pipeline replicated from scratch.

Resources

Contributing

Stars

10 stars

Watchers

0 watching

Forks

Latest commit

 

History

18 Commits

Folders and files

Repository files navigation

Kimi K3 for All

Technical Blog: https://arneshbanerjee.dev/blog/kimi-k3-for-all.html

A small-scale Kimi-K3 trained from scratch on English Wikipedia, using Moonshot's original, unmodified K3 modeling code. The full 2.8T K3 is unrunnable locally; this is a faithful ~1.27B-parameter version (0.36B active) that runs on a laptop and can be fine-tuned for downstream tasks.

The point is architectural fidelity, not just "a small MoE". Every structural switch that makes K3 what it is — lower-bounded KDA decay, Attention Residuals, Gated MLA, Stable LatentMoE, SiTU-GLU, Per-Head Muon — is preserved. Only the scale is reduced.


Model

Kimi-K3 (official) this model
total params 2.8T 1.270B
activated params 104B 0.364B (ex-embed)
layers 93 24
dense layers 1 1
attention 69 KDA + 24 Gated MLA 18 KDA + 6 Gated MLA
hidden size 7168 1024
attention heads 96 14
experts 896 40
experts / token 16 4
shared experts 2 2
latent MoE dim 3584 512
expert FFN dim 3072 448
vocab 163,840 163,840

Training completed 2026-08-05: 15,600 steps, 4.601B tokens = 1.00 epoch of English Wikipedia (every token seen exactly once). Shipped weights: checkpoints/hf/, 2.4 GiB.

Measured (python other/benchmark.py, H100 bf16)

validation perplexity 13.89 (CE 2.6313, 102,400 held-out tokens)
HellaSwag (zero-shot, 1000 ex.) 33.7% (random 25.0%)
peak VRAM, inference 3.50 GiB (2048 prefill + KV cache)
prefill, 2048 tokens 206 ms
decode, batch 1 / batch 16 14.7 / 135.5 tok/s

Decode is launch-bound rather than compute-bound — 16x the batch gives 9x the throughput, the same reason training sat near 13% MFU (see Throughput). torch.compile is not applied at inference and would likely help.

HellaSwag at 33.7% is meaningfully above the 25% floor — the model has learned real commonsense structure, not nothing — but it is far from the ~50%+ a well-trained model of this size reaches. That gap is the token budget, and it is the point of the v2 run.

This model is undertrained (~18% of Chinchilla-optimal) and it shows: it is fluent and frequently wrong. "World War II began in" continues correctly with September 1939, while "Photosynthesis is the process by which" degenerates into a circular definition. Treat it as a base model for study and fine-tuning, not as a general-purpose assistant.

Ratios preserved under scaling

ratio K3 this model
latent / hidden 0.5000 0.5000
expert FFN / hidden 0.4286 0.4375
attention width / hidden 1.7143 1.7500
KDA : MLA layers 2.875 3.000
final layer is MLA yes yes

K3's per-head geometry is kept exactly — MLA qk_nope=128 / qk_rope=64 / v=128 and KDA head_dim=128 are unchanged from the released config. Only the head count scales (96 → 14), which is what preserves the 1.71x attention expansion.

Matched verbatim from the released config.json: gate_lower_bound=-5.0, attn_res_block_size=12, mla_use_output_gate=true (Gated MLA), mla_use_nope=true, latent_moe_use_norm=true, hidden_act="situ" with betas 4.0 / 25.0, sigmoid noaux_tc routing, moe_renormalize=true, first_k_dense_replace=1, rms_norm_eps=1e-5, tie_word_embeddings=false.


Layout

kimi_k3/                 SHARED CORE (used by training and inference alike)
  kimi_original/         Moonshot's released K3 code, BYTE-IDENTICAL to HuggingFace
  config.py              our scaled config (the table above)
  train/patches.py       training-side patches (see "Why patches" below)
  train/muon.py          Per-Head Muon optimizer
  train/mtp.py           Multi-Token-Prediction head

training/                everything used to produce the model
  train.py               the training loop
  prepare_data.py        Wikipedia -> Kimi-tokenized uint32 memmap shards
  download_wiki.py       corpus download
  launch_training.sh     start detached, under the supervisor
  supervisor.sh          auto-resumes the trainer from checkpoint on crash
  stop_training.sh       clean stop (checkpoints first, resumable)
  check_progress.sh      status ( -f to follow live )

inference/               everything needed to RUN the model
  generate.py            load checkpoints/hf/ and generate text
  fla_shim.py            pure-PyTorch KDA fallback for Mac/CPU (no Triton)

other/                   things you can run to validate or benchmark
  benchmark.py           speed, VRAM, perplexity, HellaSwag, samples
  test_components.py     Muon / MTP unit tests (CPU)
  test_moe_equiv.py      batched MoE == per-expert loop (value + gradients)
  gpu_kernel_check.py    does fla/Triton actually work on this GPU?
  count_params.py        total / active parameter counts for a config
  sweep_throughput.py    throughput + VRAM across batch / seq / checkpointing
  probe_moe_load.py      per-expert load: dead experts, entropy, imbalance
  probe_moe_specialisation.py  do experts specialise by content? (JS divergence)

Scripts add the repo root to sys.path themselves, so they run from any working directory: python inference/generate.py -p "..." works from anywhere.

Directories are configured in paths.conf, not hardcoded. DATA_DIR, CKPT_DIR, HF_CACHE, LOG_DIR and KIMI_ORIGINAL are read by kimi_k3/paths.py and sourced by the shell scripts, so there is one file and no drift. Relative values resolve against the repo root (so a fresh clone works unedited); absolute values are honoured as-is, which is how you put an 18 GB corpus on a different disk. Environment variables of the same name override the file for one-off runs. Check what is in effect with python -m kimi_k3.paths.

See also article.md (the full write-up) and CONTRIBUTING.md.

Quick start

python3 -m venv .venv && source .venv/bin/activate
pip install torch --index-url https://download.pytorch.org/whl/cu126   # needs triton >= 3.3
pip install "transformers==4.57.1" datasets accelerate safetensors tiktoken blobfile einops numpy
pip install -U fla-core

python training/prepare_data.py            # downloads + tokenizes Wikipedia (~4.6B tokens, 18 GB)
python training/train.py --smoke           # 5-minute end-to-end GPU check
./training/launch_training.sh              # the real run

Running the model (this is the point of the project)

python inference/generate.py --self-test                       # does it load and generate?
python inference/generate.py -p "The capital of France is"
python inference/generate.py -p "Alan Turing was" --temperature 0.8 --top-p 0.9 -n 80
python inference/generate.py --interactive

inference/generate.py needs only checkpoints/hf/ (2.4 GiB: model.safetensors + config.json) and the tokenizer in kimi_k3/kimi_original/. Nothing from the training pipeline. It picks the backend automatically: real fla Triton kernels on CUDA, and inference/fla_shim.py -- a pure PyTorch KDA implementation -- on Mac/CPU where Triton is unavailable. Both paths were verified to produce identical text from the same checkpoint.

Decoding uses a KV / recurrent-state cache. Without it every new token re-runs the whole prefix, which is O(n^2) and especially punishing for the pure-PyTorch KDA fallback (it walks the sequence in a Python loop). Cached and uncached outputs are byte-identical; the cache is purely a speed fix.

transformers must be 4.57.x — 5.x breaks the model's internal imports and <4.56 fails its version assert. fla-core needs Triton >= 3.3, so the cu124 torch wheel (Triton 3.2) will not work; use cu126 or newer.


Why patches instead of editing the model

kimi_k3/kimi_original/ is kept byte-identical to the HuggingFace release. The released code is inference-only, so everything needed to train is applied at runtime by kimi_k3/train/patches.py:

released code patch
KimiMoEGate.forward has assert not self.training training-capable gate, identical math
KimiSparseMoeBlock.forward raises NotImplementedError when training; moe_infer is @torch.no_grad() differentiable dispatch
KimiLinearModel.forward never calls a checkpoint fn, so gradient_checkpointing_enable() is a no-op real gradient checkpointing
dt_bias and e_score_correction_bias are torch.empty(...) and never written by _init_weights proper from-scratch init

The MoE dispatch is also batched (3 bmm calls instead of 3xE small GEMMs). It is verified bit-identical in forward and gradient-matched to bf16 tolerance against the reference per-expert loop — see other/test_moe_equiv.py.

torch.compile fuses the elementwise-bound hot spots (RMSNorm / SiTU / AttnRes), worth ~1.45x. That is a kernel-level change only; the math and parameters are untouched.


Known deviations from K3

Stated plainly, because "faithful" should be falsifiable:

  1. Quantile Balancing → DeepSeek-style sign update. K3 balances experts via Quantile Balancing, which is a training-side procedure absent from the released inference code and described only qualitatively in the paper/blog. We use the aux-loss-free sign update on e_score_correction_bias instead.
  2. Expert sparsity. K3 activates 16/896 = 1.8% of experts. With 40 experts the closest workable setting is 4/40 = 10%; 1.8% would require top-1 of 56+, which routes too coarsely to train at this scale.
  3. Text-only. K3's MoonViT-V2 vision tower (401M) is dropped.
  4. bf16, not MXFP4/MXFP8. We train in bf16 rather than the released quantized format.
  5. MTP enabled (num_nextn_predict_layers=1); the released config ships 0.
  6. Attention backend: SDPA rather than FlashAttention-2 (flash-attn needs nvcc to build). Numerically equivalent, different kernel.

Training notes

Current settings

Single source of truth — change values here and in supervisor.sh, not in prose elsewhere.

knob value why
seq len 2048 longer is measurably slower (see Throughput)
batch x accum 24 x 6 294,912 tokens/step
--warmup-frac 0.05 shorter ramps collapse the router
--moe-gamma 0.01 balancing step size — do not raise, see below
--ce-chunks 8 4 is ~2% faster but costs +16 GiB
_BATCHED_MOE_MAX_WASTE 6.0 padded-MoE fallback threshold
token budget 4.601B = 0.997 epochs one true pass, no repetition

MoE routing — the number to watch

moe_imb (max expert load / mean) is the health signal for this architecture. Healthy is ~1.5-2.5. The ceiling is num_experts / top_k (10.0 here) and means total collapse.

Collapse during warmup. With a short LR ramp the router collapses onto a handful of experts within ~20 steps and pins at the ceiling, leaving ~90% of the model dead. The trap: warmup is a fraction of total steps, so shortening a run silently shortens warmup — dropping max_steps from 100k to 2.2k took warmup from ~1000 steps to 22 and collapsed the run. Fixed by --warmup-frac 0.05.

Slow upward drift — and why it was a bug, not a tuning problem. Imbalance crept from ~1.5 to 5.9 over 4000 steps. Inspecting e_score_correction_bias in a checkpoint showed the cause: every expert's bias had drifted to **+4.4 with a spread of only 0.09**. Since top-k is invariant to adding a constant to all scores, essentially all of that magnitude was doing nothing while the differential signal that actually balances load stayed tiny. Two defects, both now fixed:

  • Counts were overwritten, not accumulated. The gate stored per-expert counts on every forward, so with grad_accum=6 the balancing step only ever saw the last micro-batch — 1/6 of the tokens. The sign update was largely chasing sampling noise.
  • The update was not zero-centred. A plain +-gamma sign update drifts in common mode whenever the counts of over- and under-loaded experts differ. With 40 experts at top-4, ~36 sit below the mean each step, pushing the whole vector up by ~0.8*gamma per step. The update is now mean-subtracted, which is exactly routing-neutral and keeps the bias centred.

recentre_moe_bias() clears accumulated common-mode drift from an old checkpoint on resume.

Do not "fix" imbalance by raising --moe-gamma. It was tried at 0.02 and collapsed the router to 10.0 within one step, costing ~40 steps of progress. The sign update never converges -- it random-walks with step size gamma -- so a larger gamma means a larger oscillation amplitude and worse instantaneous balance. Lowering the dispatch threshold to 3.5 was tried at the same time and also reverted: it raised VRAM rather than lowering it. Sustained imbalance is a routing problem; fix it at the router, not at the dispatcher.

moe_imb is a bad metric -- read this before reacting to it

moe_imb is max_load / mean_load of the single worst layer, not a model-wide figure. It cannot distinguish two completely different states, both of which read exactly 10.0 here:

  • 4 experts absorb every token -> ~90% of the model is dead, or
  • 1 expert is picked by every token while the other slots spread normally -> harmless.

Measured directly at step 7531 with other/probe_moe_load.py:

measured uniform collapsed
busiest expert share 11.7% 2.5% 25%
top-4 share 31.2% 10% 100%
dead experts 0 / 40 0 36
routing entropy 0.897 1.0 ~0

Per layer the imbalance runs 1.80 (layer 0) to 8.34 (layer 22, the outlier being reported), mean 4.68. So a logged moe_imb of ~9.9 never meant the model was dead -- it meant one deep layer was concentrated. Prefer routing entropy or dead-expert count. max/mean of the worst layer cost several false alarms and one bad change here.

Is the MoE earning its parameters?

other/probe_moe_specialisation.py measures Jensen-Shannon divergence of expert usage across text domains, against a noise floor taken from two halves of the same text:

layers within-domain (noise) cross-domain ratio
0-2 0.11 0.10 0.9x (none)
5 0.09 0.26 2.7x
21 0.10 0.35 3.4x
mean 0.097 0.222 2.29x

Routing is content-dependent and sharpens with depth -- biography and mathematics select disjoint expert sets, while mathematics and code share experts. But the first three MoE layers show no specialisation at all, and 2.29x overall is modest. At ~18% of Chinchilla-optimal, expert specialisation is still forming. A dense ~0.4B model would plausibly match this one today; the MoE is here for fidelity to K3, and its capacity advantage needs more tokens to materialise.

Data coverage

Sampling random offsets with replacement only touches 1-1/e ~ 63% of unique tokens per nominal epoch. TokenLoader sweeps shuffled non-overlapping windows instead, so one pass is a true epoch, and the sweep position is checkpointed so a resume continues rather than restarts it.

Throughput — bandwidth-bound, not FLOP-bound

~17.5k tok/s on one H100 at seq 2048 (1 epoch ~ 70 h), which is only ~13% "MFU" — but MFU is the wrong metric here. Measured with other/sweep_throughput.py:

config tok/s peak VRAM
seq2048 bs12 grad-ckpt on 16.0k 14.8 GiB
seq2048 bs8 grad-ckpt off 15.4k 54.4 GiB
seq 4096 bs12 15.3k 24.4 GiB
seq 8192 bs6 11.1k 25.0 GiB
seq2048 bs24 ce_chunks 4 17.5k 41.0 GiB
seq2048 bs24 ce_chunks 8 17.1k 24.9 GiB (chosen)

Two results to internalise before "optimising" this:

  • Disabling gradient checkpointing is SLOWER — it removes ~19% of the FLOPs and still loses, because activations balloon to 54 GiB and bandwidth/allocator pressure dominates. That is the proof the model is not FLOP-bound.
  • Longer sequences hurt (-4% at 4096, -31% at 8192): MLA is O(n^2) and activations grow, while the KDA layers gain nothing.

Structural cause: hidden_size=1024 keeps every GEMM small (1024x1024, 512x448) where an H100 wants >=4096, and K3 is deliberately elementwise-heavy — fp32 SiTU tanh+sigmoid, fp32 RMSNorm, and an fp32 AttnRes softmax at every layer. All burn bandwidth, none add FLOPs.

What did work, by payoff: batched MoE dispatch, chunked CE, removing hidden device->host syncs (bincount/nonzero size their output from data and stall the pipeline), and torch.compile on the elementwise hot spots. Together 4.4k -> 17.5k tok/s (~4x).

Checkpointing and recovery

Checkpoints are fully resumable (model + MTP + both optimizers + step + RNG + data position), written atomically every 20 min, with weights-only snapshots every 6 h. supervisor.sh restarts the trainer from the last checkpoint on crash, treats signal exits as deliberate, and gives up after 10 attempts with no progress rather than looping forever. stop_training.sh stops cleanly: the trainer finishes its step, checkpoints, and exits.


Credits and licensing

kimi_k3/kimi_original/ is Moonshot AI's released Kimi-K3 code, redistributed unmodified under the license in kimi_k3/kimi_original/LICENSE (Kimi K3 License; the MLA / MoE gating portions are adapted from DeepSeek-V3 under Apache 2.0). All credit for the architecture belongs to Moonshot AI.

Training data: wikimedia/wikipedia (20231101.en), CC BY-SA 3.0 / GFDL.

About

Kimi K3 training pipeline replicated from scratch.

Resources

Contributing

Stars

10 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages