Technical Blog: https://arneshbanerjee.dev/blog/kimi-k3-for-all.html
A small-scale Kimi-K3 trained from scratch on English Wikipedia, using Moonshot's original, unmodified K3 modeling code. The full 2.8T K3 is unrunnable locally; this is a faithful ~1.27B-parameter version (0.36B active) that runs on a laptop and can be fine-tuned for downstream tasks.
The point is architectural fidelity, not just "a small MoE". Every structural switch that makes K3 what it is — lower-bounded KDA decay, Attention Residuals, Gated MLA, Stable LatentMoE, SiTU-GLU, Per-Head Muon — is preserved. Only the scale is reduced.
| Kimi-K3 (official) | this model | |
|---|---|---|
| total params | 2.8T | 1.270B |
| activated params | 104B | 0.364B (ex-embed) |
| layers | 93 | 24 |
| dense layers | 1 | 1 |
| attention | 69 KDA + 24 Gated MLA | 18 KDA + 6 Gated MLA |
| hidden size | 7168 | 1024 |
| attention heads | 96 | 14 |
| experts | 896 | 40 |
| experts / token | 16 | 4 |
| shared experts | 2 | 2 |
| latent MoE dim | 3584 | 512 |
| expert FFN dim | 3072 | 448 |
| vocab | 163,840 | 163,840 |
Training completed 2026-08-05: 15,600 steps, 4.601B tokens = 1.00 epoch of English
Wikipedia (every token seen exactly once). Shipped weights: checkpoints/hf/, 2.4 GiB.
| validation perplexity | 13.89 (CE 2.6313, 102,400 held-out tokens) |
| HellaSwag (zero-shot, 1000 ex.) | 33.7% (random 25.0%) |
| peak VRAM, inference | 3.50 GiB (2048 prefill + KV cache) |
| prefill, 2048 tokens | 206 ms |
| decode, batch 1 / batch 16 | 14.7 / 135.5 tok/s |
Decode is launch-bound rather than compute-bound — 16x the batch gives 9x the throughput,
the same reason training sat near 13% MFU (see Throughput). torch.compile is not applied
at inference and would likely help.
HellaSwag at 33.7% is meaningfully above the 25% floor — the model has learned real commonsense structure, not nothing — but it is far from the ~50%+ a well-trained model of this size reaches. That gap is the token budget, and it is the point of the v2 run.
This model is undertrained (~18% of Chinchilla-optimal) and it shows: it is fluent and
frequently wrong. "World War II began in" continues correctly with September 1939, while
"Photosynthesis is the process by which" degenerates into a circular definition. Treat it
as a base model for study and fine-tuning, not as a general-purpose assistant.
| ratio | K3 | this model |
|---|---|---|
| latent / hidden | 0.5000 | 0.5000 |
| expert FFN / hidden | 0.4286 | 0.4375 |
| attention width / hidden | 1.7143 | 1.7500 |
| KDA : MLA layers | 2.875 | 3.000 |
| final layer is MLA | yes | yes |
K3's per-head geometry is kept exactly — MLA qk_nope=128 / qk_rope=64 / v=128 and KDA
head_dim=128 are unchanged from the released config. Only the head count scales (96 → 14),
which is what preserves the 1.71x attention expansion.
Matched verbatim from the released config.json: gate_lower_bound=-5.0,
attn_res_block_size=12, mla_use_output_gate=true (Gated MLA), mla_use_nope=true,
latent_moe_use_norm=true, hidden_act="situ" with betas 4.0 / 25.0, sigmoid noaux_tc
routing, moe_renormalize=true, first_k_dense_replace=1, rms_norm_eps=1e-5,
tie_word_embeddings=false.
kimi_k3/ SHARED CORE (used by training and inference alike)
kimi_original/ Moonshot's released K3 code, BYTE-IDENTICAL to HuggingFace
config.py our scaled config (the table above)
train/patches.py training-side patches (see "Why patches" below)
train/muon.py Per-Head Muon optimizer
train/mtp.py Multi-Token-Prediction head
training/ everything used to produce the model
train.py the training loop
prepare_data.py Wikipedia -> Kimi-tokenized uint32 memmap shards
download_wiki.py corpus download
launch_training.sh start detached, under the supervisor
supervisor.sh auto-resumes the trainer from checkpoint on crash
stop_training.sh clean stop (checkpoints first, resumable)
check_progress.sh status ( -f to follow live )
inference/ everything needed to RUN the model
generate.py load checkpoints/hf/ and generate text
fla_shim.py pure-PyTorch KDA fallback for Mac/CPU (no Triton)
other/ things you can run to validate or benchmark
benchmark.py speed, VRAM, perplexity, HellaSwag, samples
test_components.py Muon / MTP unit tests (CPU)
test_moe_equiv.py batched MoE == per-expert loop (value + gradients)
gpu_kernel_check.py does fla/Triton actually work on this GPU?
count_params.py total / active parameter counts for a config
sweep_throughput.py throughput + VRAM across batch / seq / checkpointing
probe_moe_load.py per-expert load: dead experts, entropy, imbalance
probe_moe_specialisation.py do experts specialise by content? (JS divergence)
Scripts add the repo root to sys.path themselves, so they run from any working
directory: python inference/generate.py -p "..." works from anywhere.
Directories are configured in paths.conf, not hardcoded. DATA_DIR, CKPT_DIR,
HF_CACHE, LOG_DIR and KIMI_ORIGINAL are read by kimi_k3/paths.py and sourced by
the shell scripts, so there is one file and no drift. Relative values resolve against the
repo root (so a fresh clone works unedited); absolute values are honoured as-is, which is
how you put an 18 GB corpus on a different disk. Environment variables of the same name
override the file for one-off runs. Check what is in effect with python -m kimi_k3.paths.
See also article.md (the full write-up) and CONTRIBUTING.md.
python3 -m venv .venv && source .venv/bin/activate
pip install torch --index-url https://download.pytorch.org/whl/cu126 # needs triton >= 3.3
pip install "transformers==4.57.1" datasets accelerate safetensors tiktoken blobfile einops numpy
pip install -U fla-core
python training/prepare_data.py # downloads + tokenizes Wikipedia (~4.6B tokens, 18 GB)
python training/train.py --smoke # 5-minute end-to-end GPU check
./training/launch_training.sh # the real runpython inference/generate.py --self-test # does it load and generate?
python inference/generate.py -p "The capital of France is"
python inference/generate.py -p "Alan Turing was" --temperature 0.8 --top-p 0.9 -n 80
python inference/generate.py --interactiveinference/generate.py needs only checkpoints/hf/ (2.4 GiB: model.safetensors + config.json) and
the tokenizer in kimi_k3/kimi_original/. Nothing from the training pipeline. It picks the
backend automatically: real fla Triton kernels on CUDA, and inference/fla_shim.py -- a pure
PyTorch KDA implementation -- on Mac/CPU where Triton is unavailable. Both paths were verified
to produce identical text from the same checkpoint.
Decoding uses a KV / recurrent-state cache. Without it every new token re-runs the whole prefix, which is O(n^2) and especially punishing for the pure-PyTorch KDA fallback (it walks the sequence in a Python loop). Cached and uncached outputs are byte-identical; the cache is purely a speed fix.
transformers must be 4.57.x — 5.x breaks the model's internal imports and <4.56 fails its
version assert. fla-core needs Triton >= 3.3, so the cu124 torch wheel (Triton 3.2) will not
work; use cu126 or newer.
kimi_k3/kimi_original/ is kept byte-identical to the HuggingFace release. The released code is
inference-only, so everything needed to train is applied at runtime by
kimi_k3/train/patches.py:
| released code | patch |
|---|---|
KimiMoEGate.forward has assert not self.training |
training-capable gate, identical math |
KimiSparseMoeBlock.forward raises NotImplementedError when training; moe_infer is @torch.no_grad() |
differentiable dispatch |
KimiLinearModel.forward never calls a checkpoint fn, so gradient_checkpointing_enable() is a no-op |
real gradient checkpointing |
dt_bias and e_score_correction_bias are torch.empty(...) and never written by _init_weights |
proper from-scratch init |
The MoE dispatch is also batched (3 bmm calls instead of 3xE small GEMMs). It is verified
bit-identical in forward and gradient-matched to bf16 tolerance against the reference
per-expert loop — see other/test_moe_equiv.py.
torch.compile fuses the elementwise-bound hot spots (RMSNorm / SiTU / AttnRes), worth ~1.45x.
That is a kernel-level change only; the math and parameters are untouched.
Stated plainly, because "faithful" should be falsifiable:
- Quantile Balancing → DeepSeek-style sign update. K3 balances experts via Quantile
Balancing, which is a training-side procedure absent from the released inference code and
described only qualitatively in the paper/blog. We use the aux-loss-free sign update on
e_score_correction_biasinstead. - Expert sparsity. K3 activates 16/896 = 1.8% of experts. With 40 experts the closest workable setting is 4/40 = 10%; 1.8% would require top-1 of 56+, which routes too coarsely to train at this scale.
- Text-only. K3's MoonViT-V2 vision tower (401M) is dropped.
- bf16, not MXFP4/MXFP8. We train in bf16 rather than the released quantized format.
- MTP enabled (
num_nextn_predict_layers=1); the released config ships0. - Attention backend: SDPA rather than FlashAttention-2 (flash-attn needs nvcc to build). Numerically equivalent, different kernel.
Single source of truth — change values here and in supervisor.sh, not in prose elsewhere.
| knob | value | why |
|---|---|---|
| seq len | 2048 | longer is measurably slower (see Throughput) |
| batch x accum | 24 x 6 | 294,912 tokens/step |
--warmup-frac |
0.05 | shorter ramps collapse the router |
--moe-gamma |
0.01 | balancing step size — do not raise, see below |
--ce-chunks |
8 | 4 is ~2% faster but costs +16 GiB |
_BATCHED_MOE_MAX_WASTE |
6.0 | padded-MoE fallback threshold |
| token budget | 4.601B = 0.997 epochs | one true pass, no repetition |
moe_imb (max expert load / mean) is the health signal for this architecture. Healthy is
~1.5-2.5. The ceiling is num_experts / top_k (10.0 here) and means total collapse.
Collapse during warmup. With a short LR ramp the router collapses onto a handful of experts
within ~20 steps and pins at the ceiling, leaving ~90% of the model dead. The trap: warmup is a
fraction of total steps, so shortening a run silently shortens warmup — dropping max_steps
from 100k to 2.2k took warmup from ~1000 steps to 22 and collapsed the run. Fixed by
--warmup-frac 0.05.
Slow upward drift — and why it was a bug, not a tuning problem. Imbalance crept from ~1.5 to
5.9 over 4000 steps. Inspecting +4.4 with a spread of only 0.09**. Since top-k is invariant to
adding a constant to all scores, essentially all of that magnitude was doing nothing while the
differential signal that actually balances load stayed tiny. Two defects, both now fixed:e_score_correction_bias in a checkpoint showed the cause: every
expert's bias had drifted to **
- Counts were overwritten, not accumulated. The gate stored per-expert counts on every
forward, so with
grad_accum=6the balancing step only ever saw the last micro-batch — 1/6 of the tokens. The sign update was largely chasing sampling noise. - The update was not zero-centred. A plain
+-gammasign update drifts in common mode whenever the counts of over- and under-loaded experts differ. With 40 experts at top-4, ~36 sit below the mean each step, pushing the whole vector up by ~0.8*gamma per step. The update is now mean-subtracted, which is exactly routing-neutral and keeps the bias centred.
recentre_moe_bias() clears accumulated common-mode drift from an old checkpoint on resume.
Do not "fix" imbalance by raising
--moe-gamma. It was tried at 0.02 and collapsed the router to 10.0 within one step, costing ~40 steps of progress. The sign update never converges -- it random-walks with step size gamma -- so a larger gamma means a larger oscillation amplitude and worse instantaneous balance. Lowering the dispatch threshold to 3.5 was tried at the same time and also reverted: it raised VRAM rather than lowering it. Sustained imbalance is a routing problem; fix it at the router, not at the dispatcher.
moe_imb is max_load / mean_load of the single worst layer, not a model-wide figure. It
cannot distinguish two completely different states, both of which read exactly 10.0 here:
- 4 experts absorb every token -> ~90% of the model is dead, or
- 1 expert is picked by every token while the other slots spread normally -> harmless.
Measured directly at step 7531 with other/probe_moe_load.py:
| measured | uniform | collapsed | |
|---|---|---|---|
| busiest expert share | 11.7% | 2.5% | 25% |
| top-4 share | 31.2% | 10% | 100% |
| dead experts | 0 / 40 | 0 | 36 |
| routing entropy | 0.897 | 1.0 | ~0 |
Per layer the imbalance runs 1.80 (layer 0) to 8.34 (layer 22, the outlier being reported), mean
4.68. So a logged moe_imb of ~9.9 never meant the model was dead -- it meant one deep layer was
concentrated. Prefer routing entropy or dead-expert count. max/mean of the worst layer cost
several false alarms and one bad change here.
other/probe_moe_specialisation.py measures Jensen-Shannon divergence of expert usage across text
domains, against a noise floor taken from two halves of the same text:
| layers | within-domain (noise) | cross-domain | ratio |
|---|---|---|---|
| 0-2 | 0.11 | 0.10 | 0.9x (none) |
| 5 | 0.09 | 0.26 | 2.7x |
| 21 | 0.10 | 0.35 | 3.4x |
| mean | 0.097 | 0.222 | 2.29x |
Routing is content-dependent and sharpens with depth -- biography and mathematics select disjoint expert sets, while mathematics and code share experts. But the first three MoE layers show no specialisation at all, and 2.29x overall is modest. At ~18% of Chinchilla-optimal, expert specialisation is still forming. A dense ~0.4B model would plausibly match this one today; the MoE is here for fidelity to K3, and its capacity advantage needs more tokens to materialise.
Sampling random offsets with replacement only touches 1-1/e ~ 63% of unique tokens per nominal
epoch. TokenLoader sweeps shuffled non-overlapping windows instead, so one pass is a true
epoch, and the sweep position is checkpointed so a resume continues rather than restarts it.
~17.5k tok/s on one H100 at seq 2048 (1 epoch ~ 70 h), which is only ~13% "MFU" — but MFU is the
wrong metric here. Measured with other/sweep_throughput.py:
| config | tok/s | peak VRAM |
|---|---|---|
| seq2048 bs12 grad-ckpt on | 16.0k | 14.8 GiB |
| seq2048 bs8 grad-ckpt off | 15.4k | 54.4 GiB |
| seq 4096 bs12 | 15.3k | 24.4 GiB |
| seq 8192 bs6 | 11.1k | 25.0 GiB |
seq2048 bs24 ce_chunks 4 |
17.5k | 41.0 GiB |
seq2048 bs24 ce_chunks 8 |
17.1k | 24.9 GiB (chosen) |
Two results to internalise before "optimising" this:
- Disabling gradient checkpointing is SLOWER — it removes ~19% of the FLOPs and still loses, because activations balloon to 54 GiB and bandwidth/allocator pressure dominates. That is the proof the model is not FLOP-bound.
- Longer sequences hurt (-4% at 4096, -31% at 8192): MLA is O(n^2) and activations grow, while the KDA layers gain nothing.
Structural cause: hidden_size=1024 keeps every GEMM small (1024x1024, 512x448) where an H100
wants >=4096, and K3 is deliberately elementwise-heavy — fp32 SiTU tanh+sigmoid, fp32
RMSNorm, and an fp32 AttnRes softmax at every layer. All burn bandwidth, none add FLOPs.
What did work, by payoff: batched MoE dispatch, chunked CE, removing hidden device->host syncs
(bincount/nonzero size their output from data and stall the pipeline), and torch.compile on
the elementwise hot spots. Together 4.4k -> 17.5k tok/s (~4x).
Checkpoints are fully resumable (model + MTP + both optimizers + step + RNG + data position),
written atomically every 20 min, with weights-only snapshots every 6 h. supervisor.sh restarts
the trainer from the last checkpoint on crash, treats signal exits as deliberate, and gives up
after 10 attempts with no progress rather than looping forever. stop_training.sh stops cleanly:
the trainer finishes its step, checkpoints, and exits.
kimi_k3/kimi_original/ is Moonshot AI's released Kimi-K3 code, redistributed unmodified
under the license in kimi_k3/kimi_original/LICENSE (Kimi K3 License; the MLA / MoE gating
portions are adapted from DeepSeek-V3 under Apache 2.0). All credit for the architecture belongs
to Moonshot AI.
- Model: https://huggingface.co/moonshotai/Kimi-K3
- Paper: https://arxiv.org/abs/2607.24653
- Blog: https://www.kimi.com/blog/kimi-k3
- Article: https://x.com/waterloo_intern/status/2081762065392541951
Training data: wikimedia/wikipedia
(20231101.en), CC BY-SA 3.0 / GFDL.