LLM inference that actually exploits NVIDIA H200 / Hopper — not a generic CUDA port.
Teams with H200 (and other Hopper) capacity who want DeepSeek-V4.1-Flash to ride SM90 + HBM3e hard — inference platforms chasing real tokens/s, not a checkbox “CUDA builds.”
Upstream DeepInfer is DeepSeek’s production-grade offline engine. The public tree targets SM100 and Ascend 950. Hopper (H200 / H100 / H20, SM90) is out of scope there.
DeepInfer Hopper is a personal, public adaptation focused on:
- Landing an SM90 path (building on community work such as DeepInfer#92)
- Tuning for H200 first, then A100
- Reproducible throughput benches — accept rate, decode tokens/s, memory — not vibes
git clone --recurse-submodules https://github.com/kabishou11/deepinfer-hopper.git
cd deepinfer-hopper
export DEEPINFER_ACCELERATOR=nvidia
export DEEPINFER_CUDA_ARCH=sm90
./build.sh
python examples/offline_inference.py --ep 8Checkpoint (upstream): deepseek-ai/DeepSeek-V4.1-Flash-1017. Full Hopper notes: doc/sm90.md, doc/h200.md, doc/h100.md. Named layouts: configs/h200 (141 GB), configs/h100 (H100 SXM ~80 GB). Offline driver: benchmarks/offline. The unmodified upstream README is at docs/upstream-README.md.
H200-first. Same SM90 family as the H20-3e path in DeepInfer#92; that series is what this tree ports. Do not treat H20-3e numbers as H200 numbers. H100 shares that SM90 path but not the 141 GB EP/concurrency table — see doc/h100.md.
Prerequisites
| Item | Requirement |
|---|---|
| OS | Linux |
| Python | 3.12, empty venv |
| Compiler | GCC >= 13 |
| CUDA toolkit | 13.x (pinned PyTorch is torch==2.13.0+cu132) |
| Driver | R590 (CUDA minor-version compatibility), or R570 plus cuda-compat-13-2 with that libcuda first on LD_LIBRARY_PATH |
| Extra headers | libdw-dev (DeepGEMM and DeepEP) |
| Submodules | clone with --recurse-submodules, or git submodule update --init --recursive |
H200-priority switches
export DEEPINFER_ACCELERATOR=nvidia
export DEEPINFER_CUDA_ARCH=sm90 # set this on CPU-only build hosts; Hopper boxes detect 9.x as sm90
./build.shOn sm90, build.sh installs requirements-sm90.txt, builds flash_mla_sm90 from vendor/FlashMLA-SM90 (FlashMLA ba89a34 plus the V4.1 patch), and applies the SM90 vendor patches under scripts/vendor_patches/.
Verified vs expected
| What | Status |
|---|---|
| End-to-end DeepSeek-V4.1-Flash on 8x / 4x H20-3e | Verified in DeepInfer#92 (doc/sm90.md) |
Same SM90 code paths on H200 (PLATFORM.is_sm90, sm90_ops/) |
Expected — H200 is compute capability 9.0 |
get_bf16_tflops_limit() |
989 TFLOPS on H200; 148 TFLOPS only when the NVML name matches H20 / H20-3e as a whole word |
| FP8 GEMM launch configs | Tuned on H20-3e; other SM90 parts (including H200) use the heuristic fallback. CPU-safe sweep plan: benchmarks/kernel_sweeps. No H200 winner tiles in this tree |
humming-kernels |
#92 validated 0.1.12 on H20-3e. H200 testers reported a misaligned local-memory read in 0.1.12; this tree pins 0.1.13, which includes that alignment fix. Full-model H200 still unverified |
| H200 tok/s, TTFT, GSM8K | Not measured here. Phase 2 runbook + CPU bring-up check are in-tree (docs/h200-phase2-runbook.md, scripts/sm90_bringup_check.py; scripts/h200_bringup_check.py is the H200 alias); live numbers stay TBD |
If you skipped --recurse-submodules on clone:
git submodule update --init --recursive| Phase | Focus | Status |
|---|---|---|
| 0 | Branding + positioning | ✅ |
| 1 | Import DeepInfer + SM90 path, H200/H100 config matrix, offline bench scaffolding | ✅ |
| 2 | H200 bring-up & baselines | 🚧 runbook + CPU bring-up/CI ready; blocked on hardware — no tok/s / TTFT / VRAM in this tree. See docs/h200-phase2-runbook.md |
| 3 | Hopper kernels / graphs / MTP · FP8·NVFP4 where it wins | 🚧 operator wiring + MTP/DSpark/CUDA-graph scaffold + watchlists + NVFP4 note in-tree; live H200 numbers and graph-capture proof still Phase 2 |
| 4 | Multi-node layouts · A100 (SM80) | later |
Personal fork by kabishou11. Not affiliated with DeepSeek or NVIDIA. DeepInfer remains © DeepSeek under Apache-2.0; NVIDIA / Hopper / H200 are trademarks of their owners. No claim of production parity with DeepSeek’s full serving stack (KV store, PD disaggregation, and LB are still out of the public upstream).
Issues and PRs welcome. Prefer small, measured changes with benches.
Apache-2.0 for imported DeepInfer code (see LICENSE). Logo and docs in this repository © Zhu Lei (kabishou11), unless noted.
Hopper-first. Throughput-obsessed. Built in public.