* Fix ut mla-test-1-gpu-amd (sgl-project#4813)

Co-authored-by: Zhang Kaihong <[email protected]>

* Remove Unintended Capture Batch Sizes in AMD HIP Graph Runner (sgl-project#4638)

* [k8s] Clarified the usage of shared memory. (sgl-project#4341)

* gemma3: impl `get_attention_sliding_window_size` for attn init (sgl-project#4823)

* add partial_json_parser and einops (sgl-project#4827)

* fix the release doc dependency issue (sgl-project#4828)

* Update doc for DeepSeek-V3-0324 (sgl-project#4825)

* deps: lazy import optional dependencies `gguf` and `torchvision` (sgl-project#4826)

* Update MMMU Benchmark instructions (sgl-project#4694)

* Fix the nightly eval by lowering the threshold of `neuralmagic/gemma-2-2b-it-FP8` (sgl-project#4830)

* Basic Cleanup (sgl-project#4833)

* Support (1 <= dp < tp) in the dp attention in DeepEP (sgl-project#4770)

Co-authored-by: Cheng Wan <[email protected]>

* [Fix] Add compressed_tensors as deps (sgl-project#4819)

* Fix error due to CustomAllreduce setup failure (sgl-project#4815)

Signed-off-by: Kebe <[email protected]>

* use default for torch.ops (sgl-project#4835)

* [CI] Remove unused imports with Ruff to pre-commit config, only to benchmarks/docs/examples folder (sgl-project#3969)

* [Misc] Fix issues reported by torchfix (sgl-project#4837)

* Include context length in /v1/models response. (sgl-project#4809)

* [Fix] `self.worker` assignment in `TpModelWorker` and refactor references (sgl-project#4788)

Signed-off-by: Xinyuan Tong <[email protected]>

* Fix the lora adapter when lora path is none (sgl-project#4799)

Co-authored-by: Beichen Ma <[email protected]>

* fix: fix typo of comments in w8a8_fp8.py (sgl-project#4843)

* Remove retry in nightly tests (sgl-project#4846)

* Fix CI of test_patch_torch (sgl-project#4844)

* IPv6 support (sgl-project#3949)

Signed-off-by: Brayden Zhong <[email protected]>

* ci: add condition for daily docker build (sgl-project#4487)

* [Fix] fix output_top_logprobs is not exist (sgl-project#4597)

* fix: when use SGLANG_PORT this env,port is str (sgl-project#4528)

Signed-off-by: rongfu.leng <[email protected]>

* Support Page Size > 1 for FA3 (sgl-project#4832)

Co-authored-by: Qingquan Song <[email protected]>
Co-authored-by: Baizhou Zhang <[email protected]>

* Fix Engine error when enabling DP attention (sgl-project#4648)

* fix: Inappropriate lack of Optional type on OpenAI ChatCompletionRequest (sgl-project#4681)

* Support controlling nsys start and end range programmatically (sgl-project#4688)

* Remove empty tool function name (sgl-project#4704)

Signed-off-by: Kebe <[email protected]>

* Fix missing arguments in SchedulePolicy and RadixCache initialization in tests. (sgl-project#4712)

* get the python version from env (sgl-project#4729)

* Fix torch.cuda.MemPool() internal assertion failure (sgl-project#4687)

Co-authored-by: Lianmin Zheng <[email protected]>

* Super tiny remove unused code (sgl-project#4750)

* Support with_stack and record_shapes in profiler (sgl-project#4740)

Co-authored-by: Lianmin Zheng <[email protected]>

* test: reduce `mem_fraction_static` for gemma3 vision test (sgl-project#4840)

* Fix CI tests (sgl-project#4853)

* Fix fa3 cuda graph page_size > 1 precision and page_size=1 speed (sgl-project#4855)

* Revert "get the python version from env (sgl-project#4729)" (sgl-project#4863)

* [Feature] add multi-rank support for Lora (sgl-project#4492)

Co-authored-by: rudy152 <[email protected]>

* Clean up `import vllm` in quantization/__init__.py (sgl-project#4834)

* Fix wrong variable name when stopping memory profile (sgl-project#4772)

* [Feat] support deepgemm for cmake (sgl-project#4864)

* Make torch compile configurable for biased_grouped_topk (sgl-project#4749)

* update sgl-kernel test ci (sgl-project#4866)

* fix sampling issue (sgl-project#4871)

* bump sgl-kernel 0.0.5.post4 (sgl-project#4768)

* fix sgl-kernel cu118 build (sgl-project#4872)

* [Feature] Support FA3 backend for MLA (sgl-project#4831)

* upgrade sgl-kernel 0.0.5.post4 (sgl-project#4873)

* update torch compile doc (sgl-project#4874)

* bump v0.4.4.post3 (sgl-project#4878)

* Fix BadRequestError wrong arguments and remove openai dependency (sgl-project#4882)

* Improve stack trace of retry errors (sgl-project#4845)

* Tiny fix doc error (sgl-project#4795)

* [Docs] Update DeepGEMM at README.md (sgl-project#4886)

* Update CODEOWNERS (sgl-project#4889)

* Delete test_deep_gemm.py (sgl-project#4891)

* Add deepseek style fused moe group gate selection kernel (sgl-project#4530)

* quick fix: add default for new kernel (sgl-project#4898)

* remove setup for sgl-kernel (sgl-project#4899)

* [Misc] Clean m.def and add Development Tips (sgl-project#4890)

* fix allreduce test (sgl-project#4909)

* Support page size > 1 + eagle (sgl-project#4908)

* Fix retract for page size > 1 (sgl-project#4914)

* [Feature] use pytest for sgl-kernel (sgl-project#4896)

* fix bmm fp8 (sgl-project#4926)

* Fix the timeout for unit-test-2-gpu in pr-test.yml (sgl-project#4927)

* Fix 2-gpu CI test and suppress some warnings (sgl-project#4930)

* [feat] add fa3 in sgl-kernel (sgl-project#4902)

Co-authored-by: Sleepcoo <[email protected]>

* Fix sglang frontend's incorrect dependency on torch (sgl-project#4931)

* [Fix] avoid stream sync and torch compile in prefill for fa3 backend (sgl-project#4932)

* cleanup sgl-kernel (sgl-project#4933)

* [Fix] Improve Lora tests and reduce CI runtime (sgl-project#4925)

* Fix DeepSeek bug causing 2.2% MMLU drop when TP!=DP (sgl-project#4883)

Co-authored-by: ch-wan <[email protected]>

* [Fix] Add torch compile for torch.clamp back (sgl-project#4936)

* Fix oom error for large page size (sgl-project#4913)

Co-authored-by: Lianmin Zheng <[email protected]>

* [feat] interface for platforms abstraction (sgl-project#4928)

* [Fix] revert clean m.def for cudagraph (sgl-project#4944)

* refactor: multimodal data (sgl-project#4754)

* bump sgl-kernel v0.0.6 (sgl-project#4950)

* [Build] Fix cuda12.8 build error in nvfp4_scaled_mm_kernels.cu (sgl-project#4953)

* use fa3 in sgl-kernel (sgl-project#4954)

* Revert PR 4764 & 4813 related to R1 RoPE (sgl-project#4959)

* [Feature] Support DeepEP Low Latency (sgl-project#4767)

Co-authored-by: sleepcoo <[email protected]>
Co-authored-by: laixinn <[email protected]>
Co-authored-by: ch-wan <[email protected]>

* update bench_serving (sgl-project#4958)

* Prevent memory leak of retract_decode when page_size > 1 (sgl-project#4977)

* [VLM RLHF] Take Image input for verl vlm rollout (sgl-project#4915)

Signed-off-by: Xinyuan Tong <[email protected]>
Co-authored-by: GeLee <[email protected]>

* Large page size aligned hierarchical caching (sgl-project#4581)

* bug fix for hicache host eviction (sgl-project#4989)

* sgl scaled_fp8_quant support output padding (sgl-project#4861)

* Add Eagle Speculative Decoding to FA3 Backend (sgl-project#4951)

Co-authored-by: hebiao064 <[email protected]>
Co-authored-by: Baizhou Zhang <[email protected]>
Co-authored-by: zcnrex <[email protected]>

* Update tokenizer_manager.py (sgl-project#5008)

* [sgl-kernel] per token group quant support COLUMN MAJOR (sgl-project#4817)

* update cutlass tag (sgl-project#5011)

* Feature/revise docs ci (sgl-project#5009)

* fix: fix illegal cuda memory access at fused_moe_kernel (sgl-project#4727)

Co-authored-by: yuethe <[email protected]>

* [Build] Support build sgl-kernel with ccache (sgl-project#5020)

* fix deepgemm as well (sgl-project#5030)

* try to fix ci oserror (sgl-project#5024)

* Replace enable_flashinfer_mla argument with attention_backend (sgl-project#5005)

* Small refactor DeepEPMode to clean up code a bit (sgl-project#4992)

* [Fix] fix fa3 build at cu118 (sgl-project#5036)

* Revert "Replace enable_flashinfer_mla argument with attention_backend" (sgl-project#5048)

* bump sgl-kernel v0.0.7 (sgl-project#5046)

* update eagle-3 docs (sgl-project#4796)

Co-authored-by: Yifan Zhang <[email protected]>

* Add LlavaLlamaForCausaLM in MultiModal Processors (sgl-project#5039)

Co-authored-by: Ravi Theja Desetty <[email protected]>

* Update the retry count (sgl-project#5051)

* upgrade sgl-kernel v0.0.7 (sgl-project#5049)

* [2/3] fix dsv3 awq issue  (sgl-project#4625)

Co-authored-by: 晟海 <[email protected]>
Co-authored-by: laixinn <[email protected]>

* Feature/revise docs ci (sgl-project#5056)

* Add H20 fused MoE kernel tuning configs for DeepSeek V3/R1 (sgl-project#5057)

* [fix] remove `cuda_device_count_stateless` (sgl-project#5060)

* Small refactor DeepEPDispatcher into subclasses (sgl-project#4994)

* Support async DeepEP by splitting into two stages (sgl-project#4995)

* Cleanup unused resources after DeepEP operation (sgl-project#4996)

* Add DeepSeek V3/R1 shared experts fusion (sgl-project#4918)

* [deepep] fix: shared experts are not initialized when shared experts fusion is enabled (sgl-project#5072)

* fix dummy-load deepseekv2 (sgl-project#4535)

* support sgl-kernel on blackwell (sgl-project#5074)

* FA3 Spec Decoding to support top k = 1 and add cuda graph support (sgl-project#5050)

Co-authored-by: Qingquan Song <[email protected]>
Co-authored-by: Chunan Zeng <[email protected]>

* [Revision] Replace enable_flashinfer_mla argument with attention_backend (sgl-project#5052)

* upgrade transformers 4.51.0 (sgl-project#5088)

* sgl-kernel transfer custom allreduce from trt kernel to vllm kernel (sgl-project#5079)

* bump sgl-kernel 0.0.8 (sgl-project#5089)

* python transfer custom allreduce from trt kernel to vllm kernel (sgl-project#5080)

* bump v0.4.4.post4 (sgl-project#5091)

* Fix: Reduce the number of document ci attempts to avoid long ci running (sgl-project#5097)

Co-authored-by: shuaills <[email protected]>

* Add Llama4 support (sgl-project#5092)

Co-authored-by: Cheng Wan <[email protected]>
Co-authored-by: fzyzcjy <[email protected]>
Co-authored-by: ispobock <[email protected]>

* Fix refactor error - fp8.py (sgl-project#5106)

Co-authored-by: Lianmin Zheng <[email protected]>

* bump v0.4.5 (sgl-project#5117)

* Workaround for async copy issue in HPU eager mode (sgl-project#1)

Signed-off-by: Rahul Vijayaraghavan <[email protected]>
Co-authored-by: Rahul Vijayaraghavan <[email protected]>

* [SW-223847]: Fix sgl_kernel module not available (sgl-project#2)

Co-authored-by: vikram singh shekhawat <[email protected]>

* [Base] Enable torch compile (sgl-project#4)

* [SW-226331] disable dynamic shape in torch compile mode

Signed-off-by: Mohit Sinha <[email protected]>

---------

Signed-off-by: Kebe <[email protected]>
Signed-off-by: Xinyuan Tong <[email protected]>
Signed-off-by: Brayden Zhong <[email protected]>
Signed-off-by: rongfu.leng <[email protected]>
Signed-off-by: Rahul Vijayaraghavan <[email protected]>
Signed-off-by: Mohit Sinha <[email protected]>
Co-authored-by: strgrb <[email protected]>
Co-authored-by: Zhang Kaihong <[email protected]>
Co-authored-by: AinL <[email protected]>
Co-authored-by: Jiří Suchomel <[email protected]>
Co-authored-by: Juwan Yoo <[email protected]>
Co-authored-by: Yineng Zhang <[email protected]>
Co-authored-by: Ke Bao <[email protected]>
Co-authored-by: Ravi Theja <[email protected]>
Co-authored-by: Lianmin Zheng <[email protected]>
Co-authored-by: Daniel Holanda <[email protected]>
Co-authored-by: tarinkk <[email protected]>
Co-authored-by: Cheng Wan <[email protected]>
Co-authored-by: Junrong Lin <[email protected]>
Co-authored-by: Kebe <[email protected]>
Co-authored-by: Brayden Zhong <[email protected]>
Co-authored-by: Jon Durbin <[email protected]>
Co-authored-by: XinyuanTong <[email protected]>
Co-authored-by: Qiaolin Yu <[email protected]>
Co-authored-by: Beichen Ma <[email protected]>
Co-authored-by: Jiaqi <[email protected]>
Co-authored-by: fzyzcjy <[email protected]>
Co-authored-by: Vincent <[email protected]>
Co-authored-by: warjiang <[email protected]>
Co-authored-by: lambert0312 <[email protected]>
Co-authored-by: rongfu.leng <[email protected]>
Co-authored-by: Stefan He <[email protected]>
Co-authored-by: Qingquan Song <[email protected]>
Co-authored-by: Baizhou Zhang <[email protected]>
Co-authored-by: BroadbentJim <[email protected]>
Co-authored-by: vikram singh shekhawat <[email protected]>
Co-authored-by: DavidChan <[email protected]>
Co-authored-by: chaobo jia <[email protected]>
Co-authored-by: rudy152 <[email protected]>
Co-authored-by: Fr4nk1in <[email protected]>
Co-authored-by: yinfan98 <[email protected]>
Co-authored-by: Yi Zhang <[email protected]>
Co-authored-by: Adarsh Shirawalmath <[email protected]>
Co-authored-by: Sleepcoo <[email protected]>
Co-authored-by: SEPLOS <[email protected]>
Co-authored-by: Zhiqiang Xie <[email protected]>
Co-authored-by: JieXin Liang <[email protected]>
Co-authored-by: Mick <[email protected]>
Co-authored-by: Yuhong Guo <[email protected]>
Co-authored-by: Jinyan Chen <[email protected]>
Co-authored-by: laixinn <[email protected]>
Co-authored-by: GeLee <[email protected]>
Co-authored-by: Xiaoyu Zhang <[email protected]>
Co-authored-by: zcnrex <[email protected]>
Co-authored-by: Kaiyu Yang <[email protected]>
Co-authored-by: renxin <[email protected]>
Co-authored-by: saltyfish66 <[email protected]>
Co-authored-by: yuethe <[email protected]>
Co-authored-by: simveit <[email protected]>
Co-authored-by: Yifan Zhang <[email protected]>
Co-authored-by: Ravi Theja Desetty <[email protected]>
Co-authored-by: AniZpZ <[email protected]>
Co-authored-by: 晟海 <[email protected]>
Co-authored-by: Tommy Yang <[email protected]>
Co-authored-by: Cheng Wan <[email protected]>
Co-authored-by: inkcherry <[email protected]>
Co-authored-by: mlmz <[email protected]>
Co-authored-by: shuaills <[email protected]>
Co-authored-by: Chang Su <[email protected]>
Co-authored-by: fzyzcjy <[email protected]>
Co-authored-by: HAI <[email protected]>
Co-authored-by: Rahul Vijayaraghavan <[email protected]>
Co-authored-by: Rahul Vijayaraghavan <[email protected]>
Co-authored-by: Jay Thakur <[email protected]>
Co-authored-by: Anshuman Tripathy <[email protected]>

* Support with_stack and record_shapes in profiler (sgl-project#4740)

Co-authored-by: Lianmin Zheng <[email protected]>

* test: reduce `mem_fraction_static` for gemma3 vision test (sgl-project#4840)

* Fix CI tests (sgl-project#4853)

* Fix fa3 cuda graph page_size > 1 precision and page_size=1 speed (sgl-project#4855)

* Revert "get the python version from env (sgl-project#4729)" (sgl-project#4863)

* [Feature] add multi-rank support for Lora (sgl-project#4492)

Co-authored-by: rudy152 <[email protected]>

* Clean up `import vllm` in quantization/__init__.py (sgl-project#4834)

* Fix wrong variable name when stopping memory profile (sgl-project#4772)

* [Feat] support deepgemm for cmake (sgl-project#4864)

* Make torch compile configurable for biased_grouped_topk (sgl-project#4749)

* update sgl-kernel test ci (sgl-project#4866)

* fix sampling issue (sgl-project#4871)

* bump sgl-kernel 0.0.5.post4 (sgl-project#4768)

* fix sgl-kernel cu118 build (sgl-project#4872)

* [Feature] Support FA3 backend for MLA (sgl-project#4831)

* upgrade sgl-kernel 0.0.5.post4 (sgl-project#4873)

* update torch compile doc (sgl-project#4874)

* bump v0.4.4.post3 (sgl-project#4878)

* Fix BadRequestError wrong arguments and remove openai dependency (sgl-project#4882)

* Improve stack trace of retry errors (sgl-project#4845)

* Tiny fix doc error (sgl-project#4795)

* [Docs] Update DeepGEMM at README.md (sgl-project#4886)

* Update CODEOWNERS (sgl-project#4889)

* Delete test_deep_gemm.py (sgl-project#4891)

* Add deepseek style fused moe group gate selection kernel (sgl-project#4530)

* quick fix: add default for new kernel (sgl-project#4898)

* remove setup for sgl-kernel (sgl-project#4899)

* [Misc] Clean m.def and add Development Tips (sgl-project#4890)

* fix allreduce test (sgl-project#4909)

* Support page size > 1 + eagle (sgl-project#4908)

* Fix retract for page size > 1 (sgl-project#4914)

* [Feature] use pytest for sgl-kernel (sgl-project#4896)

* fix bmm fp8 (sgl-project#4926)

* Fix the timeout for unit-test-2-gpu in pr-test.yml (sgl-project#4927)

* Fix 2-gpu CI test and suppress some warnings (sgl-project#4930)

* [feat] add fa3 in sgl-kernel (sgl-project#4902)

Co-authored-by: Sleepcoo <[email protected]>

* Fix sglang frontend's incorrect dependency on torch (sgl-project#4931)

* [Fix] avoid stream sync and torch compile in prefill for fa3 backend (sgl-project#4932)

* cleanup sgl-kernel (sgl-project#4933)

* [Fix] Improve Lora tests and reduce CI runtime (sgl-project#4925)

* Fix DeepSeek bug causing 2.2% MMLU drop when TP!=DP (sgl-project#4883)

Co-authored-by: ch-wan <[email protected]>

* [Fix] Add torch compile for torch.clamp back (sgl-project#4936)

* Fix oom error for large page size (sgl-project#4913)

Co-authored-by: Lianmin Zheng <[email protected]>

* [feat] interface for platforms abstraction (sgl-project#4928)

* [Fix] revert clean m.def for cudagraph (sgl-project#4944)

* refactor: multimodal data (sgl-project#4754)

* bump sgl-kernel v0.0.6 (sgl-project#4950)

* [Build] Fix cuda12.8 build error in nvfp4_scaled_mm_kernels.cu (sgl-project#4953)

* use fa3 in sgl-kernel (sgl-project#4954)

* Revert PR 4764 & 4813 related to R1 RoPE (sgl-project#4959)

* [Feature] Support DeepEP Low Latency (sgl-project#4767)

Co-authored-by: sleepcoo <[email protected]>
Co-authored-by: laixinn <[email protected]>
Co-authored-by: ch-wan <[email protected]>

* update bench_serving (sgl-project#4958)

* Prevent memory leak of retract_decode when page_size > 1 (sgl-project#4977)

* [VLM RLHF] Take Image input for verl vlm rollout (sgl-project#4915)

Signed-off-by: Xinyuan Tong <[email protected]>
Co-authored-by: GeLee <[email protected]>

* Large page size aligned hierarchical caching (sgl-project#4581)

* bug fix for hicache host eviction (sgl-project#4989)

* sgl scaled_fp8_quant support output padding (sgl-project#4861)

* Add Eagle Speculative Decoding to FA3 Backend (sgl-project#4951)

Co-authored-by: hebiao064 <[email protected]>
Co-authored-by: Baizhou Zhang <[email protected]>
Co-authored-by: zcnrex <[email protected]>

* Update tokenizer_manager.py (sgl-project#5008)

* [sgl-kernel] per token group quant support COLUMN MAJOR (sgl-project#4817)

* update cutlass tag (sgl-project#5011)

* Feature/revise docs ci (sgl-project#5009)

* fix: fix illegal cuda memory access at fused_moe_kernel (sgl-project#4727)

Co-authored-by: yuethe <[email protected]>

* [Build] Support build sgl-kernel with ccache (sgl-project#5020)

* fix deepgemm as well (sgl-project#5030)

* try to fix ci oserror (sgl-project#5024)

* Replace enable_flashinfer_mla argument with attention_backend (sgl-project#5005)

* Small refactor DeepEPMode to clean up code a bit (sgl-project#4992)

* [Fix] fix fa3 build at cu118 (sgl-project#5036)

* Revert "Replace enable_flashinfer_mla argument with attention_backend" (sgl-project#5048)

* bump sgl-kernel v0.0.7 (sgl-project#5046)

* update eagle-3 docs (sgl-project#4796)

Co-authored-by: Yifan Zhang <[email protected]>

* Add LlavaLlamaForCausaLM in MultiModal Processors (sgl-project#5039)

Co-authored-by: Ravi Theja Desetty <[email protected]>

* Update the retry count (sgl-project#5051)

* upgrade sgl-kernel v0.0.7 (sgl-project#5049)

* [2/3] fix dsv3 awq issue  (sgl-project#4625)

Co-authored-by: 晟海 <[email protected]>
Co-authored-by: laixinn <[email protected]>

* Feature/revise docs ci (sgl-project#5056)

* Add H20 fused MoE kernel tuning configs for DeepSeek V3/R1 (sgl-project#5057)

* [fix] remove `cuda_device_count_stateless` (sgl-project#5060)

* Small refactor DeepEPDispatcher into subclasses (sgl-project#4994)

* Support async DeepEP by splitting into two stages (sgl-project#4995)

* Cleanup unused resources after DeepEP operation (sgl-project#4996)

* Add DeepSeek V3/R1 shared experts fusion (sgl-project#4918)

* [deepep] fix: shared experts are not initialized when shared experts fusion is enabled (sgl-project#5072)

* fix dummy-load deepseekv2 (sgl-project#4535)

* support sgl-kernel on blackwell (sgl-project#5074)

* FA3 Spec Decoding to support top k = 1 and add cuda graph support (sgl-project#5050)

Co-authored-by: Qingquan Song <[email protected]>
Co-authored-by: Chunan Zeng <[email protected]>

* [Revision] Replace enable_flashinfer_mla argument with attention_backend (sgl-project#5052)

* upgrade transformers 4.51.0 (sgl-project#5088)

* sgl-kernel transfer custom allreduce from trt kernel to vllm kernel (sgl-project#5079)

* bump sgl-kernel 0.0.8 (sgl-project#5089)

* python transfer custom allreduce from trt kernel to vllm kernel (sgl-project#5080)

* bump v0.4.4.post4 (sgl-project#5091)

* Fix: Reduce the number of document ci attempts to avoid long ci running (sgl-project#5097)

Co-authored-by: shuaills <[email protected]>

* Add Llama4 support (sgl-project#5092)

Co-authored-by: Cheng Wan <[email protected]>
Co-authored-by: fzyzcjy <[email protected]>
Co-authored-by: ispobock <[email protected]>

* Fix refactor error - fp8.py (sgl-project#5106)

Co-authored-by: Lianmin Zheng <[email protected]>

* bump v0.4.5 (sgl-project#5117)

* [ci] fix llama4 ci error (sgl-project#5126)

* Refactor and Optimize FA3 Code (sgl-project#5090)

Co-authored-by: Qingquan Song <[email protected]>

* Add Llama4 user guide (sgl-project#5133)

Co-authored-by: Cheng Wan <[email protected]>

* [Misc] Use pytest.mark.skipif in sgl-kernel test (sgl-project#5137)

* feat: disable grammar restrictions within reasoning sections (sgl-project#4984)

Co-authored-by: tianhaoyu <[email protected]>
Co-authored-by: DarkSharpness <[email protected]>

* [modelopt] automatically inspect if model is ModelOpt quantized and set quantization method (sgl-project#5145)

* [AMD] Fix missing per_token_group_quant_fp8 for ROCm (sgl-project#5140)

* fix multimodal hash feature (sgl-project#5083)

* Fix run time error in ROCm platform (sgl-project#5147)

Co-authored-by: wunhuang <[email protected]>
Co-authored-by: root <[email protected]>

* [FA3 Feature] Support multi modal Llama-3.2-11B-Vision-Instruct (sgl-project#5103)

* Add unit test on page_size > 1 and mla and  integration test for Flash Attention 3 (sgl-project#4760)

* Use public model for FA3 speculative decode testing (sgl-project#5152)

* Add dummy grok test to amd CI. (sgl-project#5115)

* fix empty_cache error in pt_weights_iterator (sgl-project#5151)

Co-authored-by: dangkai.dk <[email protected]>

* Fix torch compile errors (sgl-project#5158)

* Fix loading KV quantization scale; Enable modelopt kv cache (sgl-project#4686)

Co-authored-by: qingquansong <[email protected]>

* [PD] Fix unclosed prefill connection warning of mini_lb (sgl-project#5155)

Signed-off-by: Shangming Cai <[email protected]>

* Add optimized native kernels in sgl-kernel (sgl-project#5150)

Co-authored-by: Chunyuan WU <[email protected]>
Co-authored-by: YanbingJiang <[email protected]>
Co-authored-by: blzheng <[email protected]>

* [PD] Simplify mini LB (sgl-project#4911)

Co-authored-by: Liangsheng Yin <[email protected]>

* Small improvement of native api docs (sgl-project#5139)

Co-authored-by: zhaochenyang20 <[email protected]>

* [feat&refactor] Enhance multimodal input support with refactor io_struct (sgl-project#4938)

Signed-off-by: Xinyuan Tong <[email protected]>

* Support 2x8xH100 for Llama 4 (sgl-project#5159)

* FP4 weight loading and inference (2/2) (sgl-project#3972)

* Fix multimodal hashing error (sgl-project#5174)

* Tiny disable model that does not work (sgl-project#5175)

* [Bugfix] Fix index out of bounds in local attention with large sequences (sgl-project#5173)

* [Fix] DeepEP Compatibility with Low Latency (sgl-project#5068)

Co-authored-by: ch-wan <[email protected]>

* docs: remove the use of Downward API for LWS_WORKER_INDEX (sgl-project#5110)

Signed-off-by: Kay Yan <[email protected]>

* feat: add DeepGEMM build warning (sgl-project#5176)

Co-authored-by: grimoire <[email protected]>

* fix: use DeepEPDispatcher on CUDA (sgl-project#5180)

* [DeepEP] fix: import buffer error (sgl-project#5179)

* Let `bench_one_batch` support `enable_dp_attention` (sgl-project#4058)

* [Misc] clean up vllm in sgl-kernel test (sgl-project#5189)

* Fix ci test "test_eval_fp8_accuracy" failed (sgl-project#5185)

Co-authored-by: wunhuang <[email protected]>

* Optimize topk operation in llama4 (sgl-project#5128)

* Support Llama4 fp8 inference (sgl-project#5194)

Co-authored-by: laixinn <[email protected]>
Co-authored-by: sleepcoo <[email protected]>
Co-authored-by: zhyncs <[email protected]>

* [ci] fix ci test fused_moe op (sgl-project#5102)

* model: support mllama4 (sgl-project#5144)

* update grok test (sgl-project#5171)

* sgl-kernel use cutlass latest version for fp8 blockwise gemm (sgl-project#5207)

* Add H20 dtype fp8_w8a8 fused MoE kernel tuning configs for DeepSeek V3/R1 (sgl-project#5196)

* fix: log warning when disable cuda graph (sgl-project#5209)

* [metrics] Add in queue metrics (sgl-project#4444)

* Fix DeepSeek error when using DeepEP mode (sgl-project#5190)

* reduce moe_align_block_size_kernel small batch mode overhead (sgl-project#5086)

* [PD] Support KV transfer with mooncake (sgl-project#4880)

Signed-off-by: Shangming Cai <[email protected]>
Co-authored-by: Shangming Cai <[email protected]>
Co-authored-by: Xuchun Shang <[email protected]>
Co-authored-by: shangmingc <[email protected]>

* [PD] Add get_contiguous_buf_infos interface for MLATokenToKVPool (sgl-project#5204)

* Update deps for mllama4 (sgl-project#5215)

* Fix deepseek-v3 with torch.compile in PyTorch 2.6. (sgl-project#5213)

* ROCm sgl-kernel: compatible to later torch (sgl-project#5167)

* [Misc] Clean sgl-kernel test  (sgl-project#5216)

* Update Makefile / build script to avoid installing incompatible torch dependency (sgl-project#5245)

* Fix torch.compile cacheing (sgl-project#5259)

Co-authored-by: zhyncs <[email protected]>

* ROCm/AITER CK_MoE: update 2-stage kernels & support both Activations (sgl-project#5228)

* Optimize attention in llama4 (sgl-project#5127)

* Optimize GPU memory usage in FlashAttentionBackend's strided indexing (sgl-project#5262)

Co-authored-by: ch-wan <[email protected]>

* Support `--enable-llama4-multimodal` (sgl-project#5254)

* [fix] fix mrope positions not picked up (sgl-project#5265)

* doc: nested loop code for offline engine (sgl-project#5244)

* fix: examples for token_in_token_out_vlm  (sgl-project#5193)

* Fix a 404 link in send_request.ipynb (sgl-project#5280)

Signed-off-by: windsonsea <[email protected]>

* fix: enable fp4 compilation on cu128 (sgl-project#5286)

* feat: add cu128 identifier for sgl-kernel (sgl-project#5287)

* chore: relax the torch version restriction for sgl-kernel compilation (sgl-project#5288)

* chore: bump sgl-kernel v0.0.8.post1 (sgl-project#5289)

* [PD] fix: skip warmup request in disaggregation mode to prevent crash on timeout (sgl-project#5292)

* [Docs] Supported Model Docs - Major restructuring (sgl-project#5290)

Co-authored-by: zhaochenyang20 <[email protected]>

* fix: update update_wheel_index for cu128 (sgl-project#5300)

* [Docs] Remove the older supported docs section (sgl-project#5301)

* remove moe_align_block_size torch.zeros in small batch/expert mode (sgl-project#5298)

* feat: add blackwell Dockerfile (sgl-project#5302)

* feat: add blackwell workflow (sgl-project#5303)

* fix: use fa3 unit test on hopper only (sgl-project#5304)

* misc: update blackwell Dockerfile (sgl-project#5306)

* fix: remove cublas_grouped_gemm (sgl-project#5307)

* fix: update flash attn (sgl-project#5308)

* fix: use deepgemm only on hopper (sgl-project#5310)

* [VLM] Adopt fast image processor by default (sgl-project#5065)

* Adjust ci test threshold (sgl-project#5271)

* Blackwell Cutlass MLA kernel (sgl-project#5142)

* misc: cleanup 3rdparty (sgl-project#5311)

* update variable naming and comments for rocm (sgl-project#5299)

* Fix w8a8_int8 model shared experts fusion load weights error (sgl-project#5120)

* Add flash_attn_varlen_func to sgl-kernel (sgl-project#5315)

* Fix fa3 window size setup (sgl-project#5316)

* chore: bump sgl-kernel v0.0.8.post2 (sgl-project#5317)

* feat: use fa3 mla by default on hopper (sgl-project#5210)

Co-authored-by: yundai424 <[email protected]>
Co-authored-by: hebiao064 <[email protected]>

* Fix: docs/backend/structured_outputs.ipynb (sgl-project#4884)

* Delete python/sglang/srt/layers/moe/fused_moe_triton/configs/E=257,N=… (sgl-project#5321)

* refine fused_moe tuning docs (sgl-project#5294)

* Support server based rollout in Verlengine (sgl-project#4848)

Co-authored-by: Jin Pan <[email protected]>
Co-authored-by: Chayenne <[email protected]>
Co-authored-by: Jinn <[email protected]>

* [Feat] Add sparse attn to sgl-kernel (sgl-project#5327)

* fix: solve cu118 issue for cutlass mla (sgl-project#5331)

* chore: bump sgl-kernel v0.0.8.post3 (sgl-project#5332)

* ci: update release node (sgl-project#5333)

* fix: determine if flashinfer is installed (sgl-project#5336)

* feat: adapt merge_state (sgl-project#5337)

* misc: update sagemaker Dockerfile (sgl-project#5341)

* Fix: Ensure tensors for dist.broadcast match NCCL backend device (sgl-project#5322)

* docs: update adoption and sponsorship list with Oracle (sgl-project#5343)

* chore: upgrade sgl-kernel 0.0.8.post3 (sgl-project#5342)

* Fix typo: infight -> inflight (sgl-project#5357)

* [PD] Add transfer backend abstraction (sgl-project#5328)

* fix MLATokenToKVPoolHost get_size_per_token bug (sgl-project#5161)

Co-authored-by: AniZpZ <[email protected]>

* fix sgl-project#5322 (sgl-project#5359)

* feat: update experiment_runner (sgl-project#5360)

* [DeepEP] Reduce routed scaling overhead (sgl-project#5277)

Co-authored-by: Cheng Wan <[email protected]>

* Free metadata_buffer_index after transfer finished (sgl-project#5364)

* Free metadata_buffer_index after transfer finished (sgl-project#5364)

* Fix DeepSeek DP Attention + torch compile (sgl-project#5367)

Co-authored-by: ispobock <[email protected]>

* Support for Qwen2.5-VL Model in bitsandbytes Format (sgl-project#5003)

* Fix PD disaggregation bugs (sgl-project#5326)

* [PD Bug] fix  MLA get_contiguous_buf_infos error (sgl-project#5384)

* [perf] experimental enhance fp8 per-tensor quant (sgl-project#5370)

* Apply deepseek cuda rope (sgl-project#5385)

Co-authored-by: Yineng Zhang <[email protected]>

* apply fused moe gate in ds v3/r1 (sgl-project#5371)

Co-authored-by: Yineng Zhang <[email protected]>

* fix: update test config (sgl-project#5392)

* [Fix] Turn off DeepGEMM by default (sgl-project#5263)

* minor clean up of sgl-kernel/CMakeLists.txt (sgl-project#5393)

* Add A800 shared experts fused MoE kernel tuning configs for DeepSeek V3/R1 (sgl-project#5368)

* Add H20 dtype fp8_w8a8 shared experts fused MoE kernel tuning configs for DeepSeek V3/R1 (sgl-project#5291)

Co-authored-by: ximing.wxm <[email protected]>

* [fix/misc] remove duplicate row in deepseek v2 model (sgl-project#5279)

* chore: upgrade DeepGEMM (sgl-project#5395)

* fix: update pr-test-sgl-kernel (sgl-project#5399)

* kernel: support slightly faster merge_state_v2 cuda kernel (sgl-project#5381)

* chore: bump sgl-kernel 0.0.9 (sgl-project#5400)

* chore: upgrade sgl-kernel 0.0.9 (sgl-project#5401)

* Tiny fix DeepseekScalingRotaryEmbedding always use forward_native (sgl-project#5406)

* Fix bench_serving with random-ids (sgl-project#5214)

* [misc] fix ci flaky case (sgl-project#5352)

* [FIX] Fix concatenation error in capture_bs when open --disable-cuda-graph-padding and without MTP (sgl-project#5412)

* Support dynamic connection and TP 16 (sgl-project#5351)

Co-authored-by: luoyuan.luo <[email protected]>

* Fix broadcast use cuda device lead to memory capacity unbalanced (sgl-project#5416)

* [PD] Fix dynamic port support and MLA buffer for Mooncake (sgl-project#5415)

Signed-off-by: Shangming Cai <[email protected]>
Co-authored-by: ybyang <[email protected]>

* Distinguish bootstrap key only in decode server (sgl-project#5422)

* [PD] Remove unused bootstrap param and fix port table type (sgl-project#5423)

* [minor] cleanup cmakelists.txt (sgl-project#5420)

* bugfix: fix merge_state_v2 cuda graph (sgl-project#5419)

* chore: bump sgl-kernel v0.0.9.post1 (sgl-project#5430)

* fix: solve release issue (sgl-project#5434)

* BLackwell cutlass mla: Add check for bad page size/block num combinations (sgl-project#5431)

* feat: update model_specific_adjustment (sgl-project#5344)

Co-authored-by: hebiao064 <[email protected]>

* chore: upgrade sgl-kernel 0.0.9.post1 (sgl-project#5436)

* Fix ignore_eos parameter when loading a chat template (sgl-project#5264)

* add attention backend supporting matrix in the doc (sgl-project#5211)

Co-authored-by: Stefan He <[email protected]>

* Support BNB quantization for llama/mllama (sgl-project#5038)

Co-authored-by: Yuhao Yang <[email protected]>

* [Docs] Update start/install.md (sgl-project#5398)

* [Minor] Move torch.compile patch to a better place (sgl-project#5397)

* [Bug fix] need record start time in pd mode (sgl-project#5425)

* Support MHA with chunked prefix cache for DeepSeek chunked prefill (sgl-project#5113)

* chore: bump v0.4.5.post1 (sgl-project#5445)

* Revert "[SW-226289] rebase sglang to tag v0.4.5 (sgl-project#12)"

This reverts commit 0eac714.

---------

Signed-off-by: Xinyuan Tong <[email protected]>
Signed-off-by: Shangming Cai <[email protected]>
Signed-off-by: Kay Yan <[email protected]>
Signed-off-by: windsonsea <[email protected]>
Co-authored-by: fzyzcjy <[email protected]>
Co-authored-by: Lianmin Zheng <[email protected]>
Co-authored-by: Juwan Yoo <[email protected]>
Co-authored-by: Qingquan Song <[email protected]>
Co-authored-by: Yineng Zhang <[email protected]>
Co-authored-by: chaobo jia <[email protected]>
Co-authored-by: rudy152 <[email protected]>
Co-authored-by: Fr4nk1in <[email protected]>
Co-authored-by: yinfan98 <[email protected]>
Co-authored-by: Baizhou Zhang <[email protected]>
Co-authored-by: Ke Bao <[email protected]>
Co-authored-by: Yi Zhang <[email protected]>
Co-authored-by: Adarsh Shirawalmath <[email protected]>
Co-authored-by: Sleepcoo <[email protected]>
Co-authored-by: SEPLOS <[email protected]>
Co-authored-by: ch-wan <[email protected]>
Co-authored-by: Zhiqiang Xie <[email protected]>
Co-authored-by: JieXin Liang <[email protected]>
Co-authored-by: Mick <[email protected]>
Co-authored-by: Yuhong Guo <[email protected]>
Co-authored-by: Jinyan Chen <[email protected]>
Co-authored-by: laixinn <[email protected]>
Co-authored-by: XinyuanTong <[email protected]>
Co-authored-by: GeLee <[email protected]>
Co-authored-by: Xiaoyu Zhang <[email protected]>
Co-authored-by: hebiao064 <[email protected]>
Co-authored-by: zcnrex <[email protected]>
Co-authored-by: Kaiyu Yang <[email protected]>
Co-authored-by: renxin <[email protected]>
Co-authored-by: saltyfish66 <[email protected]>
Co-authored-by: yuethe <[email protected]>
Co-authored-by: simveit <[email protected]>
Co-authored-by: Yifan Zhang <[email protected]>
Co-authored-by: Ravi Theja <[email protected]>
Co-authored-by: Ravi Theja Desetty <[email protected]>
Co-authored-by: AniZpZ <[email protected]>
Co-authored-by: 晟海 <[email protected]>
Co-authored-by: Tommy Yang <[email protected]>
Co-authored-by: Cheng Wan <[email protected]>
Co-authored-by: inkcherry <[email protected]>
Co-authored-by: mlmz <[email protected]>
Co-authored-by: shuaills <[email protected]>
Co-authored-by: Chang Su <[email protected]>
Co-authored-by: fzyzcjy <[email protected]>
Co-authored-by: HAI <[email protected]>
Co-authored-by: tianhaoyu <[email protected]>
Co-authored-by: DarkSharpness <[email protected]>
Co-authored-by: Yun Dai <[email protected]>
Co-authored-by: Hubert Lu <[email protected]>
Co-authored-by: huangtingwei <[email protected]>
Co-authored-by: kk <[email protected]>
Co-authored-by: wunhuang <[email protected]>
Co-authored-by: root <[email protected]>
Co-authored-by: Yubo Wang <[email protected]>
Co-authored-by: saienduri <[email protected]>
Co-authored-by: DangKai <[email protected]>
Co-authored-by: dangkai.dk <[email protected]>
Co-authored-by: shangmingc <[email protected]>
Co-authored-by: Ma Mingfei <[email protected]>
Co-authored-by: Chunyuan WU <[email protected]>
Co-authored-by: YanbingJiang <[email protected]>
Co-authored-by: blzheng <[email protected]>
Co-authored-by: Byron Hsu <[email protected]>
Co-authored-by: Liangsheng Yin <[email protected]>
Co-authored-by: zhaochenyang20 <[email protected]>
Co-authored-by: Trevor Morris <[email protected]>
Co-authored-by: Kay Yan <[email protected]>
Co-authored-by: grimoire <[email protected]>
Co-authored-by: HandH1998 <[email protected]>
Co-authored-by: Zhaoyang Hao <[email protected]>
Co-authored-by: Teng Ma <[email protected]>
Co-authored-by: Shangming Cai <[email protected]>
Co-authored-by: Xuchun Shang <[email protected]>
Co-authored-by: Richard Zou <[email protected]>
Co-authored-by: Elfie Guo <[email protected]>
Co-authored-by: Michael Yao <[email protected]>
Co-authored-by: Yusong Gao <[email protected]>
Co-authored-by: Zhaoyi Li <[email protected]>
Co-authored-by: lambert0312 <[email protected]>
Co-authored-by: tianlian yi <[email protected]>
Co-authored-by: Jin Pan <[email protected]>
Co-authored-by: Jinn <[email protected]>
Co-authored-by: yulei <[email protected]>
Co-authored-by: Yongtong Wu <[email protected]>
Co-authored-by: yhyang201 <[email protected]>
Co-authored-by: ybyang <[email protected]>
Co-authored-by: Ximingwang-09 <[email protected]>
Co-authored-by: ximing.wxm <[email protected]>
Co-authored-by: Yangcheng Li <[email protected]>
Co-authored-by: DefTruth <[email protected]>
Co-authored-by: Yuan Luo <[email protected]>
Co-authored-by: luoyuan.luo <[email protected]>
Co-authored-by: ybyang <[email protected]>
Co-authored-by: mRSun15 <[email protected]>
Co-authored-by: ryang <[email protected]>
Co-authored-by: Yuhao Yang <[email protected]>

fix loss problem

Signed-off-by: Ivan Butygin <[email protected]>

Signed-off-by: Ivan Butygin <[email protected]>

Signed-off-by: Ivan Butygin <[email protected]>

…anup

TP multi-GPU (Issue sgl-project#12):
- Phase 2: all_reduce MIN for D2D/H2D path agreement
- Phase 4: all_reduce MIN + barrier for graph capture sync
- Phase 3: get_num_kv_heads(tp_size) fix
- register_model: TP rank>0 socket None check

Piecewise CUDA graph (Issue R):
- Save/restore piecewise_cuda_graph_runner + attention/moe layers in graph cache
- Set _pcg_capture_stream for lazy recapture after torch.compile guard failure

Graph-KV consistency:
- Skip graph restore when KV pool rebuilt (new buffer addresses)
- Evict stale graph cache + release VMM physical pages
- init_piecewise_cuda_graphs only on graph cache MISS

Alignment consistency:
- _migrate_params_to_bump: include 256B per-tensor alignment in total_bytes
- FALLBACK: match alignment + filter persistent buffers only
- layer_map: store aligned nbytes in LayerSlice
- layer_map: clear stale entries before re-migration

Other fixes:
- io_struct: remove redundant model field (Issue I)
- tokenizer_manager: reject unregistered models (Issue N)
- weight_staging: portable _get_batch_h2d paths
- weight_staging: fix overlap log to use safe block physical address

Verified: TP=2 small 10/10, TP=1 big 10/10, TP=1 small 10/10

…s 28 xfers

v9 pool-binding-shift trace produces real differentiation:
- Phase B (KV-bound 8K random): L1+L2 -37% mean TTFT vs stock
- Phase C (mixed 4K random):     L1+L2 -38% median E2E vs stock
- Cross-pool transfers: stock 0, L1-only 0, L2-only 0, L1+L2 28

Two surprising findings documented:
1. Layer 2 alone fires zero transfers — Layer 1 retention is what
   makes Layer 2 cross the firing threshold.
2. Phase A regresses with L1 (-20% TPS) because K_big=8192 hurts on
   prefix-friendly GSP. Consistent with A2's K_big=0-wins finding.
   Adaptive K_big control marked as follow-up.

Settings status: Setting 1 marked **DONE v6 NULL + v9 PASS**.
All 4 user-requested follow-ups (sgl-project#9 Q3.A 4-arm, sgl-project#10 Sweep 1
multi-seed, sgl-project#11 Setting 4 fallback rule, sgl-project#12 Setting 1 v9 trace)
now complete.

Direct port of V4 reference Transformer (inference/model.py lines
769-809) with Eagle3 aux-hidden-state capture grafted on per the
qwen2.py:652 post-loop trap pattern documented in
wiki/pipeline.md known-pitfalls.

__init__:
- Embed, layers list (43 DeepseekV4DecoderLayer), final_norm RMSNorm
  parameters allocated.
- Top-level mHC head reduction parameters (hc_head_fn, hc_head_base,
  hc_head_scale per V4 ref lines 794-799). These fold the [b, s,
  hc_mult, d] hidden state into [b, s, d] before lm_head.
- LM head (nn.Linear hidden_size -> vocab_size, no bias).
- MTP block list. Per CLAUDE.md rule sgl-project#12 + architecture-notes.md
  "Eagle3 vs native MTP", MTP is loaded for weight-key compat but NOT
  executed at Eagle3 inference. Allocated as ModuleList of nn.Identity
  placeholders. The Eagle3 draft head replaces MTP.

forward:
- Embed -> hc_mult-copy expand -> layer loop -> post-loop trap ->
  hc-head fold -> final norm -> lm_head[..., -1] (last-token logits
  matching V4 reference get_logits).
- Aux capture grafted IN the loop: when layer index i is in
  self._eagle3_layers_to_capture, append h.mean(dim=-2) (fold over the
  hc_mult copies to get [B, T, d]) BEFORE the layer runs. This captures
  the layer's INPUT, matching the V3.2/qwen2.py convention.
- Post-loop trap: when end_layer (= num_hidden_layers) is in the
  configured capture set, capture the final hidden state. This handles
  the qwen2.py:652 pattern where set_eagle3_layers_to_capture maps
  layer_ids [0, 14, 27] -> [1, 15, 28] (+1 offset). The last layer's
  mapped index (=num_hidden_layers) is unreachable inside the for-loop;
  the post-loop trap captures it before normalization, matching the
  pre-norm convention used for the other aux states.
- Aux feature dim per captured layer: hidden_size (mean over hc_mult
  copies). For V4-Flash slot triple [1, 21, 41]: 3 layers x 4096 dim =
  12288 aux feature dim, matching SpecForge draft expectation.

load_weights: still raises NotImplementedError, with a detailed TODO
spelling out the 5 sub-tasks (key map, FP4 expert dequant, FP8
non-expert handling, tid2eid table, MTP weights). Pattern follows
deepseek_common/deepseek_weight_loader.py with FP4 additions.

What this enables (when load_weights lands):
- Random-init forward pass for shape testing (works today, modulo
  sparse_attn_v4 stub raising)
- Real-checkpoint forward pass for numerical agreement testing vs HF
  reference
- Eagle3 training data generation via sglang as the target backend
  (per CLAUDE.md rule sgl-project#1: --target-model-backend sglang)

Open TODOs (still on Phase 1 in flight):
- TODO(phase1-loader): load_weights body
- TODO(phase1-kernel): sparse_attn_v4 wiring to NSA tilelang
- TODO(phase1-nsa): plug NSAIndexer into V4Attention.indexer (CSA path)
- TODO(phase1-tp): swap nn.Linear -> ColumnParallelLinear / RowParallelLinear
- TODO(phase1-fp4): V4-aware FP4 Linear that keeps experts packed
- TODO(phase1-quant): act_quant calls for FP8 simulation in V4Attention

Phase 1 file at this commit: ~1300 lines. End-to-end forward path
plumbed; runs out of stubs only at sparse_attn_v4 and load_weights.

Replaces the brief 5-step TODO with a fully-detailed weight-key map
extracted 2026-04-30 from /tmp/v4-flash-meta/model.safetensors.index.json
(5.4 MB, 69187 keys, 82 distinct patterns).

The map lists every distinct V4 checkpoint key pattern and its target
parameter on our model class, with three classes of annotation:

  *   = checkpoint uses RMSNorm submodule (.norm.weight); our code uses
        Parameter (norm_weight). Two paths to fix: refactor to RMSNorm
        submodule (preferred — kernel reuse) or write loader-side remap.

  **  = checkpoint stores quantized weight + separate scale tensor (FP8
        e4m3fn + e8m0fnu for non-experts; FP4 e2m1fn_x2 + e8m0fnu for
        experts). Current code uses nn.Linear with no .scale. Two paths:
        refactor to FP8/FP4-aware Linear (preferred — perf) or loader
        dequantizes weight*scale -> BF16 on load.

  *** = mHC params stored flat at layer level (layers.0.hc_attn_fn) but
        our code wraps in V4HCBlock submodule (layers[0].hc.hc_attn_fn).
        Loader needs to remap.

Includes:
- Complete top-level key list (embed, head, hc_head_*)
- Complete per-main-layer key list (43 layers, varying by attention type)
- Complete MTP-layer key list (loaded for HF compat, not run per rule sgl-project#12)
- 5-phase implementation outline for the GPU resume session:
    1. Top-level renames
    2. Per-layer remap (RMSNorm, mHC submodule)
    3. MTP split
    4. FP8/FP4 dequant (if not using FP-aware Linear)
    5. load_state_dict with strict=False + missing/unexpected logging
- Key-count statistics (66048 of 69187 keys are expert weight + scale
  tensors — the lion's share of the loader work is the FP4 dequant step)

Body still raises NotImplementedError. Implementation is GPU-VM work
because (a) FP4/FP8 dequant needs torch CUDA support for the
e4m3fn / e2m1fn_x2 / e8m0fnu dtypes, and (b) testing requires loading
the actual 159GB checkpoint which is impractical on a laptop.

The docstring is now actionable: a senior IC with sglang + V4 checkpoint
on a GPU VM can implement load_weights in <1 day from this doc alone.

Replaces the bare load_weights raise with a 3-phase skeleton:
  Phase 1: _remap_v4_checkpoint_keys (DONE, last commit 4ae9da8)
  Phase 2: _split_v4_mtp_keys (DONE, this commit)
  Phase 3: FP8/FP4 dequant (TODO(phase1-fp4); GPU-only)
  Phase 4: self.load_state_dict(strict=False) + missing/unexpected log
           (commented out, ready to uncomment after Phase 3)

_split_v4_mtp_keys(state_dict) -> (main_dict, mtp_dict):
  Per CLAUDE.md rule sgl-project#12, MTP block weights are loaded for HF
  checkpoint compat but NOT executed at Eagle3 inference. Splitting
  the state_dict here lets the load_state_dict call's
  unexpected-key log stay meaningful (only flag actually-unexpected
  keys, not the 120-ish intentionally-discarded MTP keys per layer).

  Pure key-prefix filter; no tensor manipulation. ~10 lines.

skeleton load_weights:
  Concrete prelude that calls _remap_v4_checkpoint_keys +
  _split_v4_mtp_keys, then raises with a clear "n input keys ->
  m main + k MTP keys; FP8/FP4 dequant is GPU-required" message
  including key counts. The GPU resume agent can verify the prelude
  works on a real V4 checkpoint by catching the NotImplementedError
  and inspecting state_dict shape before adding the dequant step.

  The finalization block (load_state_dict + log) is included as a
  commented-out template — the GPU agent uncomments it after the
  dequant step lands.

test_split_mtp_keys added: validates partition correctness with a
synthetic state_dict containing a mix of main + mtp keys. Verifies:
  - len(main) + len(mtp) == len(sd_in)
  - main and mtp are disjoint
  - all mtp keys start with "mtp."
  - no main keys start with "mtp."

Test counts: 19 total tests, 4 still skipped (3 GPU-required forward
tests + 1 load_weights-body test). 15 active tests on laptop.

What this enables for the GPU resume agent:
- The HF checkpoint -> our model parameter rename pipeline is now
  ~70% laptop-deliverable.
- They can boot a GPU VM, run pytest test/manual/models/
  test_deepseek_v4_models.py to verify all 15 active tests pass,
  then start adding the FP8/FP4 dequant pass with high confidence
  that everything else is correct.
- Time-to-Phase-4-smoke goes from "1 day for full load_weights" to
  "half a day for dequant + load_state_dict".

Net Phase 1 sglang fork progress as of this commit: 14 commits on
eagle3/deepseek-v4 branch. 2078 lines in deepseek_v4.py + 553 lines
in test_deepseek_v4_models.py.

Phase H.1 acceptance: live SGLang server on 2x p150a returns "Paris"
for the capital-of-France smoke. 3 tests, all gated by SGLANG_PLATFORM=
tenstorrent (skipped on CPU-only CI):

  test_paris_via_chat_format        — chat-format prompt → "Paris."
  test_paris_via_qa_completion      — Q/A completion → " Paris..."
  test_spec_verbatim_prompt_*       — spec's literal prompt → coherent

Live result (server running with --device cpu --context-length 16384):
  3 passed in 0.58s

Spec §8.1 deviation note (folded into test file docstring): the spec's
verbatim "The capital of France is" prompt yields a descriptive
continuation ("a city that is steeped in history and culture") from
Llama-3.1-8B-Instruct, NOT the literal word "Paris". Instruct tuning
prefers continuation over factual recall in completion-style prompts.
The chat-format + Q/A variants exercise the same backend code path
(EXTEND -> DECODE loop, greedy on host) and reliably elicit "Paris".

Spec §8.1 acceptance criteria as written:
- HTTP 200 ✓
- response includes "Paris" ✓ (via chat or Q/A format)
- E2E ≤ 5 min ✓ (< 2s in practice)
- server alive after ✓

The third test asserts only that the spec-verbatim prompt produces
NON-EMPTY ENGLISH text — catches FutureMap -1-sentinel garbage
(spec §10 risk sgl-project#12) without requiring a prompt format the spec didn't
anticipate.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

…ecks (#12)

#9 — `_SWA_AWARE_DECODE_BACKENDS` was a single frozenset that silently
defaulted to the extend-window mask rule for any unclassified backend.
Adding a new SWA-aware decode backend without registering it would make
the reference quietly use the wrong rule. Replace with two explicit sets
(`_SWA_DECODE_MIN_SEQ_LEN_WINDOW`, `_SWA_DECODE_EXTEND_WINDOW`) and a
`_swa_decode_uses_min_seq_len_rule(case)` helper that raises if the
backend is in neither — future backends fail loudly until classified.

#12 — Hybrid dispatch fan-out tests (M19/M20) used
`assert_called_once_with(*positional_args)` which would silently break if
production refactored `HybridLinearAttnBackend.init_forward_metadata*` to
forward arguments by keyword instead of position. Add a shared
`_assert_fanout_forwarded` helper that calls `assert_called_once()` for
the count check and verifies each expected sentinel object identity is
present in `call_args.args + call_args.kwargs.values()`. The mutation
detection is unchanged (slice mutations still result in 0 calls); the
test is now robust to positional↔keyword refactors.

Full sweep: 97 tests, 3 skipped, 0 failures.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

TEMP (sgl-project#12 clean capture). On the FIRST matching large-T C4 prefill (cold:
JIT+autotune), run the production entrypoint twice and log per-head
cold-vs-warm cosine + save /tmp/flydsl_capture_c4_cold.pt, to prove whether
the prior [16:32]u[48:64] 32-head anomaly is a cold-start artifact. Warm
capture (after SKIP) keeps the recompute/verify-before-save self-consistency
check. Adds offline 3-way per-head-min-cosine compare harness. Strict no-op
unless SGLANG_FLYDSL_CAPTURE=1; _USE_FLYDSL_KERNEL stays False.

…/forensic

Round-15 audit found 16 items. All applied. Most consequential is
replacing the round-14-part-2 'assert feasible' with a fatal()
forensic-dump halt + new §10 two-fault-class invariant.

MAJOR fixes:
- #1 joint_decide infeasibility: was assert (round-14 part 2). Now
  fatal('joint_decide_infeasible', ...) writes structured JSON to
  <daemon-data>/forensic/joint_decide_infeasible_<ts>.json
  capturing event, state snapshot, candidates, DP inputs, traceback,
  then sys.exit(1). User mandate: deployment-bug-class faults
  expose themselves with full forensic context; load-class faults
  absorb. The fault classification is now an explicit §10 invariant.
- #2 unavailability_cost signature fix: was transfer_time(add_tiers,
  u, state) — wrong signature (transfer_time is per σ→τ pair).
  Rewrote as sum over add_tiers of p_hat × P(serve fails) × reload.
  Caller passes state correctly.
- sgl-project#3 bw_free cold-start: pinned time_since_last_sample_s = +Inf at
  boot (trips the > LINK_IDLE_SECONDS branch → returns peak_bw_bps,
  the correct value for an unused link).
- sgl-project#4 multi-rank throughput aggregation: added per-field aggregation
  rule table to §6 — sum for byte fields, sum for link bandwidth
  (each rank has own slice), MEAN for throughput EMAs (same logical
  work observed N×), max for time_since_last_sample_s, identical
  for static config + n_bytes + subpool keys.
- sgl-project#5 cross-rank subpool key consistency: explicit fatal() on
  disagreement.

CLARITY:
- sgl-project#6 PLAN.md drift: removed 'PUT updates cache file atomically' (cache
  was dropped in round-14), removed 'feasibility-fallback' (now
  fatal), 'samples_in_window' → 'time_since_last_sample_s'.
- sgl-project#7 forecast_horizon: explicit branch for recent_event_rate=0 cold-
  start (returns heartbeat_s).
- sgl-project#8 mean_unit_bytes empty-HBM guard: short-circuit returns ∞ to
  collapse top_k_pressure to K_MIN.
- sgl-project#9 peak_bw_bps > 0 / h_max > 0 / prefill_bps > 0 as deployment-bug
  invariant (§10 Physical inputs); fatal() on violation.
- sgl-project#10 APPLY_FAILED unknown batch_id: explicit silent drop (load-class,
  not deployment-bug); next state-fetch reconverges via always-fresh
  + idempotent invariants.
- sgl-project#11 program state intermediates: explicit '4 states are exhaustive,
  intermediate states collapsed at dump time' in §5 schema comment.
- sgl-project#12 'policy is not learned': softened 'in scope as an extension' →
  'neither in scope nor ruled out'; consistent with paper which
  also dropped the PPO/DQN paragraph.
- sgl-project#14 §12 S4 axis count: 7 → 9 (3 HBM + 3 DRAM-cap + 3 DISK-cap
  axes; the subtractive arithmetic in the original was wrong).
- sgl-project#15 §12 scenario composition: explicit note that S1-S5 compose
  by subpool-key union; S2+S5 = {full, swa, draft}, S3+S5 =
  {full, mamba, draft}, etc.

PLAN.md additions:
- Daemon impl item 11: fatal(reason, **context) helper as the
  shared entry point for every deployment-bug-class halt;
  enumerates the call sites.

Items not surfacing as a doc change:
- sgl-project#13 forecast recompute overhead: noted in audit but the spec
  already says always-fresh covers this; no edit needed.
- sgl-project#16 meta-invariant deployment-bug vs load-class: now covered by
  the new §10 'Two fault classes' invariant (above).

…-bench (jybsuper#12, sgl-project#20)

Squashes two LoRA MoE bench/test PRs onto lora-opti (replaces the dropped
EP8-vs-TP8 baseline-harness commit):
- sgl-project#12: Kimi-K2.5 down-proj LoRA-B expand-add UT + micro-bench + opt-in
  force_block_size_n in specialized_expand.py (BLOCK_SIZE_N tuning).
- sgl-project#20: bench_shrink_splitk.py micro-bench for the MoE LoRA-A shrink.

Co-Authored-By: Yusheng Su <[email protected]>
Co-Authored-By: fzyzcjy <[email protected]>

… bs>1 tests

Adversarial audit verified the DP core correct (~270k randomized
trials incl. parent-pointer reconstruction, bs>1 multi-axis, ties,
over-relief clamping) — no correctness bug.  Closed:

* sgl-project#9 (real, perf/DoS) — the auditor broke the DESIGN "microseconds"
  claim: |dp| reaches ~1.8M cells / 34s with large distinct relief/
  acquire bucket-deltas, and nothing guarded it, so a pathological
  candidate set could stall the daemon event loop for tens of seconds.
  Added a max_dp_cells ceiling (default 1e6, ~10x DESIGN's real-case
  estimate): past it both DPs raise KnapsackBudgetExceededError (with
  forensic ctx) → joint_decide maps to fatal (crash-only, DESIGN §10),
  failing loud instead of stalling.  Stage E1 trips it; DESIGN §9
  "microseconds" claim corrected.
* sgl-project#8 — KnapsackInfeasibleError.context now carries the candidate items
  (DESIGN's fatal(candidates=...)) so ops can see WHICH candidates were
  available (top-k undersizing vs a dropped candidate).  C0 asserts it.
* sgl-project#10/sgl-project#11/sgl-project#12 (test depth) — E0: exact-vs-brute at bucket>1 with
  MULTIPLE relief AND cap axes (A2 was 1+1 at bucket=1), via a
  quantising oracle; E2: empty items / zero need / zero budget.
* contract/NITs — docstrings state the phase precondition (min-cost
  takes Migrate/Pause, max-value takes Resume), bucket_size>0, and the
  safe-direction quantisation-can-halt note.

verify/t34: 15 stages (12 + E0/E1/E2), all green.  No DP-core change.

…determinism-CI seed)

Server-level guard crystallizing the B1 classification: boots the real
weightless server (TP=DCP=3) + a TP=1-solo baseline on the same GGUF and
asserts (1) EXTEND next-token logits bit-identical (Δ==0), (2) DECODE
free-gen trajectory diverges only within the intrinsic decode-kernel
fp-order band, with any first argmax flip landing on a near-degenerate
baseline tie. Assertion logic validated offline against the B1 captures
(extend Δ=0.0 exact; first flip @ step 4 at a 0.000-margin baseline tie).

…e fast-lane vs upstream

TOPOLOGIES section sgl-project#4 + mechanism diagram sgl-project#12 (session KV spill): add the
measured during-spill isolation matrix for the DEVICE-RESIDENT (non-spilled)
session — 10.4/13.4/19.9/26.0 tok/s at tick-interval 1/2/4/8 (pre-spill ~40),
spilled session ~7-8 tok/s. State honestly that the device session is still
dragged along: the isolation target holds only from tick-interval 4 up and is
violated at tick 1/2 because the spill step still runs eager on the shared
scheduler tick. Mark the planned bs=1 spill-graph (Step 5) as not-yet-built /
not-measured; numbers labelled MEASURED.

TOPOLOGIES section sgl-project#7 (MoE expert offload): replace the unrealistic 2x80GB /
EP-only stock reference with a realistic homogeneous config that also uses host
RAM — 2x RTX 3090 24 GB (TP=2) + --cpu-offload-gb (verified in server_args.py).
Neutral mechanism difference: --cpu-offload-gb is a generic per-layer-weight
offload that streams the offloaded weights every forward, whereas the fork
offload is expert-granular (per token-wave fetches only the routed top-K
experts) — a capability / data-volume difference, not a speed claim. Whole
stock panel estimated (stock cpu-offload not benched).

TOPOLOGIES section sgl-project#9 + mechanism diagram sgl-project#13 (fast-lane): name the precise,
code-verified difference vs upstream priority scheduling. Upstream = continuous
integer priority + preemption when priority_diff > preemption_threshold, no
protection for the preempted. Fork fast-lane = binary two-tier lane on that
path whose delta is two anti-starvation guarantees (reserved heavy slots never
preempted below a floor; heavy-aging promotes a long-waiting normal request
ahead of the fast tier) plus session-spill coupling. Stated as what each side
does, not a ranking. Framing stays neutral toward upstream throughout.

All SVGs regenerated via topologies/gen.py; validated well-formed and
overflow-free, all img references resolve.