Repository navigation
[Bug] KV cache corruption at radix cache block boundary with --enable-deterministic-inference (prefix_len == block_size) #22819
Description
Checklist
- I searched related issues but found no solution.
- The bug persists in the latest version.
- Issues without environment info and a minimal reproducible demo are hard to resolve and may receive no feedback.
- If this is not a bug report but a general question, please start a discussion at https://github.com/sgl-project/sglang/discussions. Otherwise, it will be closed.
- Please use English. Otherwise, it will be closed.
Describe the bug
We found a KV cache corruption bug in SGLang's radix cache that reproduces with --enable-deterministic-inference enabled (overlap prefill-decode scheduler disabled). Identical prompts at temperature=0 produce completely different output sequences across runs.
The key trigger: a request whose prefix_len is exactly equal to the KV block size (64 tokens) consistently gets corrupted while requests with prefix_len=0 in the same concurrent batch are unaffected.
- Reproduction rate: 10/10 runs
--enable-deterministic-inference: does NOT fix it — this is not an overlap scheduler race- Confirmed via: automated fuzzing, then isolated and re-confirmed manually
Reproduction
Among 15 problematic issues, we report the cleanest one:
repro script with finding_03330 baked in: repro.py
Reproduction
Step 1 — start SGLang:
python3 -m sglang.launch_server \
--model Qwen/Qwen2.5-0.5B-Instruct \
--context-length 32768 \
--mem-fraction-static 0.8 \
--enable-deterministic-inference \
--port 8000 &Step 2 — replay the finding using the attached standalone script:
python3 repro.py --base-url http://localhost:8000 --runs 10detail of finding_03330
11 requests sent across two burst waves. Only r02 diverges — the one request whose prefix_len=64 equals exactly one KV block (block size = 64 tokens). Every other request has prefix_len=0 and is clean.
Request trace
| request | offset_ms | prompt_len | prefix_len | max_tokens | n | diverged |
|---|---|---|---|---|---|---|
| r01 | 0 | 128 | 0 | 32 | 4 | |
| r02 | 45 | 256 | 64 | 32 | 4 | ✓ |
| r03 | 109 | 64 | 0 | 32 | 4 | |
| r04_s_b_s_s_s | 385 | 2048 | 0 | 32 | 4 | |
| r02_b_s_b_s_s_s | 385 | 1024 | 64 | 32 | 4 | |
| r03_s_s_s_s | 486 | 256 | 0 | 32 | 4 | |
| r01_b_s_s_s_s | 518 | 512 | 0 | 32 | 4 | |
| r03_b_s_s_s_s | 526 | 256 | 0 | 32 | 4 | |
| r04_s_s_s_s | 537 | 2048 | 0 | 32 | 4 | |
| r04_b_s_s_s_s | 542 | 2048 | 0 | 32 | 4 | |
| r02_b_s_s_s_s | 547 | 1024 | 64 | 32 | 4 |
Key observations:
r02(prefix_len=64) diverges in every run — 3 distinct output variants across 10 runs.r03(prefix_len=0, prompt_len=64 — same number of new tokens as r02) is perfectly stable.- All 8 other requests with prefix_len=0 are stable, including
r04_s_b_s_s_s(prompt_len=2048). r02_b_s_b_s_s_sandr02_b_s_s_s_salso have prefix_len=64 but are stable across runs — consistent with the corruption being timing-sensitive to the first burst window when the block is first allocated.
Diverged output for r02
variant 1: ' u7892_1146 u7892_1086 u7892_108 u78...'
variant 2: ' u7892_1446 u7892_144 u7892_144 u789...'
variant 3: ' u7892_1446 u7892_1144 u7892_1142 u7...'
Divergence starts at token 1 (_1146 vs _1446 vs _1446). The correct output should be stable across runs at temperature=0 — instead, r02 produces 3 distinct sequences across 10 runs. The other 10 requests, including r03 which has the same prompt length, are perfectly stable.
The outputs are non-random, each variant is internally consistent, which is consistent with a stale or misassigned KV block being deterministically reused from a prior request's prefix.
below is the repro results, using the scripts provided above!
============================================================
Finding : finding_03330_2142180798.json
Bug : kv_corruption
Expect : ['r02'] to diverge
Runs : 10
============================================================
[DIVERGED — 1/11 requests non-deterministic]
r02 ✓ (expected)
variant 1: ' u7892_1146 u7892_1086 u7892_108 u78...'
variant 2: ' u7892_1446 u7892_144 u7892_144 u789...'
variant 3: ' u7892_1446 u7892_1144 u7892_1142 u7...'
[STABLE — 10 requests consistent across all runs]
r01, r01_b_s_s_s_s, r02_b_s_b_s_s_s, r02_b_s_s_s_s,
r03, r03_b_s_s_s_s, r03_s_s_s_s, r04_b_s_s_s_s, r04_s_b_s_s_s, r04_s_s_s_s
RESULT: CONFIRMED (1/1 expected divergences reproduced)
============================================================
Environment
check_env
Python: 3.12.3 (main, Mar 3 2026, 12:15:18) [GCC 13.3.0]
CUDA available: True
GPU 0: NVIDIA RTX A6000
GPU 0 Compute Capability: 8.6
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 12.9, V12.9.86
CUDA Driver Version: 590.48.01
PyTorch: 2.9.1+cu129
sglang: 0.5.10.post1
sglang-kernel: 0.4.1
flashinfer_python: 0.6.7.post3
flashinfer_cubin: 0.6.7.post3
flashinfer_jit_cache: 0.6.7.post3+cu129
triton: 3.5.1
transformers: 5.3.0
torchao: 0.9.0
numpy: 2.3.5
aiohttp: 3.13.5
fastapi: 0.135.3
huggingface_hub: 1.9.2
interegular: 0.3.3
modelscope: 1.35.3
orjson: 3.11.8
outlines: 0.1.11
packaging: 26.0
psutil: 7.2.2
pydantic: 2.12.5
python-multipart: 0.0.24
pyzmq: 27.1.0
uvicorn: 0.44.0
uvloop: 0.22.1
vllm: Module Not Found
xgrammar: 0.1.32
openai: 2.6.1
tiktoken: 0.12.0
anthropic: 0.92.0
litellm: Module Not Found
torchcodec: 0.9.1
NVIDIA Topology:
�[4mGPU0 NIC0 NIC1 NIC2 CPU Affinity NUMA Affinity GPU NUMA ID�[0m
GPU0 X NODE NODE SYS 16-31 1 N/A
NIC0 NODE X PIX SYS
NIC1 NODE PIX X SYS
NIC2 SYS SYS SYS X
Legend:
X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks
NIC Legend:
NIC0: mlx5_2
NIC1: mlx5_3
NIC2: mlx5_bond_0
ulimit soft: 1024
Activity
rouye-he commentedon Apr 15, 2026
Hi @Yunzez , could you share the repro.py script? I'd like to try to reproduce and investigate this bug.
fix: wire disable_finished_insert for deterministic inference (sgl-pr…
ZhichaoWang970201 commentedon Apr 15, 2026
Can you try if this can fix the problem: #22841?
Yunzez commentedon Apr 16, 2026
Tested on: H100 80Gs, Qwen25-0.58-Instruct, --men-fraction-static 0.53 -context-Length 32768 -enable-deterinistic-inference
9/11 requests still diverged, doesn't seem to be fixed
zhaoyboo commentedon Apr 16, 2026
Also tested with
--disable-radix-cache, divergences disappear, seems to be an radix cache contention issue.fix: also gate cache_unfinished_req in deterministic mode (sgl-projec…
ZhichaoWang970201 commentedon Apr 16, 2026
Can you try again on this branch: https://github.com/ZhichaoWang970201/sglang/tree/fix/deterministic-disable-finished-insert
I used to have trouble reproducing it in B200.
Yunzez commentedon Apr 17, 2026
@ZhichaoWang970201 To reproduce on B200 which has 192 GB memory, I think you will have to change the setting to --men-fraction-static 0.23, since Qwen25-0.58-Instruct is extremly small for B200.
ZhichaoWang970201 commentedon Apr 17, 2026
@Yunzez I have updated this code, try it agin: https://github.com/ZhichaoWang970201/sglang/tree/fix/deterministic-disable-finished-insert
ZhichaoWang970201 commentedon Apr 19, 2026
@Yunzez Let me know if it fixes your problem.
Yeuvoir commentedon May 16, 2026
Hi, I'm looking into this.
I was able to reproduce the divergence on H100 with Qwen/Qwen2.5-0.5B-Instruct and
--enable-deterministic-inference. In my local runs, the default deterministic path resolves to FA3 with radix cache enabled, and the replay diverges. The same replay is clean with--disable-radix-cache, and is also clean with--attention-backend tritonwhile keeping radix cache enabled.So the unsafe combination appears to be deterministic FA3 + radix prefix-cache reuse, rather than finished-request insertion. I'm preparing a conservative PR that stops treating FA3 as radix-compatible under deterministic inference: with radix cache enabled, deterministic inference will default to Triton; if FA3 is explicitly requested, radix cache will be disabled.
This avoids silent correctness issues while leaving a deeper FA3+radix fix as separate follow-up work.
feat(steppable_engine): drop unimplemented PR fix toggle fields (keep s…
github-actions commentedon Jul 16, 2026
This issue has been automatically closed due to inactivity. Please feel free to reopen it if needed.
jwoongkim11 commentedon Jul 27, 2026
Hi — I tried to reproduce this on current
main-era code and could not, so I'm posting the data in case it is useful for deciding whether this should be reopened or closed as fixed.Configuration
Method. Prefix lengths were controlled exactly by sending
input_idsrather than text. For each length N, a prefix P of exactly N tokens plus a fixed 5-token suffix Q was run three ways and the outputs compared:cold—/flush_cache, then runP+Qself— runP+Qagain immediately (its own prefix now cached)primed—/flush_cache, runPalone (caches exactly N tokens), thenP+QOutputs were compared as a hash over output token ids, per-token logprobs, text and finish reason, so any bit-level difference shows up.
Results — no divergence observed
I also ran the standard repeatability battery (single / mixed / prefix / structured) with radix cache ON at 300 trials x 9 request slots, concurrency 8 = 2,700 generations: every slot produced exactly 1 unique output hash.
What I am not claiming. This is a non-reproduction under one configuration, not evidence that the bug does not exist. In particular I did not vary
page_size, did not test multi-request cache eviction pressure, and used a single model/GPU generation.Questions