Repository navigation
M5 Max (T6050) total-system watchdog panic under ~112 GB engine pool: 'no checkins from watchdogd in 91 s' (macOS 27.2/26B123) #735
Description
Machine / stack
- Mac Studio M5 Max 128 GB (T6050), macOS 27.2 (26B123), Darwin 27.2.0 xnu-13432.41.4~25 RELEASE_ARM64_T6050
iogpu.wired_limit_mb = 124518- Splash 1.4.0 serving Qwen3.8-Flash-Next PLE8 — ~112 GB live Metal engine pool — plus 2 MLX
qwen3_ttsvoice sidecars - Always-on inference server, sleep disabled; the box runs at zero memory headroom by design
What happened
Two kernel panics in one day (2026-11-09, 14:27 and 18:33 local), identical signature:
panic(cpu N caller ...): watchdog timeout: no checkins from watchdogd in 91 seconds
Total-system stall — every online core reports recently retired instr at 0x0; the ARM watchdog timer is the only thing left breathing. The offline cluster is complementary between the two deaths (first death: MACC0 offline / MACC1 panicked; second death: MACC1 offline / MACC0 panicked). At death, in both panics: memoryPressure: false, compressor effectively empty (2% / 0% of limit), wired ≈ 123.4 GB of 128 GB. Full panic-full bodies captured verbatim and filed with Apple (Feedback Assistant, FB25509461, filed 2026-11-09).
Shadow signatures we see fleet-side (503/500 from Splash across the fall window)
engine_recovering ... native loop did not answer status within 30 sfailed 3 times in a row ... native protocol reached EOF
Splash's engine is the dominant wired resident (~112 GB), so it takes the blame in the shadows — but the kernel panic is a whole-system freeze. On recovery the engine crash-loops through the two signatures above before serving again.
Framing
Metal/ANE residency maintenance under very large wired workloads appears to unwire en masse; on a zero-headroom system that dip is fatal — the stall exceeds the watchdogd checkin interval (>90 s) and the machine panics. Companion 1 s vm_stat telemetry shows the wired pool dipping sharply seconds before the stall.
Related
- Qwen3.8-Flash-Next GGUF: a request with ~10 page images crashes the engine with a Metal GPU Timeout (6 images work) #596 (closed) — Metal GPU Timeout on the same model family; userland engine death, not a kernel stall
- Automatic memory budget lets the prefix cache take ~105 GiB on a 128 GB Mac shared with other apps; the Mac panicked #325 (closed) — auto memory budget took ~105 GiB on a 128 GB Mac; the Mac panicked
- jundot/omlx#4981 — same build 26B123, dip-to-watchdog-panic with 1 s vm_stat data
- jundot/omlx#3065 — same panic string, M5 Max 128 GB, macOS 26.5.2 (upstream twin reports 94 s)
- ml-explore/mlx#3186 + exo-explore/exo#1972 — IOGPUMemory
completeMemorysiblings
Standing observation law
The fleet pages on watchdog on every boot — if this falls again we'll append a fresh full panic body to this issue within minutes.
Activity
linson007 commentedon Nov 10, 2026
Thanks for the detailed report and the panic bodies. I checked the current code (1.6.0) against it. Notes and requests below.
Version. 1.6.0 (Nov 6) is the latest release. This report is on 1.4.0, and 1.6.0 changed the memory governor, the residency set and the restart policy. If you can safely run 1.6.0 on that Mac, that is the most useful next test. The notes below apply either way.
How the pool is sized
recommendedMaxWorkingSetSizeminus the larger of 1 GiB or 2% (runtime/engine/MemoryPlan.hpp). The automatic plan takes the whole budget, and--max-memorycan only lower it. The budget caps the engine's Metal allocations, so the sidecars' memory comes on top (docs/server.md, "Memory and context").iogpu.wired_limit_mb. Its budget starts fromrecommendedMaxWorkingSetSize.--idle-release(default10m) unwires the engine's memory and frees the weights after an idle period. The next request restores them. An engine exit or restart also releases its memory.What would help
--max-memory,--idle-release,--expert-memory, and how the two sidecars run.memory_planbudget andmemory_governorblocks fromGET /status, captured while the engine holds its ~112 GB./v1/memory/releasecall, an engine exit or restart, or requests in flight at that time.vm_stattelemetry, if you can share it.iogpu.wired_limit_mbchanges therecommendedMaxWorkingSetSizethat/statusreports.The "failed 3 times in a row" line is the old limit. 1.5.0 stopped at the third failure, and current code stops at six, with backoff between tries (
server/supervision.py).Related: #220 (sparse GPU mapping stalls and WindowServer logouts on M5 Max 128 GB, closed after 1.2.0 removed placement-sparse KV mapping) and #325 (cache growth on a 128 GB Mac, closed after 1.3.2). Both are useful context, but neither is this mechanism.
Interim: a
--max-memorybelow the automatic budget leaves room for the sidecars and macOS, as docs/server.md suggests for shared Macs.Question for maintainers: should the automatic budget keep more than 2% free on 128 GB Macs when other workloads run beside it? #325 raised the same question for cache growth.