Repository navigation
fix(watchdog): restart the orchestrator on the mlx crash, and stop guessing its model id - #44
Merged
Merged
Conversation
…essing its model id Two orchestrator problems surfaced on 2026-08-24 during a long OpenCode session. 1. mlx-lm leaks live Metal buffer descriptors while decoding and throws `[metal::malloc] Resource limit (499000) exceeded`, which kills the server's single generation thread. 499000 caps the NUMBER of live Metal buffers, not bytes -- a probe allocating 499,000 tiny arrays throws at exactly that count with 2 MB in use, so this is not an out-of-memory condition. The process survives and keeps answering /v1/models with 200 while every completion hangs forever, so only a real completion detects it. The completion probe needed ORCH_BAD_LIMIT x ORCH_PROBE_INTERVAL (~9 min) to call that a wedge, and `orchestrator_served_recently` stretched it further by reading a pre-crash success as "busy, not wedged". Four crashes in 90 minutes cost ~36 minutes of hung requests. The traceback is unambiguous and free to grep, so the watchdog now scans the server log every tick from a persisted offset and restarts on first sight. Rotation and truncation reset the offset, and the first run adopts the current end so already-recovered crashes are not replayed. Upstream: ml-explore/mlx-lm #831, #1185, #1332 -- all open, no released fix. Restarting is the only lever we have. Drop this block once that changes. 2. The completion probe hardcoded the bf16 orchestrator path (65 GB) while the stack serves mixed9 (38 GB). To mlx_lm.server the `model` field is a load instruction, not a selector, so only the meter's METER_ORCH_MODEL pin stopped the health probe from loading a second copy. It now reads the served id from /v1/models, the same rule already applied in smoke-test-endpoints.sh and the dashboard's pickChatModel(). If /v1/models cannot be read the probe skips rather than guessing -- an unreadable meter is not evidence of a wedge. Verified against the live stack: the probe resolved to the mixed9 path and completed in 694 ms, and the first tick recorded log offset 784378 without restarting on the four crashes already in that log. Docs record the failure signature, the restart command, and the measured crossover where the orchestrator stops being the right endpoint (faster than the developer endpoint below 150k tokens, much slower above it -- which is also where it crashes). MODELS.md now names mixed9 as what is actually served, and the bulk-classification lane gains its stop command. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This was referenced Aug 25, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What broke
A long OpenCode session on 2026-08-24 produced 16 errors on the orchestrator inside 90 minutes. Server-side, the generation thread died four times:
This is not an out-of-memory error. 499000 caps the number of live Metal buffers a process may hold. A probe allocating 499,000 tiny arrays throws at exactly that count with 2 MB in use, and the cap is per process — a second model server on the same Mac does not eat into it. Both were measured, not assumed.
The cause is a known leak of Metal buffer descriptors in mlx-lm's decode path: #831, #1185, #1332. All open, no released fix.
Crash times were 14:49:50, 15:12:10, 15:34:45, 15:57:05 — one every 22 minutes 20 seconds while OpenCode drove the context from 34k to 174k tokens.
Why it took ~9 minutes to recover each time
The process survives the crash and keeps answering
/v1/modelswith 200. Only real inference hangs. The completion probe needs two failures 5 minutes apart, andorchestrator_served_recentlycan stretch that further by reading a pre-crash success as "busy, not wedged".Changes
Restart on the crash, not on the timeout. The watchdog now greps the server log every tick from a persisted byte offset. Log reads are free, so this is not gated by
ORCH_PROBE_INTERVAL. The first run adopts the current end of the log, and truncation resets the offset — deploying this with today's four recovered crashes still in the log must not restart the 35B, and does not.Stop guessing the model id. The probe hardcoded the bf16 path (65 GB) while the stack serves mixed9 (38 GB). To
mlx_lm.serverthemodelfield is a load instruction, not a selector, so only the meter'sMETER_ORCH_MODELpin stopped the health probe from loading a second copy. It now reads the served id from/v1/models— the rule already applied insmoke-test-endpoints.shand the dashboard'spickChatModel(). If/v1/modelscannot be read the probe skips rather than guessing.Verification
tests/test_m2_watchdog_orchestrator_crash.py) covering: crash in new log restarts; a crash already present at startup does not; a truncated log does not replay; the probe uses the reported model and never names bf16; an unreachable endpoint is not called wedged.test_m2_watchdog_probe_labeling.py's fake curl now returns a real model list for plain/v1/models, since the probe reads it.Docs
TROUBLESHOOTING.mdgets the failure signature and the manual restart.MODELS.mdnow names mixed9 as what is actually served (it listed bf16 as primary, which is what seeded the probe bug) and records the crash under the Orchestrator entry.Both record the measured crossover for keeping long contexts off this endpoint:
Not in this PR
The idle bulk-classification lane was stopped by hand (40 GB, idle 8 hours);
MODELS.mdgains its stop command. Capping OpenCode's context on the orchestrator is a change to the user's OpenCode config, not this repo.🤖 Generated with Claude Code