fix(watchdog): detect and restart wedged M2 workers, not just the orchestrator - #45

What happened

Yesterday's #44 covered the orchestrator. Overnight the same mlx crash took out both M2 workers and nothing noticed.

• developer crashed at 22:03, reviewer at 22:53 on 2026-08-24
• Both with [metal::malloc] Resource limit (499000) exceeded killing the generation thread
• From then until 05:10 every OpenCode request to either endpoint timed out at the meter's 300s header deadline — 126 failures, zero successes in seven hours

Every check the watchdog already ran kept passing. The machine was up, the Thunderbolt link was fine, memory was 98% free with no swap, and both servers answered /v1/models with 200 the whole time. /v1/models proves the process is alive, not that it can generate.

Detection

From the meter's telemetry, not another log grep — reading M2's logs would mean an ssh on every tick. The meter already records every client request, so an upstream carrying 3+ chat failures and zero successes over 15 minutes is wedged by definition.

Restricted to kind='chat', because only a real completion proves the generation thread is alive.

Deliberately left alone:

• failures alongside successes — ordinary life on a busy server with long prompts
• silence — no traffic is not evidence of a wedge, and a false positive costs a multi-minute 28 GB reload
• anything at all when M2 is unreachable — that is a link problem, handled by the branch below

Restart

No sudo needed. The workers are LaunchDaemons with KeepAlive SuccessfulExit=false running as admin, so killing the process makes launchd load a fresh one.

The pattern is mlx_lm[.]server, not mlx_lm.server: pkill -f matches full command lines, and the remote shell running the pkill carries the pattern text in its own. Without the bracket it can kill its own session.

Verification

• 58 tests pass, 8 new. They cover: only-failures restarts; failures with successes do not; silence does not; two failures are not enough; failures older than the window do not; a second tick inside the cooldown does not kill twice; the reviewer is watched on its own schedule; an unreachable M2 is not called a wedge.
• shellcheck clean.
• Both workers restarted by hand at 05:1x and verified serving through the meter on :9002 and :9003.
• Dry-run against the live stack correctly found nothing to do — both endpoints now show successes in the window.

Coverage after this

Docs updated in TROUBLESHOOTING.md (the crash affects every mlx endpoint, with the M2 restart command), OPERATIONS.md (a table of what the watchdog covers and how), and MODELS.md (the known-failure note now says every endpoint, not just the orchestrator).

🤖 Generated with Claude Code