feat(gateway): lane-health watchdog + post-deploy verification gate#34
Merged
Conversation
patelmm79
force-pushed
the
fix/lane-starvation-observability
branch
from
June 28, 2026 23:18
ba32a45 to
4be23a8
Compare
Adds L3b-side mitigation for the upstream bug in [email protected] where a wedged model_call lane starves all sibling agent lanes. The upstream runtime logs 'long-running session ... recovery=none' but takes no operator action and never preempts. Tracked in: DarojaAI/linux-desktop-seed/docs/incidents/2026-06-28-multi-session-gateway-starvation.md This commit ships three pieces: 1. scripts/lane-health-probe.sh Scans the gateway user journal every 30s (via user systemd timer) for events. When a lane exceeds the wall- clock budget (default 90s) with no stream progress AND recovery=none, kills it via 'openclaw lane kill' (best effort) and Discord-alerts the operator channel. Debounced per (sessionKey, hour). 2. scripts/install/install-lane-health-probe.sh Installs the probe as a user systemd timer (on-boot 30s, every 30s). Writes operator-channel to ~/.openclaw/operator-channel (mode 0600). Refuses on missing systemctl / unset HOME. 3. scripts/post-deploy-verify-lane-health.sh Health gate invoked at the end of deploy.sh. Refuses to declare a deploy healthy if any lane in the last 60s is wedged. Forces the operator to restart cleanly or acknowledge the wedge. 4. scripts/install/deploy.sh Wires the installer (step 7) and the verifier (step 8). Both can be skipped with SKIP_LANE_HEALTH_PROBE=1 / SKIP_POST_DEPLOY_LANE_CHECK=1 for emergency bypasses. 5. scripts/lib-extract-wedged-lanes.py Single source of truth for the long-running-session regex. Tolerant of optional fields (reason=, activeTool=, ...). Honors WEDGED_MIN_AGE_SECONDS env var. 6. tests/lane-health-probe.bats (13 cases, all passing) Covers detection, debouncing, filtering by kind/recovery/grace, installer behavior (operator channel, systemd units, symlink), post-deploy gate pass/fail. Uses stub journalctl + openclaw + systemctl on a sandboxed HOME. Verified: bats tests/lane-health-probe.bats 13/13 passing. shellcheck on all new scripts: only intentional warnings. Refs: docs/incidents/2026-06-28-multi-session-gateway-starvation.md (L3a) openclaw/openclaw issue TBD (upstream RC1-RC4 patches)
patelmm79
force-pushed
the
fix/lane-starvation-observability
branch
from
June 29, 2026 00:17
4be23a8 to
bd2cfd2
Compare
This was referenced Jun 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
On 2026-06-28 three user pings on
agent:linux_desktop_seed:discord:channel:1492701850217218268went unanswered for 13 hours. Root cause:[email protected]does not preempt a wedgedmodel_calllane. One agent's runaway model call held the event loop for 275s (recovery=none,liveness warning eventLoopDelayMaxMs=1841.3), starving every sibling agent lane.The upstream runtime logs
[diagnostic] long-running session ... recovery=nonebut takes no operator action. This PR ships the L3b-side mitigation that the upstream bug forces us to own.Full incident timeline + root causes in linux-desktop-seed/docs/incidents/2026-06-28-multi-session-gateway-starvation.md.
What
scripts/lane-health-probe.sh— watchdog that scans the gateway journal every 30s, kills wedged lanes (openclaw lane killbest-effort), alerts operator.scripts/install/install-lane-health-probe.sh— installs the probe as a user systemd timer.scripts/post-deploy-verify-lane-health.sh— deploy-time gate that fails the deploy if any lane is wedged.scripts/install/deploy.sh— wires both into the install flow (steps 7 + 8).scripts/lib-extract-wedged-lanes.py— single source of truth for the long-running-session regex.tests/lane-health-probe.bats— 13 cases (all passing).Tests
Caveats / known limits
openclaw/openclawnpm package and requires upstream maintainer action. I'll file the upstream issue + minimal-patch PR separately.bash scripts/lane-health-probe.sh) but loses the 30s scheduling — caller is expected to install the timer.openclaw lane killis a hypothetical subcommand in[email protected](the current binary may not have it). The probe falls back tosystemctl --user restart openclaw-gateway.servicewhen the subcommand is missing. Once upstream ships lane-management subcommands, the probe will use them automatically.Behavior knobs
LANE_WALL_BUDGET_SECONDSLANE_GRACE_SECONDSSKIP_LANE_HEALTH_PROBE=1SKIP_POST_DEPLOY_LANE_CHECK=1OPENCLAW_OPERATOR_CHANNELRefs: DarojaAI/linux-desktop-seed#1054 (postmortem), openclaw/openclaw issue TBD