Skip to content

Gateway fails to auto-restart after SIGTERM — launchd KeepAlive ineffective (2-hour outage) #50070

Description

@responseops-ai

OpenClaw Bug Report: Gateway fails to auto-restart after SIGTERM (launchd KeepAlive ineffective)

Summary

Gateway receives SIGTERM during config hot-reload and shuts down cleanly, but launchd does not restart it despite KeepAlive: true and ThrottleInterval: 1. Results in multi-hour outages requiring manual openclaw gateway start to recover.

Environment

  • OpenClaw: 2026.3.7 (42a1394)
  • OS: macOS 15.5 (arm64) — Mac Mini M4
  • Node: 22.22.1
  • Gateway mode: local
  • LaunchAgent: ai.openclaw.gateway

Reproduction

  1. Gateway running normally via LaunchAgent with KeepAlive: true
  2. Run openclaw config set channels.telegram.historyLimit 30 (or any config change)
  3. Gateway detects config change, hot-reloads, then receives SIGTERM ~8 seconds later
  4. Gateway shuts down cleanly (exit code 0)
  5. launchd does not restart the process

Evidence

Incident 1 — March 18, 2026 (2-hour outage)

2026-03-18T14:05:25.775 [reload] config change detected; evaluating reload (historyLimit)
2026-03-18T14:05:25.777 [gateway/channels] restarting telegram channel
2026-03-18T14:05:25.779 [reload] config hot reload applied (channels.telegram.historyLimit)
2026-03-18T14:05:26.455 [reload] config change detected; evaluating reload (contextPruning)
2026-03-18T14:05:26.457 [reload] config change applied
2026-03-18T14:05:34.173 [gateway] signal SIGTERM received
2026-03-18T14:05:34.174 [gateway] received SIGTERM; shutting down
--- 2 HOUR GAP - NO LOG ENTRIES ---
2026-03-18T16:07:06.562 [heartbeat] started  ← manual recovery by user

Incident 2 — March 17, 2026 (6-minute outage)

SIGTERM at 2026-03-17T21:56:04.832
Gateway back at 2026-03-17T22:02:25.668 (6 min 21 sec gap)

launchctl state after failure

state = running          ← only after manual restart
runs = 2                 ← launchd only started it twice total (boot + manual)
last terminating signal = Terminated: 15
properties = keepalive | runatload

LaunchAgent plist (relevant sections)

<key>KeepAlive</key>
<true/>
<key>ThrottleInterval</key>
<integer>1</integer>

Root Cause Analysis

openclaw config set writes the config file, which the running gateway detects via file watcher. The gateway hot-reloads some settings, but then receives SIGTERM ~8 seconds later. The SIGTERM appears to come from the config set CLI process itself (triggering a full restart via launchctl kickstart -k).

The gateway handles SIGTERM gracefully and exits with code 0. macOS launchd has a known quirk where a clean exit (code 0) after SIGTERM is sometimes treated as an intentional stop rather than a crash requiring restart — even with KeepAlive: true.

Expected Behavior

  1. openclaw config set should hot-reload config without killing the gateway process (it already does this for some settings — historyLimit was hot-reloaded successfully before the SIGTERM arrived)
  2. If a full restart IS needed, the restart mechanism should be reliable — the gateway should come back within seconds, not hours
  3. KeepAlive: true should guarantee auto-restart regardless of exit code

Workaround

We implemented an external watchdog script (runs every 30s via separate LaunchAgent) that checks port 18789 and runs launchctl kickstart -k if the gateway is down. This caps outage time at ~45 seconds.

Impact

  • 2-hour outage on March 18 — user's messages went unanswered, missed a business demo window
  • 6-minute outage on March 17
  • User had to manually run openclaw config set gateway.mode local && openclaw gateway install --force && openclaw gateway start each time
  • This is a production agent handling business operations via Telegram — reliability is critical

Suggested Fix

Either:

  1. Don't send SIGTERM on config changes that can be hot-reloaded (the reload already works — the SIGTERM is redundant)
  2. If SIGTERM is necessary, ensure the restart mechanism (launchctl kickstart -k) actually verifies the new process starts
  3. Add a built-in watchdog/health-check that auto-recovers if the gateway dies
  4. Exit with a non-zero code when killed externally, so launchd treats it as a crash requiring restart

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions