Bug Report: Gateway hangs after SIGUSR1 signal, systemd cannot auto-restart

🐛 Bug Description

Title: Gateway hangs after receiving SIGUSR1 signal for config reload, systemd cannot detect failure and auto-restart

Severity: High - Gateway becomes unresponsive and requires manual intervention

Version: OpenClaw 2026.3.22

📊 Environment

🔍 Reproduction Steps

Step 1: Configure systemd service

[Service]
ExecStart=/path/to/node /path/to/openclaw/dist/index.js gateway --port 18789
Restart=on-failure
RestartSec=5
KillMode=process
SuccessExitStatus=0 143
StartLimitIntervalSec=300
StartLimitBurst=10

Step 2: Trigger config change

Modify ~/.openclaw/openclaw.json to trigger auto-reload:

• Add plugin to plugins.allow
• Add plugin to plugins.entries
• Add plugin to plugins.installs

Step 3: Gateway receives SIGUSR1

Gateway logs show:

15:51:31 [reload] config change detected; evaluating reload
15:51:31 [reload] config change requires gateway restart
15:51:31 [gateway] signal SIGUSR1 received
15:51:31 [gateway] received SIGUSR1; restarting
15:51:31 [gateway] restart mode: full process restart (supervisor restart)

Step 4: Gateway hangs

What happens:

• Gateway process does NOT exit
• Gateway enters "supervisor restart" mode
• Gateway becomes unresponsive for 7+ minutes
• systemd does not detect failure (process still running)
• systemd does not trigger auto-restart

Logs:

15:51:31 [gateway] restart mode: full process restart (supervisor restart)
15:51:31 systemd: Consumed 2min 20.582s CPU time.
# NO logs for 7 minutes 17 seconds
15:58:48 systemd: Started OpenClaw Gateway (v2026.3.22).  # User manually restarted

Step 5: Manual intervention required

User must manually restart:

systemctl --user restart openclaw-gateway.service

❌ Expected Behavior

Option A: Gateway should exit gracefully

15:51:31 [gateway] received SIGUSR1; restarting
15:51:31 [gateway] graceful shutdown initiated
15:51:31 systemd: Gateway exited with code 0
15:51:36 systemd: Restarting Gateway (Restart=on-failure)
15:51:36 systemd: Started Gateway

Option B: systemd should detect hang and restart

15:51:31 Gateway receives SIGUSR1
15:56:31 systemd: Gateway unresponsive for 5 minutes
15:56:31 systemd: Killing Gateway (WatchdogSec=300)
15:56:31 systemd: Restarting Gateway

❌ Actual Behavior

15:51:31 Gateway receives SIGUSR1
15:51:31 Gateway enters "supervisor restart" mode
15:51:31 - 15:58:48 Gateway hangs (7 minutes 17 seconds)
15:58:48 User manually restarts Gateway

Key Issues:

1. Gateway process does NOT exit after SIGUSR1
2. systemd cannot detect failure (process still running)
3. systemd does NOT trigger auto-restart
4. Self-healing mechanism does NOT trigger (service didn't fail)
5. Gateway becomes completely unresponsive

🔍 Root Cause Analysis

Issue 1: Gateway "supervisor restart" mode

Problem:

• Gateway receives SIGUSR1
• Gateway attempts internal "supervisor restart"
• Gateway process does NOT exit
• Gateway becomes unresponsive during "supervisor restart"

Why systemd cannot help:

• systemd monitors process state
• Process is still running (not crashed)
• systemd sees process as "active (running)"
• systemd does NOT trigger Restart=on-failure

Issue 2: No watchdog mechanism

Problem:

• No WatchdogSec configured
• Even if Gateway hangs, systemd won't detect it
• No timeout for "supervisor restart" mode

Issue 3: Self-healing mechanism doesn't trigger

Problem:

• Self-healing triggers on service failure
• Service didn't fail (process still running)
• Self-healing script (openclaw-fix.sh) not triggered

📊 Impact

🔧 Suggested Fixes

Fix 1: Make Gateway exit gracefully on SIGUSR1

Change Gateway behavior:

// Current behavior (problematic)
process.on('SIGUSR1', () => {
  // Attempt internal supervisor restart
  // Process does NOT exit
  // Gateway hangs
});

// Suggested fix
process.on('SIGUSR1', () => {
  // Graceful shutdown
  process.exit(0);
  // Let systemd handle restart
});

Fix 2: Add timeout to supervisor restart

Add timeout mechanism:

process.on('SIGUSR1', () => {
  const timeout = setTimeout(() => {
    console.error('Supervisor restart timeout, forcing exit');
    process.exit(1);  // Exit with error to trigger systemd restart
  }, 60000);  // 1 minute timeout
  
  // Attempt supervisor restart
  // If successful, clearTimeout(timeout)
  // If fails, timeout will trigger exit
});

Fix 3: Add systemd watchdog support

Add to systemd service:

[Service]
WatchdogSec=300

Add to Gateway code:

// Send watchdog notifications
setInterval(() => {
  notifyWatchdog();
}, 60000);  // Every 1 minute

Fix 4: Improve systemd configuration

Recommended configuration:

[Service]
Restart=on-failure
RestartSec=5
KillMode=mixed  # Kill entire process group, not just main process
SendSIGKILL=yes  # Ensure process is killed
TimeoutStopSec=30  # Force kill after 30 seconds
WatchdogSec=300  # Restart if unresponsive for 5 minutes

📝 Workaround

Current Workaround

Manual restart:

# When Gateway hangs
systemctl --user restart openclaw-gateway.service

Suggested Temporary Fix

Add watchdog to systemd:

# Edit service file
systemctl --user edit openclaw-gateway.service

# Add watchdog
[Service]
WatchdogSec=300

Reload and restart:

systemctl --user daemon-reload
systemctl --user restart openclaw-gateway.service

📊 Timeline

📎 Attachments

Attachment 1: Gateway Logs (15:51-15:59)

3 月 24 15:51:31 node[555515]: [reload] config change detected
3 月 24 15:51:31 node[555515]: [reload] config change requires gateway restart
3 月 24 15:51:31 node[555515]: [gateway] signal SIGUSR1 received
3 月 24 15:51:31 node[555515]: [gateway] received SIGUSR1; restarting
3 月 24 15:51:31 node[555515]: [gateway] restart mode: full process restart (supervisor restart)
3 月 24 15:51:31 systemd[964]: Consumed 2min 20.582s CPU time.
# NO LOGS FOR 7 MINUTES
3 月 24 15:58:48 systemd[964]: Started OpenClaw Gateway (v2026.3.22).

Attachment 2: systemd Service Configuration

[Unit]
Description=OpenClaw Gateway (v2026.3.22)
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
ExecStart=/usr/bin/node /home/user/.npm-global/lib/node_modules/openclaw/dist/index.js gateway --port 18789
Restart=on-failure
RestartSec=5
TimeoutStopSec=30
KillMode=process
SuccessExitStatus=0 143
StartLimitIntervalSec=300
StartLimitBurst=10
Environment=HOME=/home/user

[Install]
WantedBy=default.target

Attachment 3: Drop-in Configuration (auto-fix.conf)

[Unit]
OnFailure=openclaw-fix.service
StartLimitIntervalSec=300
StartLimitBurst=10

[Service]
# Empty - just for OnFailure trigger

🎯 Expected Fix Priority

High Priority:

1. Fix Gateway to exit gracefully on SIGUSR1
2. Add timeout to supervisor restart mode
3. Add systemd watchdog support

Medium Priority:
4. Improve systemd configuration (KillMode, TimeoutStopSec)
5. Add health check endpoint
6. Improve self-healing trigger logic

📝 Additional Notes

Why This Is Critical

1. Service Availability: 7+ minutes downtime per incident
2. Manual Intervention: Requires user to manually restart
3. Auto-recovery Broken: systemd cannot auto-restart
4. Self-healing Broken: Doesn't trigger because service didn't fail

Related Issues

• systemd Restart=on-failure only triggers on process exit
• Gateway "supervisor restart" keeps process alive
• systemd sees process as "active (running)"
• No mechanism to detect hung process

Created: 2026-03-24 | Gateway SIGUSR1 Hang Bug Report