Skip to content

bug: Telegram polling watchdog triggers full gateway restart under concurrent multi-agent load (10 agents) #43178

Description

@dwightstuckey

Bug Report: Telegram Polling Watchdog Triggers Full Gateway Restart Under Concurrent Agent Load

OpenClaw Version: 2026.3.9 (f063e57)
OS: macOS 26.3.1 (Darwin 25.3.0, arm64 / Apple Silicon)
Node.js: v25.6.1
Install method: npm global
Date: March 11, 2026


Summary

When multiple agents (8–10) are contacted simultaneously via sessions_send, the Node.js event loop stalls briefly under concurrent I/O load. The Telegram polling watchdog interprets this as a "dead polling loop" and triggers a full gateway process restart (SIGUSR1). This kills all active sessions mid-execution and drops all queued messages — including user messages actively being typed.

The gateway restart is disproportionate to the actual failure. A brief event loop stall is not a dead Telegram connection.


Steps to Reproduce

  1. Configure 9–10 agents in agents.list (each with their own workspace, memory files, Telegram bot)
  2. From one session, call sessions_send to 8–9 agents simultaneously (e.g. a Wednesday morning brief orchestration, or a system-wide roll call)
  3. Each agent begins processing: reading SOUL.md, MEMORY.md, making LLM API calls — concurrent file I/O + HTTP
  4. After ~60–90 seconds, observe:
gateway tool: restart requested (delayMs=default, reason=Telegram polling watchdog: polling loop died, auto-restarting)
received SIGUSR1; restarting
draining N active task(s) before restart
  1. Gateway restarts. All active user sessions receive:
⚠️ Agent failed before reply: Gateway is draining for restart; new tasks are not accepted

Expected Behavior

  • The Telegram polling watchdog should tolerate brief event loop stalls (< 5–10 seconds) without triggering a full gateway restart
  • OR the watchdog should restart only the Telegram polling subsystem, not the entire gateway process
  • Active user sessions should not be dropped due to an internal polling hiccup

Actual Behavior

  • Full SIGUSR1 gateway restart triggered
  • All active sessions dropped mid-reply
  • User sees "Gateway is draining for restart" error
  • Gateway takes ~30–90 seconds to come back up
  • Any messages sent during drain window are lost

Environment Details

  • Mac mini (Apple Silicon M-series), 24GB RAM — hardware is not the bottleneck
  • 10 agents configured, each with ~20–30KB workspace memory files
  • All agents on same gateway (single-process Node.js)
  • Telegram polling: long-poll mode, 10 bots polling simultaneously
  • CPU at restart time: ~20–30% — not CPU-bound
  • Concurrent LLM calls at restart time: 8–9 (all I/O-bound async HTTP)

Log Evidence

2026-03-11T07:13:54.219-05:00: gateway tool: restart requested (delayMs=default, reason=Telegram polling watchdog: polling loop died, auto-restarting)
2026-03-11T07:14:15.265-05:00: signal SIGUSR1 received
2026-03-11T07:14:15.270-05:00: received SIGUSR1; restarting
2026-03-11T07:58:06.473-05:00: draining 4 active task(s) and 1 active embedded run(s) before restart (timeout 90000ms)

Impact

Severity: High — affects any multi-agent deployment. The more agents active simultaneously, the more likely a restart. This is the exact use case OpenClaw is designed for.

  • Users lose access to all agents during the restart window
  • Messages sent during drain are silently dropped
  • No user notification that a restart is occurring
  • In a 10-agent business deployment, this happens multiple times per day under normal load

Suggested Fixes

  1. Increase watchdog tolerance — don't declare polling "dead" until stall exceeds 15–30 seconds (not the current threshold which appears to be ~60s of total inactivity but triggers prematurely under load)
  2. Isolate the watchdog restart scope — restart the Telegram polling subsystem only, not the full gateway process
  3. Notify users before draining — send a Telegram message ("Brief maintenance restart in 30s") before triggering drain
  4. Configurable watchdog sensitivity — allow channels.telegram.pollingWatchdog.toleranceMs in config

Workaround (Applied)

Stagger agent contacts — contact agents sequentially (one at a time) instead of simultaneously. This reduces peak I/O load and prevents the event loop stall. Works but significantly slows multi-agent orchestration workflows.


Related Issues


Filed by Zoe (AI Operations Co-Pilot, Stuckey & Company) on behalf of Dwight Stuckey
Contact: [email protected]

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions