Skip to content

WhatsApp: health-monitor race condition causes multi-hour outages #70463

Description

@YYChien

OpenClaw WhatsApp Issue: Health monitor race condition with Baileys 30-minute disconnections

Problem Summary

WhatsApp Web gateway experiences a race condition that causes ~7-hour outages:

  1. Normal operation: WhatsApp server disconnects every ~30 minutes (status 499)
  2. Baileys retry mechanism: Automatically attempts reconnection (Retry 1/12 in ~2s)
  3. Health monitor: Also detects disconnection and restarts channel
  4. Race condition: Both systems try to control the same channel simultaneously
  5. Result: Channel enters shutdown state, Baileys retry stops, health monitor restart fails
  6. Outcome: ~7-hour complete outage until manual gateway restart

Expected Behavior

  • Baileys retry should continue until successful reconnection
  • Health monitor should work independently without interfering with retries
  • No manual intervention should be required for routine disconnections

Root Cause Analysis

Timing

  • messageTimeoutMs: 1,800,000ms (30 minutes) - hardcoded default
  • watchdogCheckMs: 60,000ms (1 minute) - hardcoded default
  • WhatsApp server disconnect: ~30 minutes (variable timing)

Race Condition Details

  1. minutesSinceLastMessage > 30 triggers watchdog restart
  2. Baileys retry mechanism is also active simultaneously
  3. Channel gets stuck in "channel stop exceeded 5000ms after abort" state
  4. No further retries occur from either system
03:02:13 ⚠️ No messages received in 30m - restarting connection
03:02:54 [health-monitor] restarting (reason: disconnected)
03:02:59 channel stop exceeded 5000ms after abort; continuing shutdown
03:03:16 channel exited (408)
# Silence for 7 hours until manual restart
09:56:17 [gateway] ready

Proposed Solution

Phase 1: Configurable timeouts (Priority)

Add config keys to prevent hardcoding and allow adjustments:

channels:
  whatsapp:
    web:
      messageTimeoutMinutes: 35  # Higher than server disconnect timing
      watchdogCheckSeconds: 120  # Less frequent to avoid race condition

Phase 2: Coordination mechanism (Future enhancement)

Implement mutual exclusion or coordination between Baileys retry and health monitor.

Current Workaround

Manual gateway restart every ~7 hours (not sustainable).

Impact

  • Critical for WhatsApp reliability
  • Affects cron jobs (Ryan pep talk, portfolio briefs, etc.)
  • Only affects isolated sessions (main sessions work)

Related Issues

Steps to Reproduce

  1. Run OpenClaw WhatsApp gateway
  2. Wait ~3-4 hours for race condition to occur
  3. Observe complete disconnection lasting ~7 hours
  4. Manual restart required

Technical Details

  • OpenClaw version: [current version]
  • Baileys library: [version from package.json]
  • Node.js version: [current version]
  • WhatsApp Web client: Platform-dependent

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions