Skip to content

Docker in-process gateway restart can leave command queue draining while healthz/readyz report OK #78136

Description

@maxschachere

Summary

On a Docker install, an in-process gateway restart appears able to leave the command queue in the gatewayDraining state after the gateway has restarted and logged ready. During this state, model calls are rejected with Gateway is draining for restart; new tasks are not accepted, while Docker health and OpenClaw readiness endpoints still report healthy/ready.

This looks related to restart/drain behavior in #43178, but this report is narrower: the gateway can be ready from a health/readiness perspective while still rejecting new model work because the internal drain flag is stuck.

Environment

  • OpenClaw: 2026.5.4
  • Image revision: 325df3efefe9c0887d9357732e68fc8556e78d79
  • Image: locally built from ghcr.io/openclaw/openclaw:latest
  • Install method: Docker Compose
  • Host: Raspberry Pi / Linux arm64
  • Kernel: Linux raspberrypi 6.12.62+rpt-rpi-2712 ... aarch64 GNU/Linux
  • Node inside image: 24.14.0
  • Gateway container: openclaw-gateway
  • Docker healthcheck uses /healthz

The compose file is a hardened Docker setup, but it does not set OPENCLAW_NO_RESPAWN. Verified from Docker config and from the actual gateway process environment.

What happened

A config change required a gateway restart. Separately, the gateway restart tool also requested a restart. After active work drained, two SIGUSR1 restarts were handled back-to-back. The gateway then logged ready, but subsequent model calls continued to fail with Gateway is draining for restart; new tasks are not accepted.

Sanitized log sequence:

2026-05-05T23:25:31.686+00:00 [reload] config change requires gateway restart (plugins.installs.codex) — deferring until 2 operation(s), 1 reply(ies), 1 embedded run(s) complete
2026-05-05T23:25:39.964+00:00 [gateway-tool] gateway tool: restart requested (delayMs=default, reason=Apply approved conservative plugins.allow allowlist and bundledDiscovery=allowlist configuration.)
2026-05-05T23:28:57.692+00:00 [reload] all operations and replies completed; restarting gateway now
2026-05-05T23:28:57.694+00:00 [gateway] signal SIGUSR1 received
2026-05-05T23:28:57.705+00:00 [gateway] received SIGUSR1; restarting
2026-05-05T23:28:58.143+00:00 [gateway] restart mode: in-process restart (OPENCLAW_NO_RESPAWN)
2026-05-05T23:28:58.968+00:00 [gateway] signal SIGUSR1 received
2026-05-05T23:28:58.970+00:00 [gateway] received SIGUSR1; restarting
2026-05-05T23:28:58.974+00:00 [gateway] restart mode: in-process restart (OPENCLAW_NO_RESPAWN)
2026-05-05T23:28:59.699+00:00 [gateway] ready
2026-05-05T23:29:03.217+00:00 [model-fallback/decision] model fallback decision: decision=candidate_failed requested=openai-codex/gpt-5.5 candidate=openai-codex/gpt-5.5 reason=unknown next=openai-codex/gpt-5.4-mini detail=Gateway is draining for restart; new tasks are not accepted
...
2026-05-05T23:50:01.419+00:00 [model-fallback/decision] model fallback decision: decision=candidate_failed requested=openai-codex/gpt-5.4-mini candidate=openai-codex/gpt-5.1-codex-max reason=unknown next=openrouter/openai/gpt-5-nano detail=Gateway is draining for restart; new tasks are not accepted

Note: the log says in-process restart (OPENCLAW_NO_RESPAWN), but OPENCLAW_NO_RESPAWN was not present in the container config or in /proc/<gateway-pid>/environ. This seems to be the generic in-process fallback label. The installed code also appears to intentionally disable fresh process respawn in container environments.

Health/readiness mismatch

While the gateway was still rejecting model work with Gateway is draining for restart, the container stayed healthy and the endpoints reported OK:

/healthz -> 200 {"ok":true,"status":"live"}
/readyz  -> 200 {"ready":true,"failing":[], ...}

Docker also showed the container as running/healthy, so the bad state was invisible to Docker health checks.

Expected behavior

After an in-process restart completes and the gateway logs ready, new model/tool work should be accepted again.

If the gateway is still intentionally draining/rejecting new work, /readyz should not return ready.

Duplicate/coalesced restart requests during restart/startup should not leave the command queue permanently draining.

Actual behavior

The gateway logged ready and readiness endpoints returned OK, but model calls continued to fail for at least ~20 minutes with:

Gateway is draining for restart; new tasks are not accepted

Workaround

A host-side Docker Compose recreate of only the gateway clears the bad in-memory state because it gives the gateway a fresh Node process.

Example workaround:

docker compose -f docker-compose.yml -f docker-compose.db.yml up -d --force-recreate openclaw-gateway

For now I am also considering disabling OpenClaw self-restart paths in Docker and using Compose as the only restart mechanism.

Suggested fixes

  • Ensure gatewayDraining is cleared reliably after every in-process restart path.
  • Coalesce or ignore duplicate restart signals while a restart iteration is already in progress.
  • Make /readyz fail when the gateway is rejecting new model/tool work due to drain state.
  • Consider documenting Docker installs as requiring supervisor/container-level restarts for restart-required config changes, or provide a Docker-aware restart mode that asks the host/supervisor to recreate the gateway container.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Normal backlog priority with limited blast radius.clawsweeper:fix-shape-clearClawSweeper found a clear likely implementation shape for this issue.clawsweeper:queueable-fixClawSweeper marked this issue as an existing queue_fix_pr work candidate.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.impact:otherThis issue has meaningful maintainer-visible impact outside the owned taxonomy.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.no-staleExclude from stale automation

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions