Description

The wait-for-charon.sh script creates a deadlock where the Lodestar validator client never starts.

Root Cause

In cluster/scripts/lodestar/wait-for-charon.sh line 7:

status_code=$(curl -s -o /dev/null -w "%{http_code}" http://localhost:3620/readyz)

The /readyz endpoint returns HTTP 500 with body vc not connected until a validator client connects. But the VC startup is blocked by this script waiting for /readyz to return 200.

Result: Permanent deadlock. VC never starts.

Evidence

  $ docker exec cluster-1 curl -s http://localhost:3620/readyz
  vc not connected

  $ docker exec cluster-1 curl -s http://localhost:3620/livez
  ok

  Container logs show infinite loop:
  [ INFO | lodestar-wait ] Waiting for Charon to be ready before launching lodestar...
  [ INFO | lodestar-wait ] Waiting for Charon to be ready before launching lodestar...
  ...

Proposed Fix

Change line 7 of cluster/scripts/lodestar/wait-for-charon.sh to use /livez:

• status_code=$(curl -s -o /dev/null -w "%{http_code}" http://localhost:3620/readyz) [old]

• status_code=$(curl -s -o /dev/null -w "%{http_code}" http://localhost:3620/livez) [new]

Rationale:

• /livez confirms Charon is running and responsive
• /readyz is meant for load balancer health checks (don't route traffic if no VC), not for VC startup gating
• The validator API (port 3600) is available immediately after Charon starts

Environment

• Charon version: v1.7.1
• Affected: All 5 clusters

Core DAppNode Packages versions

• bind.dnp.dappnode.eth: 0.2.12
• dappmanager.dnp.dappnode.eth: 0.2.119, commit: 79cee310
• https.dnp.dappnode.eth: 0.2.2
• ipfs.dnp.dappnode.eth: 0.2.27
• notifications.dnp.dappnode.eth: 0.1.8
• premium.dnp.dappnode.eth: 0.1.0
• vpn.dnp.dappnode.eth:
• wireguard.dnp.dappnode.eth: 0.1.3
  System info
• dockerComposeVersion: 5.0.2
• dockerServerVersion: 29.1.5
• dockerCliVersion: 29.1.5
• os: debian
• versionCodename: bookworm
• architecture: amd64
• kernel: 6.1.0-39-amd64
• Disk usage: 50%