Healthcheck for zero ingress connection count by yacovm · Pull Request #3719 · ava-labs/avalanchego

yacovm · 2025-02-13T20:00:16Z

Why this should be merged

This PR adds a new health check that monitors if the node is a primary network validator and has no node connecting to it.

Our current health checks only monitor whether the node is connected to enough stake. However, a node may be connected to enough stake by opening egress connections to all other validators, but also at the same time, not be reachable from any validator or non-validator node.

This health check does just that - it alerts when:

The node is a primary network validator.
The node has zero ingress connections to it.
More than 10 minutes (configurable) passed since its startup.

How this works

A node can establish a connection through two pathways:

It initiates a connection with a remote node
A remote node initiates connection to it.

We count the number of connections of the latter type via an atomic counter, and add a health check that checks the three conditions mentioned earlier.

How this was tested

Added unit tests and also did a manual test:

I launched a validator node on Fuji (testnet) and I added it to the validators:

03-03|17:34:09.380] INFO <P Chain> validators/logger.go:50 node added to validator set {"subnetID": "11111111111111111111111111111111LpoYY", "nodeID": "NodeID-49DuEYJ8Kub8pdHz6yucyVqKBhEw5urMZ", "publicKey": "0x80f073c922fbfe9fe8ee22a5c4d47f8b1c30d6c95574d7f5fb9d7d673f5170141751d51b2c9e6ca0a1ca2f2acde2477b", "txID": "2XuGeB5LeKcrR5N2yjPvWMJj19NVE7kTiLk6gzU437sL9qjDZN", "weight": 2991109255}

~/avalanchego$ curl -s  -H 'Content-Type: application/json' --data '{
    "jsonrpc":"2.0",
    "id"     :1,
    "method" :"health.health",
    "params": {
        "tags": ["11111111111111111111111111111111LpoYY", "29uVeLPJB1eQJkzRemU8g8wZDw5uJRqpab5U2mX9euieVwiEbL"]
    }
}' 'http://localhost:9650/ext/health' | jq '.result.checks.network.message["primary network validator health"]'
{
  "ingressConnectionCount": 1708,
  "primary network validator": true
}

I then caused all other nodes to disconnect from it by blocking port 9651:

sudo iptables -I INPUT -p tcp --dport 9651 -j DROP

After a short time, the health check started failing:

[03-03|19:08:12.853] INFO health/worker.go:261 check started passing {"name": "health", "name": "bootstrapped", "tags": ["application"]}
[03-03|19:08:12.853] INFO health/worker.go:261 check started passing {"name": "readiness", "name": "bootstrapped", "tags": ["application"]}
[03-03|19:10:42.853] WARN health/worker.go:252 check started failing {"name": "health", "name": "network", "tags": ["application"], "error": "network layer is unhealthy reason: primary network validator is unreachable"}

and probing the health check API showed it:

~/avalanchego$ curl -s  -H 'Content-Type: application/json' --data '{
    "jsonrpc":"2.0",
    "id"     :1,
    "method" :"health.health",
    "params": {
        "tags": ["11111111111111111111111111111111LpoYY", "29uVeLPJB1eQJkzRemU8g8wZDw5uJRqpab5U2mX9euieVwiEbL"]
    }
}' 'http://localhost:9650/ext/health' | jq '.result.checks.network'
{
  "message": {
    "connectedPeers": 227,
    "primary network validator health": {
      "ingressConnectionCount": 0,
      "primary network validator": true
    },
    "sendFailRate": 0,
    "timeSinceLastMsgReceived": "853.896432ms",
    "timeSinceLastMsgSent": "853.896432ms"
  },
  "error": "network layer is unhealthy reason: primary network validator is unreachable",
  "timestamp": "2025-03-03T19:11:12.853915018Z",
  "duration": 25832,
  "contiguousFailures": 2,
  "timeOfFirstFailure": "2025-03-03T19:10:42.853036554Z"
}

I then restored connectivity via deleting the iptables rule:

sudo iptables -D INPUT -p tcp --dport 9651 -j DROP

and observed that the health check recovered:

[03-03|19:12:12.853] INFO health/worker.go:261 check started passing {"name": "health", "name": "network", "tags": ["application"]}

~/avalanchego$ curl -s  -H 'Content-Type: application/json' --data '{
    "jsonrpc":"2.0",
    "id"     :1,
    "method" :"health.health",
    "params": {
        "tags": ["11111111111111111111111111111111LpoYY", "29uVeLPJB1eQJkzRemU8g8wZDw5uJRqpab5U2mX9euieVwiEbL"]
    }
}' 'http://localhost:9650/ext/health' | jq '.result.healthy'
true

Need to be documented in RELEASES.md?

Added a section about the health check added.

tsachiherman

Overall, looks great. I've added few comments.

yacovm · 2025-02-18T19:31:20Z

Overall, looks great. I've added few comments.

Thanks, addressed/responded to your comments.

tsachiherman · 2025-02-18T19:56:28Z

Thanks. Will re-review shortly.

…

On Tue, Feb 18, 2025, 2:31 PM yacovm ***@***.***> wrote: Overall, looks great. I've added few comments. Thanks, addressed/responded to your comments. — Reply to this email directly, view it on GitHub <#3719 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AF2OOHY34J6BCLZ6OD4ZA532QODB7AVCNFSM6AAAAABXDA2QMCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMNRWG42DSNJRG4> . You are receiving this because you commented.Message ID: ***@***.***> [image: yacovm]*yacovm* left a comment (ava-labs/avalanchego#3719) <#3719 (comment)> Overall, looks great. I've added few comments. Thanks, addressed/responded to your comments. — Reply to this email directly, view it on GitHub <#3719 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AF2OOHY34J6BCLZ6OD4ZA532QODB7AVCNFSM6AAAAABXDA2QMCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMNRWG42DSNJRG4> . You are receiving this because you commented.Message ID: ***@***.***>

yacovm · 2025-03-02T20:25:22Z

Parking this as a draft before I manually test it again after this code change

yacovm · 2025-03-03T19:38:33Z

@StephenButtolph the PR is ready again for re-review.

This commit adds a healthcheck that fails if: - The node is a validator of the primary network - The node has zero ingress connections - Enough time (defaults to 20 min) has passed since the node finished bootstrapping Signed-off-by: Yacov Manevich <[email protected]>

Signed-off-by: Yacov Manevich <[email protected]>

yacovm requested a review from StephenButtolph as a code owner February 13, 2025 20:00

yacovm marked this pull request as draft February 13, 2025 20:00

yacovm force-pushed the alertIfNoConnections branch 13 times, most recently from a5a21ff to b8ad75a Compare February 18, 2025 15:42

yacovm marked this pull request as ready for review February 18, 2025 16:26

yacovm force-pushed the alertIfNoConnections branch from de6b79f to d7fdf19 Compare February 18, 2025 16:32

yacovm requested a review from meaghanfitzgerald as a code owner February 18, 2025 16:32

yacovm changed the title ~~alert if validator connections drop quickly~~ Healthcheck for zero ingress connection count Feb 18, 2025

tsachiherman reviewed Feb 18, 2025

View reviewed changes

Comment thread network/network.go Outdated

Comment thread network/network.go Outdated

Comment thread network/network_test.go Outdated

Comment thread network/network_test.go Outdated

Comment thread node/no_ingress_conn_alert.go Outdated

tsachiherman approved these changes Feb 19, 2025

View reviewed changes

ceyonur reviewed Feb 24, 2025

View reviewed changes

Comment thread node/node.go Outdated

Comment thread node/no_ingress_conn_alert.go Outdated

Comment thread node/no_ingress_conn_alert.go Outdated

maru-ava approved these changes Feb 26, 2025

View reviewed changes

yacovm force-pushed the alertIfNoConnections branch from cd6d014 to a52673e Compare February 27, 2025 00:50

maru-ava approved these changes Feb 27, 2025

View reviewed changes

StephenButtolph reviewed Mar 2, 2025

View reviewed changes

Comment thread config/keys.go Outdated

Comment thread node/node.go Outdated

Comment thread node/no_ingress_conn_alert.go Outdated

Comment thread network/network.go Outdated

Comment thread network/network.go Outdated

yacovm force-pushed the alertIfNoConnections branch from d820099 to 2005233 Compare March 2, 2025 20:20

yacovm marked this pull request as draft March 2, 2025 20:20

yacovm force-pushed the alertIfNoConnections branch 11 times, most recently from 4169d09 to 1cf5056 Compare March 3, 2025 19:15

yacovm marked this pull request as ready for review March 3, 2025 19:24

yacovm force-pushed the alertIfNoConnections branch from ff2344e to d1b4d92 Compare March 10, 2025 18:33

StephenButtolph reviewed Mar 11, 2025

View reviewed changes

yacovm added 5 commits March 12, 2025 00:08

Address code review comments

9367400

Signed-off-by: Yacov Manevich <[email protected]>

Address code review comments II

b3414a8

Signed-off-by: Yacov Manevich <[email protected]>

Address code review comments III

852c961

Signed-off-by: Yacov Manevich <[email protected]>

Address code review comments IV

9c9ca56

Signed-off-by: Yacov Manevich <[email protected]>

yacovm force-pushed the alertIfNoConnections branch from cf8c3d6 to 9c9ca56 Compare March 11, 2025 23:08

StephenButtolph reviewed Mar 12, 2025

View reviewed changes

Comment thread utils/constants/networking.go Outdated

Address code review comments V

23850fd

Signed-off-by: Yacov Manevich <[email protected]>

StephenButtolph approved these changes Mar 12, 2025

View reviewed changes

StephenButtolph added this pull request to the merge queue Mar 12, 2025

Merged via the queue into ava-labs:master with commit a83e692 Mar 12, 2025
23 checks passed

cam-schultz pushed a commit that referenced this pull request Mar 24, 2025

Healthcheck for zero ingress connection count (#3719)

957fc15

Signed-off-by: Yacov Manevich <[email protected]>

Conversation

yacovm commented Feb 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why this should be merged

How this works

How this was tested

Need to be documented in RELEASES.md?

Uh oh!

tsachiherman left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yacovm commented Feb 18, 2025

Uh oh!

tsachiherman commented Feb 18, 2025 via email

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yacovm commented Mar 2, 2025

Uh oh!

yacovm commented Mar 3, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

yacovm commented Feb 13, 2025 •

edited

Loading