Refactor get_validator_blocks_v3 fallback #8186

chong-he · 2025-10-10T03:51:26Z

Issue Addressed

#7727 introduced a bug in the logging, where as long as the node failed the SSZ get_validator_blocks_v3 endpoint, it would log as Beacon node does not support.... However, the failure can be due to other reasons, such as a timed out error as found by @jimmygchen:

WARN Beacon node does not support SSZ in block production, falling back to JSON slot: 5283379, error: HttpClient(url: https://ho-h-bn-cowl.spesi.io:15052/, kind: timeout, detail: operation timed out

Proposed Changes

This PR made the error log more generic, so there is less confusion.

Additionally, suggested by @michaelsproul, this PR refactors the get_validator_blocks_v3 calls by trying all beacon nodes using the SSZ endpoint first, and if all beacon node fails the SSZ endpoint, only then fallback to JSON.

It changes the logic from:
"SSZ -> JSON for primary beacon node, followed by SSZ -> JSON for second beacon node and so on" to
"SSZ for all beacon nodes -> JSON for all beacon nodes"

This has the advantage that if the primary beacon node is having issues and failed the SSZ, we avoid retrying the primary beacon node again on JSON (as it could be that the primary beacon node fail again); rather, we switch to the second beacon node.

Additional Info

As the calling of get_validator_blocks_v3 (and it's SSZ counterpart) is shifted to another function, the remaining part in the function get_validator_block is a bit of dangling in the air, so I moved them all to the now renamed get_validator_block_and_publish_block function and delete the original get_validator_block function.

macladson

Looks good, just a small nit

It's a shame we can't detect the type of error first:

Connection refused (node offline)
SSZ not supported
Then we if get 2 we can immediately try JSON rather than moving onto the second BN.
But we would have to parse the error type from the API which probably isn't doable.

At this point, pretty much all BNs should be supporting SSZ by default so this should be fine

validator_client/validator_services/src/block_service.rs

macladson

Just a slight edit for clarity

validator_client/validator_services/src/block_service.rs

Co-authored-by: Mac L <[email protected]>

macladson · 2025-11-17T06:45:56Z

Just needs a cargo fmt for some issue with whitespaces, otherwise looks pretty good to me!

chong-he · 2025-11-18T10:27:39Z

Just needs a cargo fmt for some issue with whitespaces, otherwise looks pretty good to me!

sorry I overlook the checks, fixed now

mergify · 2025-11-27T04:16:59Z

Merge Queue Status

🚫 The pull request has left the queue (rule: default)

This pull request spent 1 hour 4 minutes 56 seconds in the queue, including 31 minutes 52 seconds waiting for CI.
The checks were run on draft #8476.

Required conditions to merge

check-success=local-testnet-success
check-success=test-suite-success

Reason

Pull request #8186 has been dequeued. The pull request could not be merged. This could be related to an activated branch protection or ruleset rule that prevents us from merging. (details: 1 review requesting changes and 1 approving review by reviewers with write access.)

Hint

You should look at the reason for the failure and decide if the pull request needs to be fixed or if you want to requeue it.
If you do update this pull request, it will automatically be requeued once the queue conditions match again.
If you think this was a flaky issue instead, you can requeue the pull request, without updating it, by posting a @mergifyio requeue comment.

mergify · 2025-12-01T01:25:44Z

Merge Queue Status

🚫 The pull request has left the queue (rule: default)

This pull request spent 37 minutes 49 seconds in the queue, including 36 minutes 15 seconds waiting for CI.
The checks were run on draft #8504.

Required conditions to merge

check-success=local-testnet-success
check-success=test-suite-success

Reason

Pull request #8186 has been dequeued. The pull request could not be merged. This could be related to an activated branch protection or ruleset rule that prevents us from merging. (details: 1 review requesting changes and 1 approving review by reviewers with write access.)

Hint

You should look at the reason for the failure and decide if the pull request needs to be fixed or if you want to requeue it.
If you do update this pull request, it will automatically be requeued once the queue conditions match again.
If you think this was a flaky issue instead, you can requeue the pull request, without updating it, by posting a @mergifyio requeue comment.

michaelsproul · 2025-12-01T03:59:38Z

@mergify requeue

mergify · 2025-12-01T03:59:54Z

requeue

✅ The queue state of this pull request has been cleaned. It can be re-embarked automatically

mergify · 2025-12-01T04:00:17Z

Merge Queue Status

🚫 The pull request has left the queue (rule: default)

This pull request spent 33 minutes 30 seconds in the queue, including 31 minutes 46 seconds waiting for CI.
The checks were run on draft #8506.

Required conditions to merge

check-success=local-testnet-success
check-success=test-suite-success

Reason

Pull request #8186 has been dequeued. The pull request could not be merged. This could be related to an activated branch protection or ruleset rule that prevents us from merging. (details: 1 review requesting changes and 1 approving review by reviewers with write access.)

Hint

You should look at the reason for the failure and decide if the pull request needs to be fixed or if you want to requeue it.
If you do update this pull request, it will automatically be requeued once the queue conditions match again.
If you think this was a flaky issue instead, you can requeue the pull request, without updating it, by posting a @mergifyio requeue comment.

stale

michaelsproul · 2025-12-01T05:23:34Z

@mergify requeue

mergify · 2025-12-01T05:23:50Z

requeue

✅ The queue state of this pull request has been cleaned. It can be re-embarked automatically

mergify · 2025-12-01T05:23:51Z

Merge Queue Status

✅ The pull request has been merged

This pull request spent 33 minutes in the queue, including 31 minutes 39 seconds waiting for CI.
The checks were run on draft #8507.

Required conditions to merge

check-success=local-testnet-success
check-success=test-suite-success

chong-he added 2 commits October 10, 2025 11:17

refactor get validator fallback

ed80ebe

remove function

70827e0

chong-he added val-client Relates to the validator client binary ready-for-review The code is ready for review labels Oct 10, 2025

michaelsproul added the v8.1.0 Post-Fulu release label Oct 29, 2025

michaelsproul requested a review from macladson November 4, 2025 05:30

macladson requested changes Nov 13, 2025

View reviewed changes

validator_client/validator_services/src/block_service.rs Show resolved Hide resolved

macladson added waiting-on-author The reviewer has suggested changes and awaits thier implementation. and removed ready-for-review The code is ready for review labels Nov 13, 2025

chong-he and others added 2 commits November 14, 2025 07:52

update comment

090e4c3

Merge branch 'unstable' into bn-ssz-error

be309c8

chong-he added ready-for-review The code is ready for review and removed waiting-on-author The reviewer has suggested changes and awaits thier implementation. labels Nov 14, 2025

macladson previously requested changes Nov 14, 2025

View reviewed changes

validator_client/validator_services/src/block_service.rs Outdated Show resolved Hide resolved

validator_client/validator_services/src/block_service.rs Outdated Show resolved Hide resolved

macladson removed the ready-for-review The code is ready for review label Nov 14, 2025

chong-he and others added 2 commits November 15, 2025 05:56

Update validator_client/validator_services/src/block_service.rs

048156a

Co-authored-by: Mac L <[email protected]>

Update validator_client/validator_services/src/block_service.rs

45f0681

Co-authored-by: Mac L <[email protected]>

cargo fmt

067131b

michaelsproul added the ready-for-review The code is ready for review label Nov 27, 2025

michaelsproul approved these changes Nov 27, 2025

View reviewed changes

michaelsproul added ready-for-merge This PR is ready to merge. and removed ready-for-review The code is ready for review labels Nov 27, 2025

mergify bot added the queued label Nov 27, 2025

mergify bot added a commit that referenced this pull request Nov 27, 2025

Merge of #8186

31d1271

mergify bot mentioned this pull request Nov 27, 2025

merge queue: embarking unstable (e21a433), #8399 and #8186 together #8476

Closed

6 tasks

mergify bot added a commit that referenced this pull request Nov 27, 2025

Merge of #8186

21db38c

mergify bot added a commit that referenced this pull request Nov 27, 2025

Merge of #8186

7aa3c3a

mergify bot added dequeued and removed queued labels Dec 1, 2025

corrections

8003c2c

mergify bot removed the dequeued label Dec 1, 2025

mergify bot added the queued label Dec 1, 2025

mergify bot added a commit that referenced this pull request Dec 1, 2025

Merge of #8186

caaf21b

mergify bot mentioned this pull request Dec 1, 2025

merge queue: embarking unstable (64031b6) and #8186 together #8504

Closed

6 tasks

mergify bot added dequeued and removed queued labels Dec 1, 2025

mergify bot added queued and removed dequeued labels Dec 1, 2025

mergify bot added a commit that referenced this pull request Dec 1, 2025

Merge of #8186

693be0f

mergify bot mentioned this pull request Dec 1, 2025

merge queue: embarking unstable (64031b6) and #8186 together #8506

Closed

6 tasks

mergify bot added dequeued and removed queued labels Dec 1, 2025

mergify bot added queued and removed dequeued labels Dec 1, 2025

mergify bot added a commit that referenced this pull request Dec 1, 2025

Merge of #8186

a643ad8

mergify bot mentioned this pull request Dec 1, 2025

merge queue: embarking unstable (64031b6) and #8186 together #8507

Closed

6 tasks

mergify bot merged commit 90dd5bb into sigp:unstable Dec 1, 2025
36 checks passed

mergify bot removed the queued label Dec 1, 2025

chong-he deleted the bn-ssz-error branch December 17, 2025 02:22

Refactor get_validator_blocks_v3 fallback #8186

Refactor get_validator_blocks_v3 fallback #8186

Conversation

chong-he commented Oct 10, 2025

Issue Addressed

Proposed Changes

Additional Info

Uh oh!

macladson left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

macladson left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

macladson commented Nov 17, 2025

Uh oh!

chong-he commented Nov 18, 2025

Uh oh!

mergify bot commented Nov 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge Queue Status

Reason

Hint

Uh oh!

mergify bot commented Dec 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge Queue Status

Reason

Hint

Uh oh!

michaelsproul commented Dec 1, 2025

Uh oh!

mergify bot commented Dec 1, 2025

✅ The queue state of this pull request has been cleaned. It can be re-embarked automatically

Uh oh!

mergify bot commented Dec 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge Queue Status

Reason

Hint

Uh oh!

michaelsproul commented Dec 1, 2025

Uh oh!

mergify bot commented Dec 1, 2025

✅ The queue state of this pull request has been cleaned. It can be re-embarked automatically

Uh oh!

mergify bot commented Dec 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge Queue Status

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mergify bot commented Nov 27, 2025 •

edited

Loading

mergify bot commented Dec 1, 2025 •

edited

Loading

mergify bot commented Dec 1, 2025 •

edited

Loading

mergify bot commented Dec 1, 2025 •

edited

Loading