[Merged by Bors] - Add more checks and logging before genesis #4730

michaelsproul · 2023-09-13T06:32:45Z

Proposed Changes

This PR adds more logging prior to genesis, particularly on networks that start with execution enabled.

There are new checks using eth_getBlockByHash/Number to verify that the genesis state's latest_execution_payload_header matches the execution node's genesis block.

The first commit also runs the merge-readiness/Capella-readiness checks prior to genesis. This has two effects:

Give more information on the execution node's status and its readiness for genesis.
Prevent the el_offline status from being set on /eth/v1/node/syncing, which previously caused the VC to complain loudly.

I would like to include this for the Holesky reboot. It would have caught the misconfig that doomed the first Holesky.

Additional Info

Geth doesn't serve payload bodies prior to genesis, which is why we use the legacy methods. I haven't checked with other ELs yet.
Currently this is logging errors with Capella genesis states generated by ethereum-genesis-generator because the withdrawals_root is not set correctly (it is 0x0). This is not a blocker for Holesky, as it starts from Bellatrix (Pari is investigating).

michaelsproul · 2023-09-18T04:24:23Z

Here's an example of the withdrawals_root being set to 0x0 by ethereum-genesis-generator:

$ curl -s "http://localhost:5052/eth/v2/debug/beacon/states/genesis" | jq .data.latest_execution_payload_header

{
  "parent_hash": "0x0000000000000000000000000000000000000000000000000000000000000000",
  "fee_recipient": "0x0000000000000000000000000000000000000000",
  "state_root": "0xac7e550c93c231f313b70a7a2fcf8b80119427f8d9c2864fd2d111ace5fc47e2",
  "receipts_root": "0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421",
  "logs_bloom": "0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000",
  "prev_randao": "0x0000000000000000000000000000000000000000000000000000000000000000",
  "block_number": "0",
  "gas_limit": "25000000",
  "gas_used": "0",
  "timestamp": "1695902400",
  "extra_data": "0x",
  "base_fee_per_gas": "1000000000",
  "block_hash": "0x03e5a32cb85ea81f536800390b9dba27a1cde3e705f98d0791d5fbc93f942118",
  "transactions_root": "0x7ffe241ea60187fdb0187bfa22de35d1f9bed7ab061d9401fd47e34a54fbede1",
  "withdrawals_root": "0x0000000000000000000000000000000000000000000000000000000000000000"
}

Lighthouse logs:

Sep 18 04:23:42.436 ERRO Genesis payload withdrawals root mismatch, execution_node_withdrawals_root: 0x792930bbd5baac43bcc798ee49aa8185ef76bb3b44ba62b91d86ae569e4bb535, consensus_node_withdrawals_root: 0x0000000000000000000000000000000000000000000000000000000000000000, info: genesis is misconfigured and likely to fail, service: slot_notifier

michaelsproul · 2023-09-19T00:02:47Z

Have tested this with the new proposed Holesky genesis from eth-clients/holesky#73 and it works correctly:

Sep 19 00:02:36.274 INFO Execution enabled from genesis genesis_payload_block_hash: 0xb5f7f912443c940f21fd611f12828d75b534364ed9e95ca4e307729a4661bde4, service: slot_notifier

paulhauner

Nice! I couldn't fault it.

michaelsproul · 2023-09-20T23:50:04Z

bors r+

## Proposed Changes This PR adds more logging prior to genesis, particularly on networks that start with execution enabled. There are new checks using `eth_getBlockByHash/Number` to verify that the genesis state's `latest_execution_payload_header` matches the execution node's genesis block. The first commit also runs the merge-readiness/Capella-readiness checks prior to genesis. This has two effects: - Give more information on the execution node's status and its readiness for genesis. - Prevent the `el_offline` status from being set on `/eth/v1/node/syncing`, which previously caused the VC to complain loudly. I would like to include this for the Holesky reboot. It would have caught the misconfig that doomed the first Holesky. ## Additional Info - Geth doesn't serve payload bodies prior to genesis, which is why we use the legacy methods. I haven't checked with other ELs yet. - Currently this is logging errors with _Capella_ genesis states generated by `ethereum-genesis-generator` because the `withdrawals_root` is not set correctly (it is 0x0). This is not a blocker for Holesky, as it starts from Bellatrix (Pari is investigating).

bors · 2023-09-20T23:56:59Z

Build failed (retrying...):

release-tests-ubuntu

## Proposed Changes This PR adds more logging prior to genesis, particularly on networks that start with execution enabled. There are new checks using `eth_getBlockByHash/Number` to verify that the genesis state's `latest_execution_payload_header` matches the execution node's genesis block. The first commit also runs the merge-readiness/Capella-readiness checks prior to genesis. This has two effects: - Give more information on the execution node's status and its readiness for genesis. - Prevent the `el_offline` status from being set on `/eth/v1/node/syncing`, which previously caused the VC to complain loudly. I would like to include this for the Holesky reboot. It would have caught the misconfig that doomed the first Holesky. ## Additional Info - Geth doesn't serve payload bodies prior to genesis, which is why we use the legacy methods. I haven't checked with other ELs yet. - Currently this is logging errors with _Capella_ genesis states generated by `ethereum-genesis-generator` because the `withdrawals_root` is not set correctly (it is 0x0). This is not a blocker for Holesky, as it starts from Bellatrix (Pari is investigating).

bors · 2023-09-21T00:21:27Z

Build failed (retrying...):

release-tests-ubuntu

michaelsproul · 2023-09-21T00:26:27Z

bors r-
bors r+

bors · 2023-09-21T00:26:34Z

Canceled.

## Proposed Changes This PR adds more logging prior to genesis, particularly on networks that start with execution enabled. There are new checks using `eth_getBlockByHash/Number` to verify that the genesis state's `latest_execution_payload_header` matches the execution node's genesis block. The first commit also runs the merge-readiness/Capella-readiness checks prior to genesis. This has two effects: - Give more information on the execution node's status and its readiness for genesis. - Prevent the `el_offline` status from being set on `/eth/v1/node/syncing`, which previously caused the VC to complain loudly. I would like to include this for the Holesky reboot. It would have caught the misconfig that doomed the first Holesky. ## Additional Info - Geth doesn't serve payload bodies prior to genesis, which is why we use the legacy methods. I haven't checked with other ELs yet. - Currently this is logging errors with _Capella_ genesis states generated by `ethereum-genesis-generator` because the `withdrawals_root` is not set correctly (it is 0x0). This is not a blocker for Holesky, as it starts from Bellatrix (Pari is investigating).

bors · 2023-09-21T01:32:50Z

Pull request successfully merged into unstable.

Build succeeded!

The publicly hosted instance of bors-ng is deprecated and will go away soon.

If you want to self-host your own instance, instructions are here.
For more help, visit the forum.

If you want to switch to GitHub's built-in merge queue, visit their help page.

## Proposed Changes This PR adds more logging prior to genesis, particularly on networks that start with execution enabled. There are new checks using `eth_getBlockByHash/Number` to verify that the genesis state's `latest_execution_payload_header` matches the execution node's genesis block. The first commit also runs the merge-readiness/Capella-readiness checks prior to genesis. This has two effects: - Give more information on the execution node's status and its readiness for genesis. - Prevent the `el_offline` status from being set on `/eth/v1/node/syncing`, which previously caused the VC to complain loudly. I would like to include this for the Holesky reboot. It would have caught the misconfig that doomed the first Holesky. ## Additional Info - Geth doesn't serve payload bodies prior to genesis, which is why we use the legacy methods. I haven't checked with other ELs yet. - Currently this is logging errors with _Capella_ genesis states generated by `ethereum-genesis-generator` because the `withdrawals_root` is not set correctly (it is 0x0). This is not a blocker for Holesky, as it starts from Bellatrix (Pari is investigating).

This PR adds more logging prior to genesis, particularly on networks that start with execution enabled. There are new checks using `eth_getBlockByHash/Number` to verify that the genesis state's `latest_execution_payload_header` matches the execution node's genesis block. The first commit also runs the merge-readiness/Capella-readiness checks prior to genesis. This has two effects: - Give more information on the execution node's status and its readiness for genesis. - Prevent the `el_offline` status from being set on `/eth/v1/node/syncing`, which previously caused the VC to complain loudly. I would like to include this for the Holesky reboot. It would have caught the misconfig that doomed the first Holesky. - Geth doesn't serve payload bodies prior to genesis, which is why we use the legacy methods. I haven't checked with other ELs yet. - Currently this is logging errors with _Capella_ genesis states generated by `ethereum-genesis-generator` because the `withdrawals_root` is not set correctly (it is 0x0). This is not a blocker for Holesky, as it starts from Bellatrix (Pari is investigating).

Log more info before genesis

8f0b5eb

michaelsproul added ready-for-review The code is ready for review low-hanging-fruit Easy to resolve, get it before someone else does! UX-and-logs labels Sep 13, 2023

michaelsproul mentioned this pull request Sep 13, 2023

[Merged by Bors] - validator client: start http api before genesis #4714

Closed

Verify EL genesis payload

7e0f969

michaelsproul changed the title ~~Log more info before genesis~~ Add more checks and logging before genesis Sep 18, 2023

michaelsproul added the v4.5.0 ETA Q4 2023 label Sep 18, 2023

michaelsproul requested a review from paulhauner September 19, 2023 00:01

paulhauner approved these changes Sep 19, 2023

View reviewed changes

michaelsproul added ready-for-merge This PR is ready to merge. and removed ready-for-review The code is ready for review low-hanging-fruit Easy to resolve, get it before someone else does! labels Sep 20, 2023

bors bot changed the title ~~Add more checks and logging before genesis~~ [Merged by Bors] - Add more checks and logging before genesis Sep 21, 2023

bors bot closed this Sep 21, 2023

michaelsproul deleted the pre-genesis-logging branch September 21, 2023 01:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Merged by Bors] - Add more checks and logging before genesis #4730

[Merged by Bors] - Add more checks and logging before genesis #4730

Uh oh!

michaelsproul commented Sep 13, 2023 •

edited

Loading

Uh oh!

michaelsproul commented Sep 18, 2023

Uh oh!

michaelsproul commented Sep 19, 2023

Uh oh!

paulhauner left a comment

Uh oh!

michaelsproul commented Sep 20, 2023

Uh oh!

bors bot commented Sep 20, 2023

Uh oh!

bors bot commented Sep 21, 2023

Uh oh!

michaelsproul commented Sep 21, 2023

Uh oh!

bors bot commented Sep 21, 2023

Uh oh!

bors bot commented Sep 21, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

[Merged by Bors] - Add more checks and logging before genesis #4730

[Merged by Bors] - Add more checks and logging before genesis #4730

Uh oh!

Conversation

michaelsproul commented Sep 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed Changes

Additional Info

Uh oh!

michaelsproul commented Sep 18, 2023

Uh oh!

michaelsproul commented Sep 19, 2023

Uh oh!

paulhauner left a comment

Choose a reason for hiding this comment

Uh oh!

michaelsproul commented Sep 20, 2023

Uh oh!

bors bot commented Sep 20, 2023

Uh oh!

bors bot commented Sep 21, 2023

Uh oh!

michaelsproul commented Sep 21, 2023

Uh oh!

bors bot commented Sep 21, 2023

Uh oh!

bors bot commented Sep 21, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

michaelsproul commented Sep 13, 2023 •

edited

Loading