Prevent unnecessary state advances pre-Fulu #8388

michaelsproul · 2025-11-10T04:34:42Z

Issue Addressed

State advances were observed as especially slow on pre-Fulu networks (mainnet).

The reason being: we were doing an extra epoch of state advance because of code that should only have been running after Fulu, when proposer shufflings are determined with lookahead.

Proposed Changes

Only attempt to cache the next epoch shuffling if the state's slot determines it (this will only be true post-Fulu). Reusing the logic for proposer_shuffling_decision_slot avoids having to repeat the fiddly logic about the Fulu fork epoch itself.

winksaville · 2025-11-11T18:43:51Z

I'm seeing the "State advance too slow", after every attestation since I updated my mainnet node:

kendall@robert 25-11-11T18:29:17.892Z:~
$ esm.sh -e 20m ms beacon-chain
niceit journalctl -u beacon-chain.service --since -20m | rg -i "attestation.*(missing|success)|Starting|Started|Stopped|Stopping|Boot|version|Sync state updated|CRIT|OOM|ERR|WARN"
Nov 11 10:24:21 robert lighthouse[1109]: Nov 11 10:24:21.376 INFO  Previous epoch attestation(s) success         epoch: 406408, validators: ["203150", "198277", "198278", "208738", "426158", "208734", "203157", "203152", "208729", "426161", "426157", "280375", "203155", "426156", "208735", "426112", "426148", "208730", "426155", "208737", "426151", "203159", "426113", "426153", "208732", "203154", "280374", "208731", "198276", "426152", "426087", "426146", "203158", "426149", "426159", "426147", "195766", "203149", "208733", "426150", "203151", "198275", "426160", "426115", "280376", "203156", "426154", "208736"]
Nov 11 10:24:23 robert lighthouse[1109]: Nov 11 10:24:23.435 WARN  State advance too slow                        head_block_root: 0x90b5e1e52065655a01ec9a9142ac285d7b79e85f9b3b03843a3f7cfebc5088a2, advanced_slot: 13005120, current_slot: 13005120, starting_slot: 13005119, msg: "system resources may be overloaded"
Nov 11 10:30:45 robert lighthouse[1109]: Nov 11 10:30:45.390 INFO  Previous epoch attestation(s) success         epoch: 406409, validators: ["203150", "198277", "198278", "208738", "426158", "208734", "203157", "203152", "208729", "426161", "426157", "280375", "203155", "426156", "208735", "426112", "426148", "208730", "426155", "208737", "426151", "203159", "426113", "426153", "208732", "203154", "280374", "208731", "198276", "426152", "426087", "426146", "203158", "426149", "426159", "426147", "195766", "203149", "208733", "426150", "203151", "198275", "426160", "426115", "280376", "203156", "426154", "208736"]
Nov 11 10:30:47 robert lighthouse[1109]: Nov 11 10:30:47.392 WARN  State advance too slow                        head_block_root: 0x16f800873cb52169f0978153a5ae339e6a31e941cdd3ee0d7fee244531b8aa47, advanced_slot: 13005152, current_slot: 13005152, starting_slot: 13005151, msg: "system resources may be overloaded"
Nov 11 10:37:09 robert lighthouse[1109]: Nov 11 10:37:09.274 INFO  Previous epoch attestation(s) success         epoch: 406410, validators: ["203150", "198277", "198278", "208738", "426158", "208734", "203157", "203152", "208729", "426161", "426157", "280375", "203155", "426156", "208735", "426112", "426148", "208730", "426155", "208737", "426151", "203159", "426113", "426153", "208732", "203154", "280374", "208731", "198276", "426152", "426087", "426146", "203158", "426149", "426159", "426147", "195766", "203149", "208733", "426150", "203151", "198275", "426160", "426115", "280376", "203156", "426154", "208736"]
Nov 11 10:37:11 robert lighthouse[1109]: Nov 11 10:37:11.067 WARN  State advance too slow                        head_block_root: 0xe891ad6074f5698a616e9a0651bbf9ddaa6610a493490c319f33c4b5a3929688, advanced_slot: 13005184, current_slot: 13005184, starting_slot: 13005183, msg: "system resources may be overloaded"
kendall@robert 25-11-11T18:42:13.143Z:~

Here is the most recent

Nov 11 10:24:17 robert lighthouse[1109]: Nov 11 10:24:17.001 INFO  Synced                                        peers: "44", exec_hash: "0x60df76260ff6115fba997fa2d7663a1a5e5aa29df562f86ca68c9d13010bd797 (verified)", finalized_root: 0x9e8e24fe5b87e66343f9ab7542a36e11acb86f00949195f39005bd10d4f0cc39, finalized_epoch: 406407, epoch: 406409, block: "0x90b5e1e52065655a01ec9a9142ac285d7b79e85f9b3b03843a3f7cfebc5088a2", slot: 13005119
Nov 11 10:24:19 robert lighthouse[1109]: Nov 11 10:24:19.413 INFO  Attestation included in aggregate             head: 0x90b5e1e52065655a01ec9a9142ac285d7b79e85f9b3b03843a3f7cfebc5088a2, index: 0, delay_ms: 366, epoch: 406409, slot: 13005119, src: "gossip", validator: 208737
Nov 11 10:24:21 robert lighthouse[1109]: Nov 11 10:24:21.376 INFO  Previous epoch attestation(s) success         epoch: 406408, validators: ["203150", "198277", "198278", "208738", "426158", "208734", "203157", "203152", "208729", "426161", "426157", "280375", "203155", "426156", "208735", "426112", "426148", "208730", "426155", "208737", "426151", "203159", "426113", "426153", "208732", "203154", "280374", "208731", "198276", "426152", "426087", "426146", "203158", "426149", "426159", "426147", "195766", "203149", "208733", "426150", "203151", "198275", "426160", "426115", "280376", "203156", "426154", "208736"]
Nov 11 10:24:21 robert lighthouse[1109]: Nov 11 10:24:21.376 INFO  Previous epoch attestation(s) failed to match head  epoch: 406408, validators: ["426156"]
Nov 11 10:24:21 robert lighthouse[1109]: Nov 11 10:24:21.376 INFO  Previous epoch attestation(s) had sub-optimal inclusion delay  epoch: 406408, validators: ["426156"]
Nov 11 10:24:23 robert lighthouse[1109]: Nov 11 10:24:23.435 WARN  State advance too slow                        head_block_root: 0x90b5e1e52065655a01ec9a9142ac285d7b79e85f9b3b03843a3f7cfebc5088a2, advanced_slot: 13005120, current_slot: 13005120, starting_slot: 13005119, msg: "system resources may be overloaded"

If you'd like I can test when you're ready

michaelsproul · 2025-11-11T21:30:01Z

The funny thing about this patch is that it's only relevant before the fork. Once Fulu happens this change is irrelevant

It will take us a bit of time to cut the v8.0.1 release, but I can ping you once this PR is reviewed and merged and you could run it prior to v8.0.1 if you would be comfortable with that

winksaville · 2025-11-11T22:04:34Z

The funny thing about this patch is that it's only relevant before the fork. Once Fulu happens this change is irrelevant

It will take us a bit of time to cut the v8.0.1 release, but I can ping you once this PR is reviewed and merged and you could run it prior to v8.0.1 if you would be comfortable with that

I'll test it

jimmygchen

Makes sense, the fix looks good!

michaelsproul · 2025-11-12T04:13:00Z

@winksaville if you'd like to test, you can compile the release-v8.0 branch (commit f854afa). It's identical to v8.0.0 but includes this fix.

winksaville · 2025-11-12T06:30:12Z

@winksaville if you'd like to test, you can compile the release-v8.0 branch (commit f854afa). It's identical to v8.0.0 but includes this fix.

I assume if something goes wrong I can just revert to v8.0.0?

michaelsproul · 2025-11-12T06:32:06Z

Yeah it's completely backwards compat, very minimal difference from v8.0.0.

winksaville · 2025-11-14T00:29:15Z

I'm running v8.0.0-47b984e with the fix on my mainnet node:

Nov 13 15:03:48 robert lighthouse[1042]: Nov 13 15:03:48.135 INFO  Lighthouse started                            version: "Lighthouse/v8.0.0-47b984e"

And there are no "WARN State advance too slow"

So issue resolved from my perspective, TXS!

michaelsproul · 2025-11-14T02:43:16Z

Great to hear!

This hot fix release includes the following fixes: * #8388 * #8406 * #8391 * #8413 Co-Authored-By: Jimmy Chen <[email protected]>

Prevent unnecessary state advances pre-Fulu

009a0d8

michaelsproul added bug Something isn't working optimization Something to make Lighthouse run more efficiently. fulu Required for the upcoming Fulu hard fork v8.0.1 Cheeky patch release for Fulu labels Nov 10, 2025

michaelsproul requested a review from jimmygchen November 11, 2025 05:51

jimmygchen approved these changes Nov 12, 2025

View reviewed changes

jimmygchen added the ready-for-merge This PR is ready to merge. label Nov 12, 2025

mergify bot added the queued label Nov 12, 2025

mergify bot added a commit that referenced this pull request Nov 12, 2025

Merge of #8388

21c0c1a

mergify bot mentioned this pull request Nov 12, 2025

merge queue: embarking release-v8.0 (e3ee7fe) and #8388 together #8397

Closed

6 tasks

mergify bot merged commit f854afa into sigp:release-v8.0 Nov 12, 2025
37 checks passed

mergify bot removed the queued label Nov 12, 2025

michaelsproul deleted the fix-state-advance-pre-fulu branch November 12, 2025 04:12

jimmygchen mentioned this pull request Nov 14, 2025

Release v8.0.1 #8414

Merged

mergify bot pushed a commit that referenced this pull request Nov 20, 2025

Release v8.0.1 (#8414)

ced49dd

This hot fix release includes the following fixes: * #8388 * #8406 * #8391 * #8413 Co-Authored-By: Jimmy Chen <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Prevent unnecessary state advances pre-Fulu #8388

Prevent unnecessary state advances pre-Fulu #8388

Uh oh!

michaelsproul commented Nov 10, 2025 •

edited

Loading

Uh oh!

winksaville commented Nov 11, 2025

Uh oh!

michaelsproul commented Nov 11, 2025

Uh oh!

winksaville commented Nov 11, 2025

Uh oh!

jimmygchen left a comment

Uh oh!

Uh oh!

michaelsproul commented Nov 12, 2025

Uh oh!

winksaville commented Nov 12, 2025

Uh oh!

michaelsproul commented Nov 12, 2025

Uh oh!

winksaville commented Nov 14, 2025

Uh oh!

michaelsproul commented Nov 14, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Prevent unnecessary state advances pre-Fulu #8388

Prevent unnecessary state advances pre-Fulu #8388

Uh oh!

Conversation

michaelsproul commented Nov 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Issue Addressed

Proposed Changes

Uh oh!

winksaville commented Nov 11, 2025

Uh oh!

michaelsproul commented Nov 11, 2025

Uh oh!

winksaville commented Nov 11, 2025

Uh oh!

jimmygchen left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

michaelsproul commented Nov 12, 2025

Uh oh!

winksaville commented Nov 12, 2025

Uh oh!

michaelsproul commented Nov 12, 2025

Uh oh!

winksaville commented Nov 14, 2025

Uh oh!

michaelsproul commented Nov 14, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

michaelsproul commented Nov 10, 2025 •

edited

Loading