Skip to content

Run reconstruction inside a scoped rayon pool #8075

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
mergify merged 48 commits into from
Sep 24, 2025
Merged

Run reconstruction inside a scoped rayon pool #8075

mergify merged 48 commits into from
Sep 24, 2025

Conversation

@eserilev
Copy link
Member

@eserilev eserilev commented Sep 18, 2025 •
edited
Loading

  • Introduced a high priority rayon thread pool which uses ~80% of cpu, or at least 3 threads.
  • Created a spawn_blocking_handle_with_rayon fn inside TaskExecutor.
  • Data column reconstruction now runs within the new scoped high priority rayon pool.
  • Moved RayonManager reference to TaskExecutor

It isn't straightforward to convert the full Reconstruction work event from an async to a blocking task. This approach is a temporary stopgap until we are able to fully separate async tasks from compute heavy tasks.

jimmygchen and others added 27 commits August 23, 2025 01:43
@jimmygchen
Use scoped rayon pool for chain segment backfill.
47a80e5
@eserilev
Merge branch 'unstable' of https://github.com/sigp/lighthouse into ba…
5cbfdb5 
…ckfill-verify-kzg-use-scoped-rayon
@eserilev
Merge branch 'unstable' of https://github.com/sigp/lighthouse into ba…
cc1b870 
…ckfill-verify-kzg-use-scoped-rayon
@eserilev
Create RayonManager struct
28f6ad2
@eserilev
Linting
583aaba
@eserilev
Fix comment
16a1b48
@eserilev
Introduce a high prio threadpool for use when backfill rate limiting …
f81f5ea 
…is disabled
@eserilev
Fix test
067b599
@eserilev
Merge branch 'unstable' of https://github.com/sigp/lighthouse into ba…
23bd8bf 
…ckfill-verify-kzg-use-scoped-rayon
@eserilev
Merge branch 'unstable' of https://github.com/sigp/lighthouse into ba…
4e884bb 
…ckfill-verify-kzg-use-scoped-rayon
@eserilev
fix test
f1144e4
@eserilev
Merge branch 'unstable' into backfill-verify-kzg-use-scoped-rayon
ba65b98
@eserilev
Add default impl
09433eb
@eserilev
Remove scoped high prio thread pool in favor of the global thread pool
b5ff451
@eserilev
Merge branch 'backfill-verify-kzg-use-scoped-rayon' of https://github…
0debde9 
….com/jimmygchen/lighthouse into backfill-verify-kzg-use-scoped-rayon
@eserilev
USe the global rayon pool
1694020
@eserilev
FMT
456af43
@eserilev
Merge branch 'unstable' of https://github.com/sigp/lighthouse into ba…
b289f49 
…ckfill-verify-kzg-use-scoped-rayon
@eserilev
fix test
9b57acc
@eserilev
Fix
dabbdc7
@eserilev
ugh
3f57327
@eserilev
fix
091aafb
@eserilev
low prio
6a525cb
@eserilev
fix comment
2d7e57a
@eserilev
update minimum thread count to 1
c0467f4
@eserilev
Move reconstruction task to a scoped high priority threadpool
f1a7269
@eserilev
fmt
dd75403
@jimmygchen jimmygchen added waiting-on-author The reviewer has suggested changes and awaits thier implementation. and removed ready-for-review The code is ready for review labels Sep 23, 2025
jimmygchen
jimmygchen requested changes Sep 23, 2025
View reviewed changes
Copy link
Member

@jimmygchen jimmygchen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @eserilev I think your suggested approach looks good and i don't see any issue with it.

@jimmygchen jimmygchen added v8.0.0-rc.0 Q3 2025 release for Fusaka on Holesky and removed v8.0.0 Q4 2025 Fusaka Mainnet Release labels Sep 23, 2025
@jimmygchen
Copy link
Member

jimmygchen commented Sep 23, 2025

Updating this to v8.0.0-rc.0 as I think it's important to have this in the testnet release, so reconstruction doesn't impact supernode block processing and attesting.

@eserilev eserilev added ready-for-review The code is ready for review and removed waiting-on-author The reviewer has suggested changes and awaits thier implementation. labels Sep 23, 2025
jimmygchen
jimmygchen approved these changes Sep 23, 2025
View reviewed changes
Copy link
Member

@jimmygchen jimmygchen left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good, thanks! Will wait for another reviewer and some testing and then merge this.

I think it might be worth clarifying on the below functions, i.e. the difference isnt just blocking /async, we'd use the 2nd one of if we want to block the async thread to await for the return values (maybe fn naming + describe when to use each?)

  • spawn_blocking_with_rayon
  • spawn_rayon_async

@eserilev
Copy link
Member Author

eserilev commented Sep 23, 2025 •
edited
Loading

Reconstruction inside the scoped high prio rayon pool (while setting reconstruction delay to 0)
tracing

tracing

@eserilev
Copy link
Member Author

eserilev commented Sep 24, 2025

When using the global rayon pool and setting reconstruction delay to 0

tracing tracing

@eserilev
Copy link
Member Author

eserilev commented Sep 24, 2025

In the scoped high prio rayon pool with reconstruction delay set to 0

over a one hour period

tree-hash-cache 90% of these were under 3ms
tracing

blobs_to_data_column_sidecars more than 90% were under 30ms
tracing

jimmygchen
jimmygchen approved these changes Sep 24, 2025
View reviewed changes
Copy link
Member

@jimmygchen jimmygchen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice numbers 👏 pretty good improvements there, shows that the scoped rayon impl is quite effective.

@jimmygchen jimmygchen added ready-for-merge This PR is ready to merge. and removed ready-for-review The code is ready for review labels Sep 24, 2025
mergify bot added a commit that referenced this pull request Sep 24, 2025
@mergify
Merge of #8075
ee4628a
@mergify mergify bot mentioned this pull request Sep 24, 2025
Closed
6 tasks
@mergify
Copy link

mergify bot commented Sep 24, 2025

This pull request has been removed from the queue for the following reason: checks failed.

The merge conditions cannot be satisfied due to failing checks:

You can check the last failing draft PR here: #8107.

You may have to fix your CI before adding the pull request to the queue again.
If you update this pull request, to fix the CI, it will automatically be requeued once the queue conditions match again.
If you think this was a flaky issue instead, you can requeue the pull request, without updating it, by posting a @mergifyio requeue comment.

@jimmygchen jimmygchen mentioned this pull request Sep 24, 2025
Closed
mergify bot added a commit that referenced this pull request Sep 24, 2025
@mergify
Merge of #8075
1d2e036
@mergify mergify bot mentioned this pull request Sep 24, 2025
Closed
6 tasks
@mergify mergify bot merged commit af27402 into sigp:unstable Sep 24, 2025
37 checks passed
lmnzx pushed a commit to lmnzx/lighthouse that referenced this pull request Nov 4, 2025
@eserilev @lmnzx
Run reconstruction inside a scoped rayon pool (sigp#8075)
b947d02 
Co-Authored-By: Jimmy Chen <[email protected]>

Co-Authored-By: Eitan Seri- Levi <[email protected]>

Co-Authored-By: Eitan Seri-Levi <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jimmygchen jimmygchen jimmygchen approved these changes

@michaelsproul michaelsproul Awaiting requested review from michaelsproul

@jxs jxs Awaiting requested review from jxs jxs is a code owner

Assignees

No one assigned

Labels

beacon-processor Glorious beacon processor, guardian against chaos yet chaotic itself optimization Something to make Lighthouse run more efficiently. ready-for-merge This PR is ready to merge. v8.0.0-rc.0 Q3 2025 release for Fusaka on Holesky

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@eserilev @jimmygchen