Skip to content

Conversation

@michaelsproul
Copy link
Member

Issue Addressed

Fix for error message seen on some nodes which are struggling slightly after Pectra:

ERROR Unable to queue converted SingleAttestation error: no available capacity, slot: 4230400

Because we convert SingleAttestations to regular attestations in a separate work event, we are now saturating the work event queue more often. This will remain an issue until a more optimised implementation of SingleAttestation processing is merged (likely for v7.1.0). See:

Proposed Changes

Double the size of the generic work event queue. This isn't guaranteed to fix the issue, as nodes could be struggling to an arbitrary degree. However it should make up for the approx doubling of work events introduced by Pectra so that only nodes logging errors prior to the fork are logging errors now.

Additional Info

I'm rolling this out to some of our Holesky nodes that exhibit this behaviour and will report back on its efficacy.

@michaelsproul michaelsproul added optimization Something to make Lighthouse run more efficiently. UX-and-logs v7.0.2 Patch release for shortly after Pectra on mainnet labels May 8, 2025
@michaelsproul michaelsproul changed the base branch from stable to release-v7.0.0 May 8, 2025 01:29
@michaelsproul michaelsproul force-pushed the bump-work-queue-size branch from bdede4f to a6e3a41 Compare May 8, 2025 01:29
@michaelsproul
Copy link
Member Author

I forgot but we also still have the hidden flag for this, so users could crank up the queue size without a new release using this flag:

--beacon-processor-work-queue-len 32768

@michaelsproul michaelsproul added the ready-for-review The code is ready for review label May 8, 2025
Copy link
Member

@eserilev eserilev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for opening this PR. Just one smol test failure, but other than that LGTM!

@michaelsproul
Copy link
Member Author

This change doesn't seem to have been effective in preventing the queue from filling. We might need a higher value.

@michaelsproul
Copy link
Member Author

michaelsproul commented May 19, 2025

I'm inclined to close this in favour of the "proper" fix in v7.1.0:

@michaelsproul michaelsproul deleted the bump-work-queue-size branch October 27, 2025 23:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

optimization Something to make Lighthouse run more efficiently. ready-for-review The code is ready for review UX-and-logs v7.0.2 Patch release for shortly after Pectra on mainnet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants