Skip to content

Only send data coumn subnet discovery requests after peerdas is scheduled #8109

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
mergify merged 3 commits into from
Sep 25, 2025
Merged

Only send data coumn subnet discovery requests after peerdas is scheduled #8109

mergify merged 3 commits into from
Sep 25, 2025

Conversation

@jimmygchen
Copy link
Member

@jimmygchen jimmygchen commented Sep 24, 2025 •
edited
Loading

Issue Addressed

#8105 (to be confirmed)

I noticed a large number of failed discovery requests after deploying latest unstable to some of our testnet and mainnet nodes. This is because of a recent PeerDAS change to attempt to maintain sufficient peers across data column subnets - this shouldn't be enabled on network without peerdas scheduled, otherwise it will keep retrying discovery on these subnets and never succeed.

Also removed some unused files.

@jimmygchen jimmygchen requested a review from jxs as a code owner September 24, 2025 07:50
@jimmygchen jimmygchen added ready-for-review The code is ready for review v8.0.0-rc.0 Q3 2025 release for Fusaka on Holesky low-hanging-fruit Easy to resolve, get it before someone else does! labels Sep 24, 2025
dapplion
dapplion reviewed Sep 24, 2025
View reviewed changes
beacon_node/lighthouse_network/src/peer_manager/mod.rs Outdated

// Maintain minimum count for custody peers.
self.maintain_custody_peers();
if self.network_globals.spec.is_peer_das_scheduled() {
Copy link
Collaborator

@dapplion dapplion Sep 24, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ideally we want to start doing this a little bit before the fork activates (say X epochs). This check is_peer_das_scheduled can be very early if a node updates its software right after the mainnet release and no one else has updated yet

Copy link
Member

@pawanjay176 pawanjay176 Sep 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree. 2 slots before is when we subscribe to the new topics, so seems like a good time

Copy link
Member Author

@jimmygchen jimmygchen Sep 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks guys, good catch! I'll make the change

Copy link
Member Author

@jimmygchen jimmygchen Sep 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm going to implement this a bit differently because peer_manager has no idea of the slot clock (and it probably shouldn't), i'm thinking to check for subscribed topics instead of 2 slots prior to Fulu - if we are subscribed to any column topics, then try to maintain custody peers.

i think the effect would be similar (with a slight delay), without introducing slot clock dependency.

Copy link
Member Author

@jimmygchen jimmygchen Sep 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done 8b7fa4a

@jimmygchen
Check for data column gossip subscriptions instead of `is_peerdas_ena…
8b7fa4a 
…bled` to avoid sending data column subnet discvery queries too early.
pawanjay176
pawanjay176 approved these changes Sep 25, 2025
View reviewed changes
Copy link
Member

@pawanjay176 pawanjay176 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.
Just noting that we are doing the gossipsub topics check here since we don't have a slot clock in the peer manager to check 2 slots before the fulu fork.
even though using the topics seem a bit hacky, its okay to do imo since the alternative would involve a lot of code that would get thrown away once we are in fulu

@jimmygchen jimmygchen added ready-for-merge This PR is ready to merge. and removed ready-for-review The code is ready for review labels Sep 25, 2025
@jimmygchen jimmygchen mentioned this pull request Sep 25, 2025
Closed
@mergify mergify bot merged commit 79b3321 into sigp:unstable Sep 25, 2025
37 checks passed
pawanjay176 pushed a commit to pawanjay176/lighthouse that referenced this pull request Sep 25, 2025
@jimmygchen @pawanjay176
Only send data coumn subnet discovery requests after peerdas is sched…
d99df0a 
…uled (sigp#8109)

sigp#8105 (to be confirmed)

I noticed a large number of failed discovery requests after deploying latest `unstable` to some of our testnet and mainnet nodes. This is because of a recent PeerDAS change to attempt to maintain sufficient peers across data column subnets - this shouldn't be enabled on network without peerdas scheduled, otherwise it will keep retrying discovery on these subnets and never succeed.

Also removed some unused files.


  


Co-Authored-By: Jimmy Chen <[email protected]>

Co-Authored-By: Jimmy Chen <[email protected]>
jimmygchen added a commit to jimmygchen/lighthouse that referenced this pull request Oct 10, 2025
@jimmygchen
Squashed commit of the following (sigp#7946):
a0f88b2 
commit 9b2de09
Author: Pawan Dhananjay <[email protected]>
Date:   Mon Oct 6 17:22:44 2025 -0700

    Rethink peer scoring

commit 5c562c6
Author: Pawan Dhananjay <[email protected]>
Date:   Mon Oct 6 11:17:57 2025 -0700

    Fix some more issues

commit 15df3d2
Merge: c491856 26575c5
Author: Pawan Dhananjay <[email protected]>
Date:   Thu Oct 2 13:15:59 2025 -0700

    Merge branch 'unstable' into blocks-then-columns

commit c491856
Merge: 826a06e ffa7b2b
Author: Pawan Dhananjay <[email protected]>
Date:   Thu Sep 25 16:23:27 2025 -0700

    Merge branch 'unstable' into blocks-then-columns

commit 826a06e
Author: Pawan Dhananjay <[email protected]>
Date:   Thu Sep 25 16:21:55 2025 -0700

    Fix variant name

commit 421e954
Author: Pawan Dhananjay <[email protected]>
Date:   Thu Sep 25 16:05:45 2025 -0700

    Revert "Revert type change in UnexpectedRequestId"

    This reverts commit 6ea14016f3d164456bc4c3cae0355ab532fe1a86.

commit 3f8998f
Author: Lion - dapplion <[email protected]>
Date:   Thu Sep 25 05:52:27 2025 +0200

    Only mark block lookups as pending if block is importing from gossip (sigp#8112)

    - PR sigp#8045 introduced a regression of how lookup sync interacts with the da_checker.

    Now in unstable block import from the HTTP API also insert the block in the da_checker while the block is being execution verified. If lookup sync finds the block in the da_checker in `NotValidated` state it expects a `GossipBlockProcessResult` message sometime later. That message is only sent after block import in gossip.

    I confirmed in our node's logs for 4/4 cases of stuck lookups are caused by this sequence of events:
    - Receive block through API, insert into da_checker in fn process_block in put_pre_execution_block
    - Create lookup and leave in AwaitingDownload(block in processing cache) state
    - Block from HTTP API finishes importing
    - Lookup is left stuck

    Closes sigp#8104

      - sigp#8110 was my initial solution attempt but we can't send the `GossipBlockProcessResult` event from the `http_api` crate without adding new channels, which seems messy.

    For a given node it's rare that a lookup is created at the same time that a block is being published. This PR solves sigp#8104 by allowing lookup sync to import the block twice in that case.

    Co-Authored-By: dapplion <[email protected]>

commit d99df0a
Author: Jimmy Chen <[email protected]>
Date:   Thu Sep 25 12:52:07 2025 +1000

    Only send data coumn subnet discovery requests after peerdas is scheduled (sigp#8109)

    sigp#8105 (to be confirmed)

    I noticed a large number of failed discovery requests after deploying latest `unstable` to some of our testnet and mainnet nodes. This is because of a recent PeerDAS change to attempt to maintain sufficient peers across data column subnets - this shouldn't be enabled on network without peerdas scheduled, otherwise it will keep retrying discovery on these subnets and never succeed.

    Also removed some unused files.

    Co-Authored-By: Jimmy Chen <[email protected]>

    Co-Authored-By: Jimmy Chen <[email protected]>

commit cf46d10
Author: Pawan Dhananjay <[email protected]>
Date:   Thu Sep 25 15:54:32 2025 -0700

    Fix issues from review

commit c2aa4ae
Author: dapplion <[email protected]>
Date:   Thu Sep 25 23:58:31 2025 +0200

    Revert type change in UnexpectedRequestId

commit 7488755
Author: dapplion <[email protected]>
Date:   Thu Sep 25 22:09:06 2025 +0200

    De-duplicate data columns by root request type

commit 7650032
Author: dapplion <[email protected]>
Date:   Thu Sep 25 23:52:47 2025 +0200

    Rename DataColumnsFromRange

commit 4b0b655
Author: Pawan Dhananjay <[email protected]>
Date:   Wed Sep 24 12:31:06 2025 -0700

    Fix nits

commit aa6a1bc
Author: Pawan Dhananjay <[email protected]>
Date:   Wed Sep 24 12:16:05 2025 -0700

    Create a custom penalize_sync_peer method for clarity

commit 27195ca
Merge: 4a59d35 af27402
Author: Pawan Dhananjay <[email protected]>
Date:   Wed Sep 24 11:16:27 2025 -0700

    Merge branch 'unstable' into blocks-then-columns

commit 4a59d35
Merge: 2f35c36 5928407
Author: Pawan Dhananjay <[email protected]>
Date:   Wed Sep 17 15:15:44 2025 -0700

    Merge branch 'unstable' into blocks-then-columns

commit 2f35c36
Author: Pawan Dhananjay <[email protected]>
Date:   Wed Sep 17 15:13:40 2025 -0700

    Add some metrics

commit b3b3756
Author: Pawan Dhananjay <[email protected]>
Date:   Wed Sep 17 15:01:46 2025 -0700

    Fix tests

commit e3aed89
Author: Pawan Dhananjay <[email protected]>
Date:   Tue Sep 16 17:46:21 2025 -0700

    Remove retry test that we do not use anymore

commit baee27a
Merge: 9db4c30 191570e
Author: Pawan Dhananjay <[email protected]>
Date:   Tue Sep 16 16:39:30 2025 -0700

    Merge branch 'unstable' into blocks-then-columns

commit 9db4c30
Author: Pawan Dhananjay <[email protected]>
Date:   Tue Sep 16 16:38:19 2025 -0700

    Fix small bug

commit 08bba3f
Author: Pawan Dhananjay <[email protected]>
Date:   Thu Sep 11 18:24:25 2025 -0700

    fmt

commit cffbd34
Author: Pawan Dhananjay <[email protected]>
Date:   Thu Sep 11 18:19:16 2025 -0700

    Reduce code duplication

commit bf09d57
Author: Pawan Dhananjay <[email protected]>
Date:   Thu Sep 11 15:06:07 2025 -0700

    Fix some issues from lion's review

commit 04398ad
Author: Pawan Dhananjay <[email protected]>
Date:   Mon Sep 8 15:37:26 2025 -0700

    Fix some more issues from review

commit 4f62a9c
Merge: e259ecd 8ec2640
Author: Pawan Dhananjay <[email protected]>
Date:   Fri Sep 5 13:00:47 2025 -0700

    Merge branch 'unstable' into blocks-then-columns

commit e259ecd
Author: Pawan Dhananjay <[email protected]>
Date:   Fri Sep 5 12:58:49 2025 -0700

    More renamings

commit 6a2a33d
Author: Pawan Dhananjay <[email protected]>
Date:   Fri Sep 5 12:48:05 2025 -0700

    Fix some issues from review

commit e0d8f04
Author: Pawan Dhananjay <[email protected]>
Date:   Thu Sep 4 18:09:07 2025 -0700

    Tests compile

commit 7e91eeb
Merge: 29c2f83 9d2f55a
Author: Pawan Dhananjay <[email protected]>
Date:   Thu Sep 4 18:08:52 2025 -0700

    Merge branch 'unstable' into blocks-then-columns

commit 29c2f83
Author: Pawan Dhananjay <[email protected]>
Date:   Tue Sep 2 13:29:19 2025 -0700

    Cleanup SyncingChain

commit 8458df6
Author: Pawan Dhananjay <[email protected]>
Date:   Mon Sep 1 12:53:01 2025 -0700

    Attempt sending awaitingDownload batches when restarting sync

commit 7a6d0d9
Author: Pawan Dhananjay <[email protected]>
Date:   Sun Aug 31 19:07:39 2025 -0700

    Force processing_target request

commit 4f60e86
Author: Pawan Dhananjay <[email protected]>
Date:   Sun Aug 31 14:54:44 2025 -0700

    Add logs to debug stuck range sync

commit b07bc6d
Author: Pawan Dhananjay <[email protected]>
Date:   Fri Aug 29 16:18:53 2025 -0700

    Force requests if batch buffer is full under certain conditions

commit 19b0a5c
Merge: 8e1337d 438fb65
Author: Pawan Dhananjay <[email protected]>
Date:   Thu Aug 28 23:25:14 2025 -0700

    Merge branch 'unstable' into blocks-then-columns

commit 8e1337d
Merge: da1aaba a134d43
Author: Pawan Dhananjay <[email protected]>
Date:   Thu Aug 28 18:03:44 2025 -0700

    Merge branch 'unstable' into blocks-then-columns

commit da1aaba
Author: Pawan Dhananjay <[email protected]>
Date:   Wed Aug 27 17:34:27 2025 -0700

    Revise error tolerance

commit 7331323
Author: Pawan Dhananjay <[email protected]>
Date:   Wed Aug 27 17:30:46 2025 -0700

    AwaitingValidation state only needs block peer

commit 8386bd9
Merge: b4bc7fe d235f2c
Author: Pawan Dhananjay <[email protected]>
Date:   Wed Aug 27 17:08:42 2025 -0700

    Merge branch 'unstable' into blocks-then-columns

commit b4bc7fe
Author: Pawan Dhananjay <[email protected]>
Date:   Wed Aug 27 14:30:46 2025 -0700

    Remove more debug logs

commit 05adb71
Author: Pawan Dhananjay <[email protected]>
Date:   Wed Aug 27 14:26:28 2025 -0700

    Fix bug with partial column responses before all column requests sent

commit a97cf88
Author: Pawan Dhananjay <[email protected]>
Date:   Tue Aug 26 18:53:51 2025 -0700

    Add docs

commit 27d0b36
Author: Pawan Dhananjay <[email protected]>
Date:   Tue Aug 26 16:06:27 2025 -0700

    Remove debug statements

commit 90d319f
Merge: 7c214f5 3e78034
Author: Pawan Dhananjay <[email protected]>
Date:   Tue Aug 26 15:38:47 2025 -0700

    Merge branch 'unstable' into blocks-then-columns

commit 7c214f5
Merge: 52762b9 daf1c7c
Author: Pawan Dhananjay <[email protected]>
Date:   Mon Aug 25 11:01:40 2025 -0700

    Merge branch 'unstable' into blocks-then-columns

commit 52762b9
Author: Pawan Dhananjay <[email protected]>
Date:   Fri Aug 22 11:52:56 2025 -0700

    Handle 0 blobs per epoch case

commit da27441
Merge: 521778b cee30d8
Author: Pawan Dhananjay <[email protected]>
Date:   Thu Aug 21 11:02:22 2025 -0700

    Merge branch 'unstable' into blocks-then-columns

commit 521778b
Author: Pawan Dhananjay <[email protected]>
Date:   Thu Aug 21 07:03:58 2025 -0700

    Random logs

commit 4540195
Author: Pawan Dhananjay <[email protected]>
Date:   Wed Aug 13 17:33:47 2025 -0700

    Request columns from global peer pool

commit fdce537
Author: Pawan Dhananjay <[email protected]>
Date:   Wed Aug 20 18:37:44 2025 -0700

    Implement responsible peer tracking

commit 17c4e34
Author: Pawan Dhananjay <[email protected]>
Date:   Wed Aug 20 14:18:15 2025 -0700

    Avoid root requests for backfill sync

commit 1a0df30
Author: Pawan Dhananjay <[email protected]>
Date:   Wed Aug 20 14:14:48 2025 -0700

    Also penalize all batch peers for availability check errors

commit 6da924b
Author: Pawan Dhananjay <[email protected]>
Date:   Wed Aug 20 14:14:26 2025 -0700

    Fix bug in initialization code

commit 68cce37
Author: Pawan Dhananjay <[email protected]>
Date:   Wed Aug 20 14:14:11 2025 -0700

    Try to avoid chains failing for rpc errors

commit ca9cfd5
Author: Pawan Dhananjay <[email protected]>
Date:   Tue Aug 19 12:17:59 2025 -0700

    Small fixes

commit 5337e46
Author: Pawan Dhananjay <[email protected]>
Date:   Mon Aug 18 22:59:38 2025 -0700

    Add a function to retry column requests that could not be made

commit 9455153
Author: Pawan Dhananjay <[email protected]>
Date:   Mon Aug 18 14:48:44 2025 -0700

    Without retries

commit 6bd8944
Author: Pawan Dhananjay <[email protected]>
Date:   Fri Aug 15 14:23:48 2025 -0700

    Reduce backfill buffer size

commit 156449c
Author: Pawan Dhananjay <[email protected]>
Date:   Thu Aug 14 09:10:28 2025 -0700

    Increase columns_by_root quota

commit 836f9c6
Author: Pawan Dhananjay <[email protected]>
Date:   Wed Aug 13 16:37:12 2025 -0700

    Priorotize status v2

commit 490b627
Author: Pawan Dhananjay <[email protected]>
Date:   Wed Aug 13 16:36:56 2025 -0700

    Penalize if invalid EL block
lmnzx pushed a commit to lmnzx/lighthouse that referenced this pull request Nov 4, 2025
@jimmygchen @lmnzx
Only send data coumn subnet discovery requests after peerdas is sched…
63597de 
…uled (sigp#8109)

sigp#8105 (to be confirmed)

I noticed a large number of failed discovery requests after deploying latest `unstable` to some of our testnet and mainnet nodes. This is because of a recent PeerDAS change to attempt to maintain sufficient peers across data column subnets - this shouldn't be enabled on network without peerdas scheduled, otherwise it will keep retrying discovery on these subnets and never succeed.

Also removed some unused files.


  


Co-Authored-By: Jimmy Chen <[email protected]>

Co-Authored-By: Jimmy Chen <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dapplion dapplion dapplion left review comments

@pawanjay176 pawanjay176 pawanjay176 approved these changes

@jxs jxs Awaiting requested review from jxs jxs is a code owner

Assignees

No one assigned

Labels

low-hanging-fruit Easy to resolve, get it before someone else does! ready-for-merge This PR is ready to merge. v8.0.0-rc.0 Q3 2025 release for Fusaka on Holesky

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@jimmygchen @pawanjay176 @dapplion