seeds: Add additional seed source and bump uptime requirements for Onion and I2P nodes #30695

virtu · 2024-08-22T14:25:07Z

This builds on #30008 and adds data exported by my crawler an additional source for seed nodes. Data covers all supported network types.

[edit: Added Luke's seeder as input as well.]

Motivation

Further decentralizes the seed node selection process (in the long term potentially enabling an n-source threshold for nodes to prevent a single source from entering malicious nodes)
No longer need to manually curate seed node list for any network type: See last paragraph of OP in seeds: Pull additional nodes from my seeder and update fixed seeds #30008. My crawler has been discovering the handful of available cjdns nodes for around two months, all but one of which meet the reliability criteria.
Alignment of uptime requirements for Onion and I2P nodes with those of clearnet nodes to 50%: If I'm reading the code correctly, seeders appear to optimize for up-to-dateness by using lower connection timeouts than Bitcoin Core to maximize throughput. Since my crawler does not have the same timeliness requirements, it opts for accuracy by using generous timeouts. As a result, its data contains additional eligible Onion (and other darknet nodes), as is shown in the histogram below. Around 4500 Onion nodes are discovered so far (blue); my data adds ~6400 more (orange); ~ 1500 nodes take longer than the default 20-second Bitcoin Core timeout and won't qualify as "good".

Here's the current results with 512 nodes for all networks except cjdns:

Using the extra data

IPv4   IPv6  Onion  I2P    CJDNS Pass
10335   2531  11545   1589     10 Initial
10335   2531  11545   1589     10 Skip entries with invalid address
5639   1431  11163   1589      8 After removing duplicates
5606   1417  11163   1589      8 Enforce minimal number of blocks
5606   1417  11163   1589      8 Require service bit 1
4873   1228  11163   1589      8 Require minimum uptime
4846   1225  11161   1588      8 Require a known and recent user agent
4846   1225  11161   1588      8 Filter out hosts with multiple bitcoin ports
512    512    512    512      8 Look up ASNs and limit results per ASN and per net

Before

IPv4   IPv6  Onion  I2P    CJDNS Pass
5772   1323    443      0      2 Initial
5772   1323    443      0      2 Skip entries with invalid address
4758   1110    443      0      2 After removing duplicates
4723   1094    443      0      2 Enforce minimal number of blocks
4723   1094    443      0      2 Require service bit 1
3732    867    443      0      2 Require minimum uptime
3718    864    443      0      2 Require a known and recent user agent
3718    864    443      0      2 Filter out hosts with multiple bitcoin ports
 512    409    443      0      2 Look up ASNs and limit results per ASN and per net

To dos

Remove manual nodes and update README
Mark nodes with connection times exceeding Bitcoin Core's default as bad in exporter: done
Regenerate mainnet seeds
Rebase, then remove WIP label once seeds: Pull additional nodes from my seeder and update fixed seeds #30008 gets merged

DrahtBot · 2024-08-22T14:25:10Z

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage

For detailed information about the code coverage, see the test coverage report.

Reviews

See the guideline for information on the review process.

Type	Reviewers
ACK	achow101, fjahr

If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

Conflicts

No conflicts as of last run.

achow101 · 2024-08-22T14:43:59Z

contrib/seeds/README.md

This will overwrite instead of append.

jonatack · 2024-08-22T17:05:52Z

contrib/seeds/nodes_main.txt

I could be misreading, but it looks like all the tor and i2p peers have been removed without being added elsewhere.

I hadn't regenerated to node lists because I wanted to wait until Ava's seeder resumed exporting good I2P node data. I have now pushed a preliminary export anyway. Everything should be there.

jonatack · 2024-08-22T17:09:17Z

As a result, its data contains additional eligible Onion (and other darknet nodes), as is shown in the histogram below.

Some very good tor and i2p peers seems absent from the update. I addnode them, but hm.

virtu · 2024-08-23T10:03:35Z

I've pushed a commit including the node lists generated with all sources. Could you recheck for the peers that were missing? [On a side note, I noticed what (based on the user agent string) might be your cjdns node is no longer part of the cjdns node list because the node didn't have the NODE_NETWORK version bit set.]

Also, how do you define very good nodes? Right now, seeds are selected more or less randomly from the set of nodes that pass all selection criteria. Instead of doing a random.shuffle(), it might be a better idea to sort them based on availability.

jonatack · 2024-08-26T21:17:35Z

your cjdns node is no longer part of the cjdns node list because the node didn't have the NODE_NETWORK version bit set

Yes, I reluctantly had to begin pruning due to the increased rate of chain data growth over the past year and a half.

Is there now only one CJDNS seed node? Edit: OK, I see at least 4 other CJDNS seed nodes.

jonatack · 2024-08-26T21:33:08Z

Also, how do you define very good nodes?

I was manually curating I2P nodes based on trusted colleagues (akin to addnode peer selection), filtered by connection reliability and regularly seeing blocks from them. (Edit: I now see that you have ones that I'd recommend, so seems good. There were a couple that were missing, but I see that they, like me, began pruning.)

luke-jr · 2024-08-27T00:40:06Z

contrib/seeds/README.md

Mine is here if you want to add it:

https://luke.dashjr.org/programs/bitcoin/files/charts/seeds.txt

nice, added!

Pull additional nodes from virtu's crawler. Data includes sufficient Onion and I2P nodes to align the uptime requirements for these networks to that of clearnet nodes (i.e., 50%). Data also includes more than three times the number of CJDNS nodes currently hardcoded into nodes_main_manual.txt, so hardcoded nodes becomes obsolete.

Pull additional nodes from Luke's seeder to further decentralize the generation of seed nodes.

Regenerate mainnet seeds from new sources without the need for hardcoded data. Result has 512 nodes from each network type except cjdns, for which only eight nodes were found that match the seed node criteria.

virtu · 2024-08-27T05:12:27Z

I was manually curating I2P nodes based on trusted colleagues (akin to addnode peer selection), filtered by connection reliability and regularly seeing blocks from them. (Edit: I now see that you have ones that I'd recommend, so seems good. There were a couple that were missing, but I see that they, like me, began pruning.)

Good to know the automatic process is getting all of the cjdns nodes you were tracking (modulo pruning).

Data from @luke-jr's seed is now included as well. As before: 512 nodes for each network type except cjdns, for which there are eight matching the seed node criteria.

achow101 · 2024-08-27T15:31:13Z

ACK b061b35

Instruction updates look correct; did not check any of the changed seeds but did check that i2p and onion mainnet seeds have been readded.

If anyone has a testnet crawler, we probably want to have additional sources for testnet and add more i2p and onion testnet seeds.

fjahr

utACK b061b35

I have reviewed the changes and verified that the included files generate the included chainparamsseeds.h. I have not tested the seed nodes but I did a quick plausibility check on the txt file changes.

Fun side note: TIL that there is a stable node with a single digit ASN, AS9, which is Carnegie Mellon University.

fjahr · 2024-08-27T15:47:10Z

contrib/seeds/README.md

-The seeds compiled into the release are created from sipa's and achow101's DNS seed and AS map
-data. Run the following commands from the `/contrib/seeds` directory:
+The seeds compiled into the release are created from sipa's and achow101's DNS seed,
+virtu's crawler, and fjahr's community AS map data. Run the following commands from the


nit: It's clear what belongs to who because all the other links have the names in the URL but still might be nicer to be explicit in the future. Ignore if you don't have to retouch.

Suggested change

virtu's crawler, and fjahr's community AS map data. Run the following commands from the

virtu's crawler (21.ninja), and fjahr's community AS map data. Run the following commands from the

…blocks, fixed seeds f0b6597 seeds: update .gitignore with signet and testnet4 (Jon Atack) 48f07ac chainparams: remove hardcoded signet seeds (Jon Atack) d4ab115 chainparams: add signet fixed seeds if default network (Jon Atack) 49f155e seeds: update fixed dns seeds (Jon Atack) 2366870 makeseeds: regex improvements (Lőrinc) 98f84d6 generate-seeds: update and add signet (Jon Atack) c4ed23e seeds: add testnet4 seeds (Jon Atack) 60f17dd seeds: add signet seeds (Jon Atack) 2bcccaa makeseeds: align I2P column header (Jon Atack) 94e21aa makeseeds: update MIN_BLOCKS, add reminder to README (Jon Atack) 6ae7a3b makeseeds: update user agent regex (Jon Atack) 9b0d2e5 makeseeds: fix incorrect regex (laanwj) Pull request description: In `makeseeds.py`: - fix the user agent regex (by laanwj) - fix the I2P column header spacing - update the regex (it was also not updated for the previous release) - update `MIN_BLOCKS` (4320 blocks/month * ~6.5 months) and add README documentation to remember to update it - further robustness/standardness/consistency improvements to the regexes (by l0rinc) Add signet and testnet4 seeds to the README and to `generate-seeds.py` Update the fixed seeds in `src/chainparamsseeds.h` In `kernel/chainparams.cpp`: - add signet fixed seeds if default network - remove hardcoded signet seeds Update `contrib/seeds/.gitignore` with signet and testnet4 The previous 2 seeds updates were #30008 and #30695. mainnet: ``` $ contrib/seeds$ python3 makeseeds.py -a asmap-filled.dat -s seeds_main.txt > nodes_main.txt Loading asmap database "asmap-filled.dat"…Done. Loading and parsing DNS seeds…Done. IPv4 IPv6 Onion I2P CJDNS Pass 17252 3630 21079 3095 12 Initial 17252 3630 21079 3095 12 Skip entries with invalid address 8444 1742 14607 2330 10 After removing duplicates 8194 1691 14321 2102 10 Enforce minimal number of blocks 7838 1578 14321 2102 10 Require service bit 1 6802 1326 14321 2102 10 Require minimum uptime 6762 1321 14320 2102 10 Require a known and recent user agent 6762 1321 14320 2102 10 Filter out hosts with multiple bitcoin ports 512 485 512 512 10 Look up ASNs and limit results per ASN and per net ``` signet: ``` $ contrib/seeds$ python3 makeseeds.py -a asmap-filled.dat -s seeds_signet.txt -m 237800 > nodes_signet.txt Loading asmap database "asmap-filled.dat"…Done. Loading and parsing DNS seeds…Done. IPv4 IPv6 Onion I2P CJDNS Pass 110 47 63 9 4 Initial 110 47 63 9 4 Skip entries with invalid address 110 47 63 9 4 After removing duplicates 83 31 58 9 4 Enforce minimal number of blocks 83 31 58 9 4 Require service bit 1 83 31 57 9 4 Require minimum uptime 83 31 57 9 4 Require a known and recent user agent 83 31 57 7 4 Filter out hosts with multiple bitcoin ports 42 30 57 7 4 Look up ASNs and limit results per ASN and per net ``` testnet: ``` $ contrib/seeds$ python3 makeseeds.py -a asmap-filled.dat -s seeds_test.txt > nodes_test.txt Loading asmap database "asmap-filled.dat"…Done. Loading and parsing DNS seeds…Done. IPv4 IPv6 Onion I2P CJDNS Pass 204 73 96 11 5 Initial 204 73 96 11 5 Skip entries with invalid address 204 73 96 11 5 After removing duplicates 204 73 96 11 5 Enforce minimal number of blocks 204 73 96 11 5 Require service bit 1 195 69 87 9 5 Require minimum uptime 193 69 87 9 5 Require a known and recent user agent 193 69 87 9 5 Filter out hosts with multiple bitcoin ports 79 39 87 9 5 Look up ASNs and limit results per ASN and per net ``` testnet4 ``` $ contrib/seeds$ python3 makeseeds.py -a asmap-filled.dat -s seeds_testnet4.txt -m 72600 > nodes_testnet4.txt Loading asmap database "asmap-filled.dat"…Done. Loading and parsing DNS seeds…Done. IPv4 IPv6 Onion I2P CJDNS Pass 149 115 69 11 4 Initial 149 115 69 11 4 Skip entries with invalid address 149 115 69 11 4 After removing duplicates 104 75 52 7 4 Enforce minimal number of blocks 104 75 52 7 4 Require service bit 1 100 73 51 7 4 Require minimum uptime 100 73 51 7 4 Require a known and recent user agent 100 73 51 7 4 Filter out hosts with multiple bitcoin ports 43 46 51 7 4 Look up ASNs and limit results per ASN and per net ``` ACKs for top commit: l0rinc: I have mostly reviewed the regexes, for the rest it's only a very lightweight ACK f0b6597 achow101: ACK f0b6597 laanwj: re-ACK f0b6597 Tree-SHA512: 86f4ea247469dbb3f131f2de884e470fbf93f399744d4854fcc26511afafcec231d7eaed37f8564244bc64d917d130b314d948aa97b13020613f8e186c70e368

virtu changed the title ~~WIP: seeds: Add additional seed source and bump uptime requirements for Onion and I2P nodes~~ [WIP] seeds: Add additional seed source and bump uptime requirements for Onion and I2P nodes Aug 22, 2024

achow101 reviewed Aug 22, 2024

View reviewed changes

contrib/seeds/README.md Outdated

Copy link

Member

achow101 Aug 22, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will overwrite instead of append.

virtu reacted with thumbs up emoji

virtu force-pushed the add-seed-source-bump-uptime-requirements branch from 8332e21 to f0a7a15 Compare August 22, 2024 15:18

jonatack reviewed Aug 22, 2024

View reviewed changes

DrahtBot mentioned this pull request Aug 23, 2024

seeds: Pull additional nodes from my seeder and update fixed seeds #30008

Merged

DrahtBot added the Needs rebase label Aug 26, 2024

luke-jr reviewed Aug 27, 2024

View reviewed changes

virtu force-pushed the add-seed-source-bump-uptime-requirements branch 2 times, most recently from 5217274 to a948af6 Compare August 27, 2024 04:47

virtu added 3 commits August 27, 2024 07:00

seeds: Pull nodes from Luke's seeder

02dc45c

Pull additional nodes from Luke's seeder to further decentralize the generation of seed nodes.

seeds: Regenerate mainnet seeds

b061b35

Regenerate mainnet seeds from new sources without the need for hardcoded data. Result has 512 nodes from each network type except cjdns, for which only eight nodes were found that match the seed node criteria.

virtu force-pushed the add-seed-source-bump-uptime-requirements branch from a948af6 to b061b35 Compare August 27, 2024 05:00

virtu changed the title ~~[WIP] seeds: Add additional seed source and bump uptime requirements for Onion and I2P nodes~~ seeds: Add additional seed source and bump uptime requirements for Onion and I2P nodes Aug 27, 2024

DrahtBot removed the Needs rebase label Aug 27, 2024

achow101 added this to the 28.0 milestone Aug 27, 2024

fjahr reviewed Aug 27, 2024

View reviewed changes

achow101 merged commit 0022c84 into bitcoin:master Aug 27, 2024

virtu deleted the add-seed-source-bump-uptime-requirements branch August 28, 2024 07:56

jonatack mentioned this pull request Feb 27, 2025

seeds: add signet/testnet4, update makeseeds regex, minblocks, fixed seeds #31960

Merged

bitcoin locked and limited conversation to collaborators Aug 28, 2025

	virtu's crawler, and fjahr's community AS map data. Run the following commands from the
	virtu's crawler (21.ninja), and fjahr's community AS map data. Run the following commands from the

seeds: Add additional seed source and bump uptime requirements for Onion and I2P nodes #30695

seeds: Add additional seed source and bump uptime requirements for Onion and I2P nodes #30695

Uh oh!

Conversation

virtu commented Aug 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

To dos

Uh oh!

DrahtBot commented Aug 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Coverage

Reviews

Conflicts

Uh oh!

achow101 Aug 22, 2024

Choose a reason for hiding this comment

Uh oh!

jonatack Aug 22, 2024

Choose a reason for hiding this comment

Uh oh!

virtu Aug 23, 2024

Choose a reason for hiding this comment

Uh oh!

jonatack commented Aug 22, 2024

Uh oh!

virtu commented Aug 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jonatack commented Aug 26, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jonatack commented Aug 26, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

luke-jr Aug 27, 2024

Choose a reason for hiding this comment

Uh oh!

virtu Aug 27, 2024

Choose a reason for hiding this comment

Uh oh!

virtu commented Aug 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

achow101 commented Aug 27, 2024

Uh oh!

fjahr left a comment

Choose a reason for hiding this comment

Uh oh!

fjahr Aug 27, 2024

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

virtu commented Aug 22, 2024 •

edited

Loading

DrahtBot commented Aug 22, 2024 •

edited

Loading

virtu commented Aug 23, 2024 •

edited

Loading

jonatack commented Aug 26, 2024 •

edited

Loading

jonatack commented Aug 26, 2024 •

edited

Loading

virtu commented Aug 27, 2024 •

edited

Loading