Skip to content

Conversation

@laanwj
Copy link
Member

@laanwj laanwj commented Mar 31, 2021

Closes #20239 and mitigates my node's problem in #21351.

  • Add a few hardcoded seeds for TorV3

    • As the bitcoin-seeder doesn't collect TorV3 addresses yet, I have extracted these from my own node using a script and added them manually. This is intended to be a temporary stop gap until 22.0's seeds update.
  • Change hardcoded seeds to variable length BIP155 binary format.

    • It is stored as a single serialized blob in a byte array, instead of pseudo-IPv6 address slots. This is more flexible and, assuming most of the list is IPv4, more compact.
    • Only the (networkID, addr, port) subset (CService). Services and time are construed on the fly as before.
  • Change input format for nodes_*.txt.

    • Drop legacy 0xAABBCCDD format for IPv4. It is never generated by makeseeds.py.
    • Stop interpreting lack of port as default port, interpret it as 'no port', to accomodate I2P and other port-less protocols (not handled in this PR). An explicit port is always generated by makeseeds.py so in practice this makes no difference right now.

A follow-up to this PR could do the same for I2P.

@jonatack
Copy link
Member

Concept ACK, will review

@laanwj laanwj force-pushed the 2021-03-torv3-hardcoded-seeds branch from 74d8869 to 1007a73 Compare March 31, 2021 11:58
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I ran https://gist.github.com/laanwj/b3d7b01ef61ce07c2eff0a72a6b90183 on my node:

  • Extra addresses I have that are not on this list:
2g5qfdkn2vvcbqhzcyvyiitg4ceukybxklraxjnu7atlhd22gdwywaid.onion:8333
2jmtxvyup3ijr7u6uvu7ijtnojx4g5wodvaedivbv74w4vzntxbrhvad.onion:8333
37m62wn7dz3uqpathpc4qfmgrbupachj52nt3jbtbjugpbu54kbud7yd.onion:8333
7cgwjuwi5ehvcay4tazy7ya6463bndjk6xzrttw5t3xbpq4p22q6fyid.onion:8333
fjdyxicpm4o42xmedlwl3uvk5gmqdfs5j37wir52327vncjzvtpfv7yd.onion:8333
fzhn4uoxfbfss7h7d6ffbn266ca432ekbbzvqtsdd55ylgxn4jucm5qd.onion:8333
ifdu5qvbofrt4ekui2iyb3kbcyzcsglazhx2hn4wfskkrx2v24qxriid.onion:8333
m7cbpjolo662uel7rpaid46as2otcj44vvwg3gccodnvaeuwbm3anbyd.onion:8333
owjsdxmzla6d7lrwkbmetywqym5cyswpihciesfl5qdv2vrmwsgy4uqd.onion:8333
vi5bnbxkleeqi6hfccjochnn65lcxlfqs4uwgmhudph554zibiusqnad.onion:8333
  • Addresses that are on this list, but not on my node:
7pyrpvqdhmayxggpcyqn5l3m5vqkw3qubnmgwlpya2mdo6x7pih7r7id.onion:8333
b64xcbleqmwgq2u46bh4hegnlrzzvxntyzbmucn3zt7cssm7y4ubv3id.onion:8333
fpz6r5ppsakkwypjcglz6gcnwt7ytfhxskkfhzu62tnylcknh3eq6pad.onion:8333
gxo5anvfnffnftfy5frkgvplq3rpga2ie3tcblo2vl754fvnhgorn5yd.onion:8333
itz3oxsihs62muvknc237xabl5f6w6rfznfhbpayrslv2j2ubels47yd.onion:8333
lrjh6fywjqttmlifuemq3puhvmshxzzyhoqx7uoufali57eypuenzzid.onion:8333
opnyfyeiibe5qo5a3wbxzbb4xdiagc32bbce46owmertdknta5mi7uyd.onion:8333
q7kgmd7n7h27ds4fg7wocgniuqb3oe2zxp4nfe4skd5da6wyipibqzqd.onion:8333
sys54sv4xv3hn3sdiv3oadmzqpgyhd4u4xphv4xqk64ckvaxzm57a7yd.onion:8333
tddeij4qigtjr6jfnrmq6btnirmq5msgwcsdpcdjr7atftm7cxlqztid.onion:8333
xqt25cobm5zqucac3634zfght72he6u3eagfyej5ellbhcdgos7t2had.onion:8333

I can connect to all of them except opnyfyeiibe5qo5a3wbxzbb4xdiagc32bbce46owmertdknta5mi7uyd.onion:8333.

  • Addresses on both this list and my node:
5g72ppm3krkorsfopcm2bi7wlv4ohhs4u4mlseymasn7g7zhdcyjpfid.onion:8333
ejxefzf5fpst4mg2rib7grksvscl7p6fvjp6agzgfc2yglxnjtxc3aid.onion:8333
rp7k2go3s5lyj3fnj6zn62ktarlrsft2ohlsxkyd7v3e3idqyptvread.onion:8333

It makes things a bit easier if the list is sorted.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you. I pushed a squashme commit to add your nodes and sort the list. I have also adapted the script to print a sorted list.

Copy link
Contributor

@vasild vasild left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(reviewed up to f3e26a167)

@DrahtBot
Copy link
Contributor

DrahtBot commented Mar 31, 2021

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

No conflicts as of last run.

Copy link
Member

@sipa sipa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Concept/approach/code review ACK. I'm ok with squashing.

@laanwj
Copy link
Member Author

laanwj commented Apr 1, 2021

I'm ok with squashing.

Right i'll squash the squashme: commits into the appropriate top-level commits once it seems the (first round of) review is over. I think this is somewhat friendlier on reviewers in progress than squashing all the time.

Copy link
Contributor

@vasild vasild left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ede28e76c6ddf5da5518d834c3c3a18ce69cfb6f looks good, modulo the squashing.

I verified that contrib/seeds/generate-seeds.py generates the same src/chainparamsseeds.h as included in this PR.

Also, the added seeds (as reported in the newly added log printout) are the same as the ones in contrib/seeds/nodes_main.txt (serialize from py and deserialize in cpp work as expected).

src/net.cpp Outdated
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This printed 1188 lines to the log. I think it is ok.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AHHH good catch, this is just to debug / test, it's convenient to be able to check if it matches the generated data. I'm not sure there is any point to it for end users 😄 But if you think it's ok we can leave it in.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are also a lot of IP ... mapped to AS0 belongs to new bucket ... messages printed by -debug=net. ~0 on this new printout.

Copy link
Contributor

@vasild vasild left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a9ee5683ce4013904f6e53f29e37632969926b0d looks good, modulo the squashing.

Copy link
Member

@jonatack jonatack left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK modulo squash. I'm running the gist (had to update the file perms for it to run) and it's taking a while as I have many (nearly 1835 and increasing) tor v3 addresses; LMK if the output would be useful. A couple nits below to pick/choose/ignore.

@jonatack
Copy link
Member

jonatack commented Apr 3, 2021

I'm running the gist (had to update the file perms for it to run) and it's taking a while as I have many (nearly 1835 and increasing) tor v3 addresses

The gist script finished. It checked 1836 v3 addresses and returned 1523 of them, sorted. I saved it in a text file if useful.

@jonatack
Copy link
Member

jonatack commented Apr 4, 2021

Update:

$ ./src/bitcoin-cli -addressinfo
{
  "addresses known": {
    "ipv4": 185,
    "ipv6": 0,
    "torv2": 5597,
    "torv3": 2220,
    "i2p": 7,
    "total": 8009
  }
}

@maflcko
Copy link
Member

maflcko commented Apr 4, 2021

Concept ACK. I'll pick up #20648 again after this is merged.

@laanwj
Copy link
Member Author

laanwj commented Apr 5, 2021

The gist script finished. It checked 1836 v3 addresses and returned 1523 of them, sorted. I saved it in a text file if useful.

Maybe! The scope of this PR is just to get things started in that regard, not to add as many nodes as possible. We'd have to decide on how many, and which ones to use; as a guide makeseeds.py limits to 512 seeds per net, but how do we want to divide this over TorV2 and TorV3?

We could do that kind of process to create a list of I2P and Tor seed nodes for 0.22.
I've openened sipa/bitcoin-seeder#92 but am not sure that is the best approach. I mean it's not rational for the DNS seeder's crawler to collect addresses it can't do anything with.

The advantage of the dedicated crawler approach to my gist is that it applies some more stringent conditions to the nodes. For example it samples uptime and gets the node's version information. But maybe that can be added, too.

./src/bitcoin-cli -addressinfo

Nice!

@laanwj laanwj force-pushed the 2021-03-torv3-hardcoded-seeds branch from a9ee568 to b2ee8b2 Compare April 5, 2021 12:03
@laanwj
Copy link
Member Author

laanwj commented Apr 5, 2021

Squashed all the squashme's into the appropriate commits a9ee5683ce4013904f6e53f29e37632969926b0d..b2ee8b207de78f03356905bd60b7b00b6f49c252

The only overall differences are @jonatack's latest comments.

@jonatack
Copy link
Member

jonatack commented Apr 5, 2021

ACK b2ee8b2

Tested hitting ConvertSeeds() on mainnet and testnet.

@laanwj laanwj merged commit 9be7fe4 into bitcoin:master Apr 6, 2021
@laanwj laanwj mentioned this pull request Apr 6, 2021
sidhujag pushed a commit to syscoin/syscoin that referenced this pull request Apr 6, 2021
laanwj added a commit that referenced this pull request May 4, 2021
142e2da net: add I2P seeds to chainparamsseeds (Jon Atack)
e01f173 contrib: add a few I2P seed nodes (Jon Atack)
ea269c7 contrib: parse I2P addresses in generate-seeds.py (Jon Atack)

Pull request description:

  Follow-up to #21560 that updated the fixed seeds infra for BIP155 addresses and then added Tor v3 ones:

  - Update contrib/generate-seeds.py to parse I2P addresses

  - Add a few I2P nodes to contrib/seeds/nodes_main.txt

  - Run generate-seeds.py and add the I2P seeds to chainparamsseeds.h

  Reviewers, see contrib/seeds/README.md for more info and feel free to use the following CLI one-liner to check for and propose additional seeds for contrib/seeds/nodes_main.txt. You can also see how many I2P peers your node knows with cli -addrinfo.

  ```rake
  bitcoin-cli getnodeaddresses 0 | jq '.[] | (select(.address | contains(".b32.i2p"))) | .address' | sort
  ```

  I verified the I2P addresses are correctly BIP155-serialized/deserialized by building with all seeds removed from chainparamsseeds.h except those added here, restarting with `-datadir=newdir -dnsseed=0` and running rpc ` getnodeaddresses 0` that initially returns only the new I2P addresses.

ACKs for top commit:
  laanwj:
    ACK 142e2da
  vasild:
    ACK 142e2da

Tree-SHA512: 040576012d5f1f034e2bd566ad654a6fdfd8ff7f6b12fa40c9fda1e948ebf8417fcea64cfc14938a41439370aa4669bab3e97274f9d4f9a6906fa9520afa9cf8
laanwj added a commit that referenced this pull request May 27, 2021
6fe0516 contrib: add torv3 seed nodes for testnet, drop v2 ones (Jon Atack)

Pull request description:

  Replace the ancient (2015) Tor V2 hardcoded seeds with new Tor V3 ones. This needs to be done before 0.22 to make sure onion-only testnet nodes can still connect to the network. Continues #21560.

  Ways to test:
  - Re-generate ` src/chainparamsseeds.h` with `cd contrib/seeds && python3 generate-seeds.py . > ../../src/chainparamsseeds.h`, check if git tree stays the same.
  - Create a new testnet node with `bitcoind -testnet -onlynet=onion -proxy=127.0.0.1:9050` (or delete `~/.bitcoin/testnet3/peers.dat`), check if it is able to connect to the network and get blocks.
  - Check if the addresses are connectable for ex.:
  ```python3
  #!/usr/bin/env python3
  import subprocess
  with open('contrib/seeds/nodes_test.txt') as f:
      for line in (line for line in (line.rstrip().split('#', 1)[0] for line in f) if line):
          subprocess.call(["nc", "-v", "-x", "127.0.0.1:9050", "-z"] + line.split(':'))
  ```

  Thanks to jonatack for providing the list.

ACKs for top commit:
  jonatack:
    ACK 6fe0516

Tree-SHA512: 61bfdb44dfab9d02b75e5cb06c089a3b1a1fe7134875e1d09166c4116e961d809aa25422fe03f068876e9423b571ecc4a0c7a7eeacba4aac3b2768717f3ee6d6
furszy added a commit to PIVX-Project/PIVX that referenced this pull request Aug 16, 2021
4aec935 net: Deserialize hardcoded seeds from BIP155 blob (Fuzzbawls)
a245b2f contrib: generate-seeds.py generates output in BIP155 format (Fuzzbawls)

Pull request description:

  Adapted from bitcoin#21560, this adds support for Tor v3 hardcoded seed nodes.

  Upstream PR description:

  > * Add a few hardcoded seeds for TorV3
  >   * As the bitcoin-seeder doesn't collect TorV3 addresses yet, I have extracted these from my own node using a [script](https://gist.github.com/laanwj/b3d7b01ef61ce07c2eff0a72a6b90183) and added them manually. This is intended to be a temporary stop gap until 22.0's seeds update.
  > * Change hardcoded seeds to variable length BIP155 binary format.
  >   * It is stored as a single serialized blob in a byte array, instead of pseudo-IPv6 address slots. This is more flexible and, assuming most of the list is IPv4, more compact.
  >   * Only the (networkID, addr, port) subset (CService). Services and time are construed on the fly as before.
  > * Change input format for nodes_*.txt.
  >   * Drop legacy 0xAABBCCDD format for IPv4. It is never generated by makeseeds.py.
  >   * Stop interpreting lack of port as default port, interpret it as 'no port', to accomodate I2P and other port-less protocols (not handled in this PR). An explicit port is always generated by makeseeds.py so in practice this makes no difference right now.

  Marked as WIP for now because we need more Tor v3 nodes on mainnet for this to really be of use to end users, but this is currently testable on testnet as we already have a number of Tor v3 capable nodes there (which have been added to the hardcoded seeds here)

ACKs for top commit:
  random-zebra:
    ACK 4aec935
  furszy:
    ACK 4aec935

Tree-SHA512: 5abf2faa49ca511001589d950fa95a6f4c346a391ab0b31b3137b962765dc5a5ba343aa4c7b7fa0d7be931d4ec4802d9c09ec9b5dd2ef90143d4623162da14af
@bitcoin bitcoin locked as resolved and limited conversation to collaborators Aug 18, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Hardcoded seeds for Torv3 addresses

6 participants