Skip to content

Conversation

@laanwj
Copy link
Member

@laanwj laanwj commented Oct 25, 2020

Stats:

  IPv4   IPv6  Onion Pass
426728  59523   7900 Initial
426728  59523   7900 Skip entries with invalid address
426728  59523   7900 After removing duplicates
426727  59523   7900 Skip entries from suspicious hosts
123226  51785   7787 Enforce minimal number of blocks
121710  51322   7586 Require service bit 1
  4706   1427   3749 Require minimum uptime
  4124   1098   3681 Require a known and recent user agent
  4033   1075   3681 Filter out hosts with multiple bitcoin ports
   512    140    512 Look up ASNs and limit results per ASN and per net

I've credited RandyMcMillan for the first commit because of #20190.

There are at least enough onions this time! Number of IPv6 nodes that pass all the requirements seems similar to last time in #18506.

For the next major release we'll want TORv3 hardcoded peers as well. This makes no sense now as there are hardly any. But it'd make sense to think about how to collect them because they cannot come from the DNS seeds.

Reviewing

2020-10-28 12:04:45     jnewbery  wumpus: Do you have any suggestions for how to review #20237 ?
2020-10-28 12:28:37     wumpus  jnewbery: previous PRs like it might be a guide there (#18506, #16999), e.g. people could try to repeat the last step in https://github.com/bitcoin/bitcoin/tree/master/contrib/seeds#seeds and see if it ends up with the same .h file, you could also repeat the entire process but as the list of peers from the seeder will be different every time that will give a (slightly, hopefully)
2020-10-28 12:28:37     wumpus  different output
2020-10-28 12:49:40     wumpus  testing what part of the peers are connectable is also useful
2020-10-28 12:51:05     wumpus  or to go deeper, whether most part of the nodes are 'good nodes' and not say spy nodes, but i don't know what means of testing

RandyMcMillan and others added 2 commits October 25, 2020 14:08
Stats:

```
  IPv4   IPv6  Onion Pass
426728  59523   7900 Initial
426728  59523   7900 Skip entries with invalid address
426728  59523   7900 After removing duplicates
426727  59523   7900 Skip entries from suspicious hosts
123226  51785   7787 Enforce minimal number of blocks
121710  51322   7586 Require service bit 1
  4706   1427   3749 Require minimum uptime
  4124   1098   3681 Require a known and recent user agent
  4033   1075   3681 Filter out hosts with multiple bitcoin ports
   512    140    512 Look up ASNs and limit results per ASN and per net
```
@laanwj laanwj added the P2P label Oct 25, 2020
@laanwj laanwj added this to the 0.21.0 milestone Oct 25, 2020
@Saibato
Copy link
Contributor

Saibato commented Oct 25, 2020

This makes no sense now as there are hardly any

There are many nodes that have there onion v3 address linked to 8333 created in /etc/tor/torrc but probably won't tell and stay onlynet onion or don't enable service calls port 9051 and cant use the additional v3 support from 0.21 by auto creating onions anyway.

But since almost all old nodes versions can do connect and addnode with long v3 address since years, if thre Tor software is above Tor 0.3.2.2-alpha (October 2017 ), it might be useful to have a short list core approved list they could add manual to there old nodes,

And btw. why not add the Tor v3 addr from the known seeders ( if they have Tor nodes ) in the vseeds list, i.e. when the user selects onlynet=onion to make sure we don't seed by ADDR messages via Tor exit nodes and stay at least for virgin bootstrap in Tor, i somewhere posted a patch for chainparms.cpp but forgot where.
That way. nodes with 0.21 update would fast learn from the seeders ADDRv2 messages new v3 and stay in pure in Tor if they wish.
If u like i can dig for that?

diff --git a/src/chainparams.cpp b/src/chainparams.cpp
index ffd2076c9..4903e3bda 100644
--- a/src/chainparams.cpp
+++ b/src/chainparams.cpp
@@ -10,6 +10,7 @@
 #include <tinyformat.h>
 #include <util/system.h>
 #include <util/strencodings.h>
+#include <util/system.h>
 #include <versionbitsinfo.h>
 
 #include <assert.h>
@@ -113,6 +114,7 @@ public:
         // This is fine at runtime as we'll fall back to using them as an addrfetch if they don't support the
         // service bits we want, but we should get them updated to support all service bits wanted by any
         // release ASAP to avoid it where possible.
+        if (gArgs.GetArg("-onlynet", "") != "onion") {
         vSeeds.emplace_back("seed.bitcoin.sipa.be"); // Pieter Wuille, only supports x1, x5, x9, and xd
         vSeeds.emplace_back("dnsseed.bluematt.me"); // Matt Corallo, only supports x9
         vSeeds.emplace_back("dnsseed.bitcoin.dashjr.org"); // Luke Dashjr
@@ -122,6 +124,11 @@ public:
         vSeeds.emplace_back("seed.bitcoin.sprovoost.nl"); // Sjors Provoost
         vSeeds.emplace_back("dnsseed.emzy.de"); // Stephan Oeste
         vSeeds.emplace_back("seed.bitcoin.wiz.biz"); // Jason Maurice
+        };
+
+        if (gArgs.GetArg("-onlynet", "") == "onion") {
+        vSeeds.emplace_back("seedersv3onionaddressasdasdasdasdad.onion:8333"); // Add seeder onion v3 address
+       // since we can not seed over Tor UDP DNS and call anyway the first node from the return +       // the exit socket  the exit Tor node give us for the FQDN of normal seeders, so why not call    
+      //if onlynet=onion the Tor seeder node and gossip from ADDRv2 or ADDR,
+        }
 
         base58Prefixes[PUBKEY_ADDRESS] = std::vector<unsigned char>(1,0);
         base58Prefixes[SCRIPT_ADDRESS] = std::vector<unsigned char>(1,5);

@Saibato
Copy link
Contributor

Saibato commented Oct 25, 2020

concept ptACK
Tested only the Tor nodes and so far all on this new list encoded and decoded at boostrap and where except two reachable.

Tyi, those two on the list

4vorvtoyegh4zbvr.onion 8333
h6a32n4blbwwyn4d.onion 8333

i was unable to reach.

@laanwj
Copy link
Member Author

laanwj commented Oct 27, 2020

@Saibato Thanks for testing! Yes, I'd assume a part of them will be unreachable at any single time, I'm surprised the other 510 were connectable.

@Saibato
Copy link
Contributor

Saibato commented Oct 27, 2020

I'm surprised the other 510 were connectable

Me 2 , i did run the test twice, at first thought also impossible, but that was the case with a ten second delay between next address connect in the list.
But its v2 so there could be nodes doubles we wont be aware with the easy connect 8333 check i did.

Btw. a big improvement compared to the 3 working fixed that we only have now in 0.20

@laanwj laanwj mentioned this pull request Oct 29, 2020
11 tasks
@jonatack
Copy link
Member

jonatack commented Nov 2, 2020

The changes look fine. I'm now running the instructions in contrib/seeds/README.md to see what I get. Edit: just saw the review tips in the PR description.

  IPv4   IPv6  Onion Pass                                               
427708  59843   8114 Initial
427708  59843   8114 Skip entries with invalid address
427708  59843   8114 After removing duplicates
427707  59843   8114 Skip entries from suspicious hosts
124199  52104   8001 Enforce minimal number of blocks
122607  51619   7794 Require service bit 1
  4769   1456   3830 Require minimum uptime
  4180   1126   3762 Require a known and recent user agent
  4089   1103   3762 Filter out hosts with multiple bitcoin ports
   512    143    512 Look up ASNs and limit results per ASN and per net

Ran makeseeds.py a second time to see the variance.

  IPv4   IPv6  Onion Pass                                               
427693  59849   8131 Initial
427693  59849   8131 Skip entries with invalid address
427693  59849   8131 After removing duplicates
427692  59849   8131 Skip entries from suspicious hosts
124185  52110   8018 Enforce minimal number of blocks
122597  51627   7810 Require service bit 1
  4752   1456   3842 Require minimum uptime
  4165   1126   3774 Require a known and recent user agent
  4076   1103   3774 Filter out hosts with multiple bitcoin ports
   512    147    512 Look up ASNs and limit results per ASN and per net

Copy link
Member

@jonatack jonatack left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK 6866259

  • reviewed the changes locally
  • ran python3 generate-seeds.py . > ../../src/chainparamsseeds.h from /contrib/seeds on top of this patch and it made no change
  • ran all the steps in /contrib/seeds/README.md, which created this diff: jonatack@084717e
  • returned to the original patchset and repeated the generate seeds step again to be sure

@vasild
Copy link
Contributor

vasild commented Nov 3, 2020

I confirm that running python3 generate-seeds.py . > ../../src/chainparamsseeds.h locally generates identical result.

Re-downloading the seeds and running python3 makeseeds.py < seeds_main.txt > nodes_main.txt changes nodes_main.txt about 12% (nodes_main.txt is 1167 lines and that command removed 137 lines and added 140).

Maybe as a future improvement we may devise some deterministic way to generate the seeds, so that everybody can generate seeds.txt and confirm http://bitcoin.sipa.be/seeds.txt.gz is not compromised. Or at least that there is a significant overlap between that file from bitcoin.sipa.be and a locally generated.

@laanwj
Copy link
Member Author

laanwj commented Nov 3, 2020

I confirm that running python3 generate-seeds.py . > ../../src/chainparamsseeds.h locally generates identical result.

@jonatack @vasild Thanks for checking !

Maybe as a future improvement we may devise some deterministic way to generate the seeds, so that everybody can generate seeds.txt and confirm http://bitcoin.sipa.be/seeds.txt.gz is not compromised. Or at least that there is a significant overlap between that file from bitcoin.sipa.be and a locally generated.

Yes, maybe.
Given the same seeds.txt.gz it should be deterministic already. Well, I guess the ASN mapping could change…
I could upload mine somewhere but I don't think it would prove anything. The thing is that the seeds.txt.gz on the server changes continuously, and there is no way to prove that mine was ever the current state there and hasn't been tampered with.

The topic of improvements to the seed generation comes up every time just before the release and then dies down immediately after it. I'm happy for someone to put effort in this though. See also #17020 which ahs quite the TODO list alrady 😄

@laanwj laanwj merged commit 95bde34 into bitcoin:master Nov 3, 2020
laanwj added a commit that referenced this pull request Nov 19, 2020
961f148 doc: update contrib/seeds/README dnspython installation info (Jon Atack)
dd7b5f4 script: fix deprecation warning in makeseeds.py (Jon Atack)

Pull request description:

  Seen while reviewing #20237.

  1. Fix a deprecation warning in `contrib/seeds/makeseeds.py`
  ```
      makeseeds.py:139: DeprecationWarning: please use dns.resolver.resolve() instead
        asn = int([x.to_text() for x in dns.resolver.query('.'.join(
  ```
    - Per https://dnspython.readthedocs.io/en/latest/whatsnew.html, `dns.resolver.query()` was deprecated in `dnspython` version 2.0.0.

    - See https://dnspython.readthedocs.io/en/latest/resolver-class.html for more info on the resolver class.

  2. Update the `dnspython` dependency installation instructions in `contrib/seeds/README`

    - The markdown rendering can be seen here: https://github.com/jonatack/bitcoin/tree/contrib-seeds-fixups/contrib/seeds

ACKs for top commit:
  laanwj:
    code review ACK 961f148

Tree-SHA512: f9c4f318a1a0d35b8de147d24b72c534a1f58eece31e7cfa00b4149a63b6a618d8ca0312f52fd8056f3c645cf2ee68574ca02319fddffdad919a70cd33395d33
sidhujag pushed a commit to syscoin/syscoin that referenced this pull request Nov 19, 2020
961f148 doc: update contrib/seeds/README dnspython installation info (Jon Atack)
dd7b5f4 script: fix deprecation warning in makeseeds.py (Jon Atack)

Pull request description:

  Seen while reviewing bitcoin#20237.

  1. Fix a deprecation warning in `contrib/seeds/makeseeds.py`
  ```
      makeseeds.py:139: DeprecationWarning: please use dns.resolver.resolve() instead
        asn = int([x.to_text() for x in dns.resolver.query('.'.join(
  ```
    - Per https://dnspython.readthedocs.io/en/latest/whatsnew.html, `dns.resolver.query()` was deprecated in `dnspython` version 2.0.0.

    - See https://dnspython.readthedocs.io/en/latest/resolver-class.html for more info on the resolver class.

  2. Update the `dnspython` dependency installation instructions in `contrib/seeds/README`

    - The markdown rendering can be seen here: https://github.com/jonatack/bitcoin/tree/contrib-seeds-fixups/contrib/seeds

ACKs for top commit:
  laanwj:
    code review ACK 961f148

Tree-SHA512: f9c4f318a1a0d35b8de147d24b72c534a1f58eece31e7cfa00b4149a63b6a618d8ca0312f52fd8056f3c645cf2ee68574ca02319fddffdad919a70cd33395d33
@bitcoin bitcoin locked as resolved and limited conversation to collaborators Feb 15, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants