BIP 155: addrv2 BIP proposal #766

laanwj · 2019-02-27T09:33:41Z

Proposal for wider v2 address messages.

Last time this went around (as a github gist) there were some considerations still to be discussed, for lack of a working mailing list I'll post them here:

Considerations

''Client MAY store and gossip address formats that they do not know about'': does it ever make sense to gossip addresses outside a certain overlay network? Say, I2P addresses to Tor? I'm not sure. Especially for networks that have no exit nodes as there is no overlap with the globally routed internet at all.
Lower precision of time field? seconds precision seems overkill, and can even be harmful, there have been attacks that exploited high precision timestamps for mapping the current network topology.
- (gmaxwell) If you care about space time field could be reduced to 16 bits easily. Turn it into a "time ago seen" quantized to 1 hour precision. (IIRC we quantize times to 2hrs regardless).
Rolling port into addr, or making the port optional, would make it possible to shave off two bytes for address types that don't have ports (however, all of the currently listed formats have a concept of port.). It could also be an optional data item (see below).
(gmaxwell) Optional (per-service) data could be useful for various things:
- Node-flavors for striping (signalling which slice of the blocks the node has in selective pruning)
- Payload for is alternative ports for other transports (e.g. UDP ports)
- If we want optional flags. I guess the best thing would just be a byte to include the count of them, then a byte "type" for each one where the type also encodes if the payload is 0/8/16/32 bits. (using the two MSB of the type to encode the length). And then bound the count of them so that the total is still reasonably sized.

luke-jr · 2019-03-29T06:05:31Z

Assigned BIP 155

laanwj · 2019-07-17T22:39:54Z

Thanks!
Filled in the BIP number and fixed the travis issues, this should be ready for merge I think?

maflcko · 2019-07-18T12:30:57Z

bip-0155.mediawiki

+<ref>[https://bitcoin.org/en/developer-reference#addr Bitcoin Developer Reference: addr message]</ref>, with the difference that the 
+fixed 16-byte IP address is replaced by a network ID and a variable-length address, and the time and services format has been changed to VARINT.
+
+This means that the message contains a serialized <code>std::vector</code> of the following structure:


I know this is how bips were written in the past, but I'd slightly prefer not to leak cpp details or Bitcoin Core implementation details such as std::vector (can be replaced by "list" or "vector"), pchCommand (can be replaced by "message type"), and static const int GOSSIP_ADDRV2_VERSION (can be omitted without loss of information).

If someone is eager to look at the cpp implementation, there is already a section that links to it.

fine with me

I think I went with explicit C++ types here for clarity as to how things would be serialized, after all there is not really a spec of the serialization format (and how the certain structures are named, like "how to serialize a list", wouldn't "a list" be vague and ambigious?) besides the implementation

Our python implementation calls it "string" https://github.com/bitcoin/bitcoin/blob/e5abb59a9a66723dd1d9a01f65e467636eb24f2d/test/functional/test_framework/messages.py#L89

Anyway, no strong opinion 😅

string seems to be worse as it implies a human-readable text

maflcko · 2019-07-18T18:22:03Z

bip-0155.mediawiki

+Send <code>addrv2</code> messages only, and exclusively, when the peer has a certain protocol version (or higher):
+<source lang="c++">
+//! gossiping using `addrv2` messages starts with this version
+static const int GOSSIP_ADDRV2_VERSION = 70016;


I haven't put much thought into this, but could you explain the tradeoffs here a bit? I think there are two ways to announce support for a new p2p feature:

Bump the protocol version

Send a p2p message to announce your support for the feature

Bumping the minimum protocol version seems to imply that nodes would have to support all protocol features that were introduced prior (like compactblocks, feefilter, ...)

Whereas with a new protocol message, even mobile wallets could make use of the new feature?

~~it does add a new protocol message (that's the addrv2), but the version tells it that the other side understands them~~

would you prefer to add another protocol message to signal support, which is always sent after version?

No, you can keep this as is. mobile wallets tend to get their addresses from dns seeds, so this shouldn't matter.

In light of yesterday's IRC discussion and on further thought, I think support for addrv2 should be announced by a service bit. This has the following advantages:

leech nodes (nodes that only want to consume addrv2 messages, e.g. "SPV clients") can find addrv2 nodes by the service bit and they can expect them to respond to sendaddrv2 messages

address relay between address relay nodes (e.g. full nodes/Bitcoin Core nodes) can be made explicit, instead of relying on heuristics such as NODE_NETWORK. See p2p: Avoid relaying ADDR messages to SPV clients bitcoin#17163 (comment)

* address relay between address relay nodes (e.g. full nodes/Bitcoin Core nodes) can be made explicit, instead of relying on heuristics such as `NODE_NETWORK`. See [bitcoin/bitcoin#17163 (comment)](https://github.com/bitcoin/bitcoin/pull/17163#issuecomment-543072194)

To be clear, you mean address relay can be made explicit by an additional service bit, e.g. NODE_ADDR_GOSSIP or something like that?

To be clear, you mean address relay can be made explicit by an additional service bit, e.g. NODE_ADDR_GOSSIP or something like that?

Yes. Currently we use heuristics such as NODE_NETWORK for that, which is fragile and not future proof. For example, pruned nodes may not have any NODE_NETWORK* bit set. See also bitcoin/bitcoin#17194

maflcko · 2019-07-18T21:38:34Z

bip-0155.mediawiki

+|-
+| <code>VARINT</code> (unsigned)
+| <code>time</code>
+| Time that this node was last seen as connected to the network. A time in Unix epoch time format, up to 64 bits wide.


Would help to explain what "last seen" means, since it appears easy to get wrong. E.g. bitcoin/bitcoin#5860 (comment)

The meaning of this field doesn't change from the current implementation. I'm fine with adding an explanation though, if you supply it, but I'm not sure what to add here.

laanwj · 2019-07-31T15:36:11Z

Came up during discussion on IRC, need to be explicit about representing IPv4 as network ID IPV4, that the old wrapped representation is no longer allowed:

==Appendix X: IPv4 address encoding==
Clients MUST send IPv4 addresses with this network ID, with the 32-bit address in the address field.

Clients SHOULD ignore IPv4 Mapped Address (::FFFF:0:0/96) addresses on receive if they come with the IPV6 network ID.

(thanks @dongcarl)

laanwj · 2019-07-31T20:16:55Z

As for changing the signaling from protocol version-based to sending a message. I think this message, when sent from one peer to another, would signal support for the first peer to receive addrv2 messages from the other one. So something like sendaddrv2? (like bip130)

I wish we had some kind of standard message for this.

I think for simplicy of implementation we'd want to mandate that this should be sent after version but before any other (non-BIP-signalling) messages, so a connection won't switch to v2 suddenly halfway other things.

Edit2: This sendaddrv2 message could also contain wider addresses that doesn't fit in the version message. e.g. addrMe, the TorV3 address that we used to connect to the peer. The version message would contain a dummy address (localhost) then.

laanwj · 2019-08-07T20:04:35Z

There was some discussion on IRC that there should be a max cap for network addresses, but it should not be 256 bits, but more something like 512 bytes, to accommodate future extensions but have some kind of limit for DoS. Currently it says:

Field addr has a variable length, with a maximum of 32 bytes (256 bits). Clients SHOULD reject
longer addresses.

Change this to: a maximum of 512 bytes, and the whole message (not just the single address) SHOULD be rejected if it contains any larger item.

dongcarl · 2019-08-22T15:19:07Z

@laanwj Just to be clear, addrv2 entries that claims they are IPv6 but come with either the IPv4-prefix, OnionCat-prefix, or CJDNS-prefix should be ignored?

For Bitcoin Core, I'm assuming our weird g_internal_prefix would also be added to that ignore list?

dongcarl · 2019-08-23T20:03:44Z

As for changing the signaling from protocol version-based to sending a message. I think this message, when sent from one peer to another, would signal support for the first peer to receive addrv2 messages from the other one. So something like sendaddrv2? (like bip130 )

I wish we had some kind of standard message for this.

I think for simplicy of implementation we'd want to mandate that this should be sent after version but before any other (non-BIP-signalling) messages, so a connection won't switch to v2 suddenly halfway other things.

Edit2: This sendaddrv2 message could also contain wider addresses that doesn't fit in the version message. e.g. addrMe, the TorV3 address that we used to connect to the peer. The version message would contain a dummy address (localhost) then.

It would seems that that addrMe has been deprecated: bitcoin/bitcoin#8740

naumenkogs · 2019-10-22T17:24:25Z

''Client MAY store and gossip address formats that they do not know about'': does it ever make sense to gossip addresses outside a certain overlay network? Say, I2P addresses to Tor? I'm not sure. Especially for networks that have no exit nodes as there is no overlap with the globally routed internet at all.

I think relaying unknown networks is very important, because the nodes are often multi-homed. So this might be actually useful for somebody who your node thinks is simply ipv4.
Also, I believe it would be easier for an attacker to attack, if we don't relay these at all.
Lastly, depending on the implementation, this specific behavior might leak that you do support Tor, while you look ipv4 to an attacker.

There is no reason to not. If an attacker wants to spend our bandwidth, might as well flood with ipv4, we don't check if the node is alive before forwarding them.

Storing is another thing. With the current rate-limiting, the bandwidth we're spending here is pretty low. But addrman size is bounded, so I wouldn't store everything. Maybe, storing a couple exotic things might make sense if you will be switching in future.

* Increase the maximum length of an address from 32 to 64 bytes and clarify that the entire message should be rejected if it contains a longer address. (from bitcoin#766 (comment)) * Remove a contradiction about unknown address types - "MUST ignore" VS "MAY store and gossip". * Recommend gossiping addresses for unknown network ID and for networks to which the node is not currently connected to. (from bitcoin#766 (comment)) * Clarify that the entire message should be rejected if it contains an address with unexpected size (e.g. IPv4 address with length 5).

* Increase the maximum length of an address from 32 to 512 bytes and clarify that the entire message should be rejected if it contains a longer address. (from bitcoin#766 (comment)) * Remove a contradiction about unknown address types - "MUST ignore" VS "MAY store and gossip". * Recommend gossiping addresses for unknown network ID and for networks to which the node is not currently connected to. (from bitcoin#766 (comment)) * Clarify that the entire message should be rejected if it contains an address with unexpected size (e.g. IPv4 address with length 5).

dongcarl · 2020-04-10T21:39:20Z

Following up on a discussion started in #907 (comment)

Was it decided how to signal support for the new addrv2 type of messages? 3 options have been discussed:

Bump the protocol version (to 70016).
pros: simple to implement and the info will be available right after the handshake, without changing the handshake itself.
cons: the protocol version is not a bitmap, so if an implementation wants to support addrv2 then it also has to support all features prior to 70016, see also #766 (comment)

Use a new network service bit (NODE_ADDRV2 = (1 << 4)).
pros: same as 1., see also #766 (comment)
cons: hmm?

Use a new message sendaddrv2 during the handshake.
pros: can send wider addresses during handshake that would not otherwise fit in addrMe, but addrMe is deprecated. See #766 (comment)
cons: complicates and requires changes to the handshake and requires handling if the new message sendaddrv2 arrives after the handshake.

To overgeneralize, I think that all else being the same I prefer the "new message" mechanism over the "service bit" mechanism over the "protocol version" mechanism. However, the "service bit" mechanism is well-suited for making sure that certain specific node configurations can discover useful peers.

The support signaling can be broken down into 2 parts:

Something for my node to signal to peers that it wants `addrv2` messages.

This should be indicated with a sendaddrv2 message, perhaps something like I've implemented here: bitcoin/bitcoin@c0414a6#diff-eff7adeaec73a769788bb78858815c91R2099-R2102

Something for my node to signal to peers that it serves `addrv2` messages.

This should be indicated with a NODE_ADDRV2 service bit so that both:

Nodes which specify an -onlynet with post-addrv2 networks, and
Nodes (probably light/custom nodes) which only support addrv2

Can discover useful peers during DNS bootstrapping

Why not just use the `NODE_ADDRV2` service bit for both signals?

As Marco implies here, if we use the same service bit for both signals, leech nodes (which only really want addrv2 messages) will be seen as signaling that they serve addrv2 messages, which they won't.

I'd love some feedback on whether the above seem reasonable, and perhaps some feedback on wether or not -onlynet makes sense for all the NetworkIDs specified here.

vasild · 2020-04-13T12:53:10Z

I'd love some feedback on whether the above seem reasonable,

What about using a service bit (different one) also for the first capability?

The second capability is actually addr/addrv2 agnostic, isn't it? It means "I will forward addresses via gossip to my peers (as addr or addrv2 messages)". This will fit perfectly into https://github.com/bitcoin/bips/pull/17194. Maybe use a name like NODE_GOSSIP_ADDRESSES to avoid confusion that this is addr or addrv2 specific?

and perhaps some feedback on whether or not -onlynet makes sense for all the NetworkIDs specified here.

ipv4, ipv6, torv2, torv3, i2p, cjdns - I think yes, although it would be strange to use just -onlynet=torv2. Maybe most often people will use -onlynet=torv2 -onlynet=torv3. Currently the options are ipv4, ipv6 and onion, the latter could be extended to do "torv2 or torv3".

maflcko · 2020-04-13T13:53:59Z

What about using a service bit (different one) also for the first capability?

I don't remember what this post to the mailing list said: https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2019-October/017428.html, so it might prove my point invalid. But I think it said something that both those properties should be negotiated on a per-link (per-connection) basis. In which case both should be done via a new message type.

If you think of service bits as a "static global" across the whole Bitcoin network, then it makes sense to only use them for things that don't change often and don't change on a per-connection basis. E.g. NODE_WITNESS that indicates you can serialize witness data comes to my mind. Non-consensus related stuff like transaction relay or addr relay can be negotiated on a per-connection basis.

dongcarl · 2020-04-15T16:26:02Z

@vasild

The second capability is actually addr/addrv2 agnostic, isn't it? It means "I will forward addresses via gossip to my peers (as addr or addrv2 messages)". This will fit perfectly into bitcoin/bips/pull/17194. Maybe use a name like NODE_GOSSIP_ADDRESSES to avoid confusion that this is addr or addrv2 specific?

Unfortunately, it is not addr/addrv2 agnostic. Think of this scenario: I am a node that is -onlynet=torv3, when doing DNS bootstrapping, if we deploy NODE_GOSSIP_ADDRESSES and I use the service bits to filter for those, it is likely that I always connect to old nodes who have no idea about addrv2, however, if we deploy NODE_ADDRV2 and filter for that, the DNS seeds will be able to help connect me to nodes who at least claim to serve addrv2 messages.

@MarcoFalke

But I think it said something that both those properties should be negotiated on a per-link (per-connection) basis. In which case both should be done via a new message type.

I'm think that the "my node wants addrv2 messages" signaling should be done via a new message type, but not the "my node serves addrv2 messages" signaling. Mostly so that we don't break DNS bootstrapping (as detailed above) for both:

Nodes which specify an -onlynet with post-addrv2 networks, and
Nodes (probably light/custom nodes) which only support addrv2

Perhaps @naumenkogs has more thoughts.

vasild · 2020-04-15T19:55:53Z

The second capability is actually addr/addrv2 agnostic, isn't it? It means "I will forward addresses via gossip to my peers (as addr or addrv2 messages)". This will fit perfectly into bitcoin/bips/pull/17194. Maybe use a name like NODE_GOSSIP_ADDRESSES to avoid confusion that this is addr or addrv2 specific?

Unfortunately, it is not addr/addrv2 agnostic. Think of this scenario: I am a node that is -onlynet=torv3, when doing DNS bootstrapping, if we deploy NODE_GOSSIP_ADDRESSES and I use the service bits to filter for those, it is likely that I always connect to old nodes who have no idea about addrv2, however, if we deploy NODE_ADDRV2 and filter for that, the DNS seeds will be able to help connect me to nodes who at least claim to serve addrv2 messages.

I meant two service bits:

NODE_ADDRV2 - support for addrv2.
- The node can parse addrv2 messages, if it receives them.
- The node can generate addrv2 messages - if the node relays addresses to his peers and if the given peer has NODE_ADDRV2 then the address will be relayed as addrv2 message. If the peer does not have NODE_ADDRV2 then the address will be relayed encoded in the old addr format.
NODE_GOSSIP_ADDRESSES - the node would relay some of the addresses to some of its peers, as it works now. This is addr vs addrv2 agnostic and would fit perfectly to p2p: Avoid forwarding ADDR messages to SPV nodes bitcoin#17194 - we will not relay addresses to nodes that did not signal NODE_GOSSIP_ADDRESSES.

So, for example, a multi-homed ipv4 and i2p node may seek to connect to ipv4 nodes that have both capabilities in the hopes to learn more i2p addresses from them.

* Increase the maximum length of an address from 32 to 512 bytes and clarify that the entire message should be rejected if it contains a longer address. (from bitcoin#766 (comment)) * Remove a contradiction about unknown address types - "MUST ignore" VS "MAY store and gossip". * Recommend gossiping addresses for networks to which the node is not currently connected to. (from bitcoin#766 (comment)) * Clarify that the entire message should be rejected if it contains an address with unexpected size (e.g. IPv4 address with length 5).

Sjors · 2020-09-16T13:06:09Z

//! gossiping using addrv2 messages starts with this version
static const int GOSSIP_ADDRV2_VERSION = 70016;

That particular version is now used for WTXID_RELAY_VERSION and bitcoin/bitcoin#19845 proposes to OR a version flag instead:

ADDRV2_FORMAT = 0x20000000;

vasild · 2020-09-16T13:18:13Z

@Sjors, #907 updates the signalling mechanism. Here is a render preview of it: https://github.com/bitcoin/bips/blob/350189a/bip-0155.mediawiki#signaling-support-and-compatibility

addrv2 BIP proposal

ec8b400

luke-jr added the New BIP label Mar 29, 2019

luke-jr changed the title ~~addrv2 BIP proposal~~ BIP 155: addrv2 BIP proposal Mar 29, 2019

laanwj mentioned this pull request Jun 15, 2019

[Feature] Support i2p bitcoin/bitcoin#16216

Closed

laanwj added 3 commits July 18, 2019 00:17

bip155: fill in BIP number

34534b5

bip155: Fix comments URL

e993478

bip155: Add table entry

f517419

maflcko reviewed Jul 18, 2019

View reviewed changes

braydonf mentioned this pull request Jul 18, 2019

Failed to use TOR-proxy bcoin-org/bcoin#206

Closed

maflcko reviewed Jul 18, 2019

View reviewed changes

luke-jr merged commit 095087b into bitcoin:master Jul 23, 2019

dongcarl mentioned this pull request Aug 28, 2019

[WIP] Add support for addrv2 (BIP155) bitcoin/bitcoin#16748

Closed

1 task

vasild mentioned this pull request Apr 10, 2020

BIP155: include changes from followup discussions #907

Merged

BIP 155: addrv2 BIP proposal #766

BIP 155: addrv2 BIP proposal #766

Uh oh!

Conversation

laanwj commented Feb 27, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Considerations

Uh oh!

luke-jr commented Mar 29, 2019

Uh oh!

laanwj commented Jul 17, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

laanwj Jul 18, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

laanwj Jul 31, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

laanwj Jul 31, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

laanwj commented Jul 31, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

laanwj commented Jul 31, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

laanwj commented Aug 7, 2019

Uh oh!

dongcarl commented Aug 22, 2019

Uh oh!

dongcarl commented Aug 23, 2019

Uh oh!

naumenkogs commented Oct 22, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dongcarl commented Apr 10, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Something for my node to signal to peers that it wants addrv2 messages.

Something for my node to signal to peers that it serves addrv2 messages.

Why not just use the NODE_ADDRV2 service bit for both signals?

Uh oh!

vasild commented Apr 13, 2020

Uh oh!

maflcko commented Apr 13, 2020

Uh oh!

dongcarl commented Apr 15, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vasild commented Apr 15, 2020

Uh oh!

Sjors commented Sep 16, 2020

Uh oh!

vasild commented Sep 16, 2020

Uh oh!

Reviewers

Assignees

laanwj commented Feb 27, 2019 •

edited

Loading

laanwj Jul 18, 2019 •

edited

Loading

laanwj Jul 31, 2019 •

edited

Loading

laanwj Jul 31, 2019 •

edited

Loading

laanwj commented Jul 31, 2019 •

edited

Loading

laanwj commented Jul 31, 2019 •

edited

Loading

naumenkogs commented Oct 22, 2019 •

edited

Loading

dongcarl commented Apr 10, 2020 •

edited

Loading

Something for my node to signal to peers that it wants `addrv2` messages.

Something for my node to signal to peers that it serves `addrv2` messages.

Why not just use the `NODE_ADDRV2` service bit for both signals?

dongcarl commented Apr 15, 2020 •

edited

Loading