The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

• #26438 (Remove mempoolfullrbf option by sdaftuar)
• #25600 (p2p: Advertise NODE_FULL_RBF and connect to 4 outbound full-rbf peers if -mempoolfullrbf is set by ariard)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

ACK fa1fff4

Might be worthwhile making the default be true for test chains -- make it easier to observe full rbf behaviour on testnet/signet well in advance of actually enabling it on mainnet?

I think it shows good faith for this PR to exist, but I don't know how well it would address the concerns if at all.

Blaming the default false -mempoolfullrbf option for a full RBF network would be holding Bitcoin Core developers responsible for the decisions of individual node operators and miners. I don't think having the option (again, default false) can directly cause a full RBF network, and likewise, I don't think removing this option removes the "risk" of a full RBF network. Said "risk" is also a misunderstanding, but that's a different conversation. AFAIK Bitcoin Core isn't doing any "advertising" or encouraging users to use this config option, it just exists. Any user that wanted full RBF (e.g. because they want more fees) for the past 7 years could have run a patch or Knots or something else.

NACK

This is a default false option. There is likely to be a negligible difference in the number of nodes relaying full rbf transactions regardless of whether this option exists given it is default false (as few people even bother to change the defaults). Furthermore, there are other nodes that do full rbf (and have existed for nearly a decade) in addition to the fact that node operators could still revert this change. I would expect the number of people who would do such a revert is on par with the number of people who would explicitly set the option, namely very few.

NACK

We've been talking about full-rbf for seven years. It is well-established that zero-conf subsists on nobody deciding to try to sidestep it, and it has been long communicated that anyone using zero-conf should be aware that it is a bad practice.

Not having full replaceability is causing a variety of issues and we are at a point where instant payment options exist that can be used where people rely on quick payments. This concrete change has been discussed for the last 18 months, I don't see how it comes as a surprise.

I would posit that now is an excellent time to start thinking about how a business is going to work in a full-rbf environment.

It should be noted that current mempoolfullrbf setting default to false, therefore the existence of full-rbf transaction-relay paths during the coming months rely solely on a significant number of full-node users deviating from the default. Social inertia being what it is, I think zero conf services operators have few months of buffer before there is effective full-rbf transaction-relay paths. Moreover it would require few miners turning on mempoolfullrbf. In anyway, from my understanding the operational risk sounds low enough to not justify neutralizing #25353, which would have the downside to set a low-grounded precedent in our release process.

I know this is closed, but I think a partial reversion of #25353 is warranted given:

1. There's a reasonable argument that the inclusion of the option may significantly change network behavior in a way that would be harmful to some current users. Specifically, that it only takes one large mining pool with a publicly known IP to enable full RBF to make it practical to replace any pending transaction. Providing a configuration option in Bitcoin Core that miners can easily use without obtaining third-party patches or running alternative node software significantly lowers the barrier to enabling full RBF on a mining pool's backend node.
2. That the above risk was not considered in the discussion of Add a -mempoolfullrbf node setting #25353, so it's possible some or all of its ACKs were based on an incomplete consideration of its consequences.
3. That the above risk was not adequately communicated to the affected users six months in advance of the change (which is the project's usual non-emergency deprecation period).

I think that we have a chance through partially reverting #25353 to reconsider the benefit of a default-off option and better communicate to users the project's plan for future replacement behavior, giving them at least the customary six months to prepare for a major non-emergency change in Bitcoin Core.

Definitely willing to reconsider a closed PR/revert something we ACKed if it makes sense

There's a reasonable argument that the inclusion of the option may significantly change network behavior in a way that would be harmful to some current users. Specifically, that it only takes one large mining pool with a publicly known IP to enable full RBF to make it practical to replace any pending transaction.

I understand this argument theoretically, but haven't found any evidence suggesting that any miner will do so and on what timescale. I've been trying to ask miners what they plan to do wrt full RBF but haven't had much luck so far. Will keep trying.

The only thing evident to me so far is that miners employ the policy they want, even if it means running a patch instead of Bitcoin Core. In that case, if miners want to use full RBF and do, mempools requiring signaling will blind their users to higher-feerate replacements, potentially causing them to get double-spent. And so I imagine we'd need to emergency-add a -mempoolfullrbf option then. That's one of the reasons I think it makes more sense to keep the option than not.

Given that the risk is speculative, would one solution be to add a "use caution!" to the helpstring, or warn when a user changes the default? Or add a section in the release notes?

In that case, if miners want to use full RBF and do, mempools requiring signaling will blind their users to higher-feerate replacements, potentially causing them to get double-spent. And so I imagine we'd need to emergency-add a -mempoolfullrbf option then. That's one of the reasons I think it makes more sense to keep the option than not.

If we were worried about that scenario (where some minority of users and miners are doing full rbf and nodes that aren't doing aren't seeing those transactions until they're mined), then the -mempoolfullrbf option isn't sufficient; you also need to be connected to other peers running the same option to see those txs before they're mined.

Providing a configuration option in Bitcoin Core that miners can easily use without obtaining third-party patches or running alternative node software significantly lowers the barrier to enabling full RBF on a mining pool's backend node.

Erecting artificial barriers to prevent or make it difficult for users to do what they want to do, is not appropriate behaviour.

On the argument the mempoolfullrbf setting may significantly increase the risk exposure of 0confs applications and service, I believe it should be weighed in the light of two factors. The first, most of 0confs applications and services announce to have additional layers of security to protect themselves against double-spends attempts [0]. AFAIU, they should provide some security against conflicting transactions, even in the presence of full-rbf nodes. The second, I empathize for the affected 0confs services/users though delaying or freezing the progress on full-rbf deployment isn't free of security consequences for contracting protocols and multi-party applications. We could have tomorrow all the coinjoins implementations censor without that much countermeasures at the client-level, at the exception of setting private-relay to miners. Lastly, in the presence of a well-reviewed and functional patch already deployed, "the cat is already out of the bag", in the sense of a miner motivated to enable full rbf would have minimal engineering effort to do so. As of today, from my understanding, engineering barriers are at the equilibrium, reverting the mempoolfullrbf setting would be more like the Bitcoin Core project throwing social opprobium on the usage. And I don't think we're all agree on here.

[0] https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2022-October/021112.html

On the second argument, while all the interdependency risks where not raised in the public discussion, especially the FX risk, the immediate concerns on 0confs applications have been underscored. Even if the full depth and width were uncertain from a protocol dev viewpoint, it should be recalled some dimensions are application specific. It is yet unclear what should be the risk scope in the design considerations of Bitcoin Core development (i.e "Is FX risk better solved at this layer, or in which proportion ?").

On the third argument, it should be noted current patch doesn't deprecate opt-in rbf as such, and the spirit design of this patch and #25600 were to offer the flexibility to the node operators to "vote" with their economic traffic. Indeed, in a presence of a "split" situation, where we have two classes of use-cases with conflicting policy requirements we might have to consider out of the sphere of Bitcoin Core decision-making what should be the policy default. If we have rough consensus on full-rbf itself, though we have more in the process of converging on a deprecation timeline, I think the conversation should be rather pursued on the mailing list or during meetings.

Thank y'all for the replies!

@glozow

I understand this argument theoretically, but haven't found any evidence suggesting that any miner will do so and on what timescale. I've been trying to ask miners what they plan to do wrt full RBF but haven't had much luck so far. Will keep trying.

Thank you for attempting to query miners! Let's imagine every miner replied "no, we don't plan to enable full RBF this release cycle". What could we do to support that decision and also provide additional assurance to services that currently expect first-seen-safe behavior? I think there are two options:

1. This PR, temporarily disable mempoolfullrbf for mainchain
2. Disable the getblocktemplate RPC on any node that has mempoolfullrbf set to true

The only thing evident to me so far is that miners employ the policy they want, even if it means running a patch instead of Bitcoin Core.

That's their right, of course, and I'd personally be happy to see them enabling full RBF on their own. I know some miners have done that in the past, most notably f2pool in 2015 IIRC.

I think an essential part of this issue is that Bitcoin Core should give appropriate notice to affected users when making changes that have a reasonable chance of disrupting existing operations. We haven't provided that notice for 24.x, and to actually provide that notice and maintain our high degree of professionalism and regard for users, all we need to do is wait six months for 25.x.

if miners want to use full RBF and do, mempools requiring signaling will blind their users to higher-feerate replacements, potentially causing them to get double-spent. And so I imagine we'd need to emergency-add a -mempoolfullrbf option then. That's one of the reasons I think it makes more sense to keep the option than not.

In addition to what @ajtowns wrote, if all we wanted to do is notify users that replacements might be circulating, we could probably pass the list of conflicting transactions when the txn-mempool-conflict error is sent via -debug=mempoolrej. That way anyone could receive notification about double spends being relayed by their peers.

Given that the risk is speculative, would one solution be to add a "use caution!" to the helpstring, or warn when a user changes the default? Or add a section in the release notes?

The people at risk aren't the people enabling the option, so I'm not sure changes to the helpstring or logging help. We should certainly provide a detailed release note about this situation no matter what action (or inaction) we decide to take.

@luke-jr

Erecting artificial barriers to prevent or make it difficult for users to do what they want to do, is not appropriate behaviour.

I agree in principle, but I also don't think it's appropriate to potentially undermine the operations of some of our users because we're unwilling to announce our intent six months in advance, as we have done historically.

@ariard

most of 0confs applications and services announce to have additional layers of security to protect themselves against double-spends attempts [0]

It seems to me that the email you linked only describes countermeasures against separate propagation of conflicting transactions and that those countermeasures would be ineffectual against a replacement that was sent more than a few seconds after the original transaction.

I empathize for the affected 0confs services/users though delaying or freezing the progress on full-rbf deployment isn't free of security consequences for contracting protocols and multi-party applications.

If removing this option will significantly delay or freeze the progress of full-RBF deployment, that seems to me to be a statement that you believe zero-conf services are at increased risk, which I think conflicts with statements made by several reviewers on the original PR about this being, "uncontroversial, given that the default is 'unchanged'".

Again, if we are changing a policy that users and businesses rely on, we should strive to provide advance warning.

a miner motivated to enable full rbf would have minimal engineering effort to do so

Patches for enabling full RBF have been available since at least 2015 and have even previously been deployed by miners. That's fine; zero-conf services and those worried about free options have always been exposed to that risk. But just because others can break an existing system doesn't mean that we should treat it as broken now.

reverting the mempoolfullrbf setting would be more like the Bitcoin Core project throwing social opprobium on the usage. And I don't think we're all agree on here.

I don't think we need to worry about people misinterpreting our actions when we can explain what we're doing, and why we're doing it, in the 24.0 release notes.

On the second argument, while all the interdependency risks where not raised in the public discussion, especially the FX risk, the immediate concerns on 0confs applications have been underscored. Even if the full depth and width were uncertain from a protocol dev viewpoint, it should be recalled some dimensions are application specific. It is yet unclear what should be the risk scope in the design considerations of Bitcoin Core development (i.e "Is FX risk better solved at this layer, or in which proportion ?").

I read the above as you agreeing that reviewers were unaware of all the risks when they gave their ACKs. If that's the case, I don't see why it's a matter of significant debate to partially revert this unreleased PR so that it can be reconsidered with all of its consequences made explicit.

On the third argument, it should be noted current patch doesn't deprecate opt-in rbf as such

This patch makes it possible for people running unmodified Bitcoin Core to change a longstanding network policy that is used by multiple businesses serving thousands of users. Whether that's technically deprecation or not, I think it warrants giving businesses a warning about the impending change.

All: I think it would be a credit to the project and to ourselves if we remove this option for just the 24.x series and take the next six months to explain what we plan to do and how it may affect users and business. That additionally gives us time to consider the new information we've received and potentially use it to improve the deployment of full RBF (e.g. I think #26323's simultaneous activation of full-RBF is much nicer than an uncoordinated deployment, such as in maximizing the efficiency of compact block filters and erlay).

Happy to reopen this, but I'll abstain from the discussion.

Thanks @harding for articulating the reasoning.

I have tried my best to summarize people's concerns from the ML / here / #26287 / #26305 in these notes. I would understand if people changed their minds but, so far, my read is that there is strong opposition to this PR and thus would not feel comfortable merging it.

Pinging those who reviewed #25353 but haven't commented here @petertodd @instagibbs @naumenkogs @darosior @dergoegge @mzumsande. Pinging @achow101 @xekyo who nacked before this was closed and reopened. Would appreciate a comment here if your views have changed.

I've commented on the "defer" PR, I'm in general against removing options. The only caveat here being that it has yet to be in a Core release, so it's not a regression per se.

My gut worry is that we will remove the option, effectively forever, and conversations on solving the American free option problem will be the next thing we have to solve before we give users control back.

If, by merging this, we agree to an upper-bound window to reverting and switching to default true, I'm less wary of it. For example, we pick some block time 3 years in the future that will be checked into the codebase for 25.0+, then if we get later feedback that 3 years is too much for merchants, we can move it up, optionally, before 25.0 release.

Sorry if this sounds like horse trading, but there has to be a stronger motivation for removing user choices, especially when the time horizon to restoring them is unbounded.

I think it should be fine to revert on master right after this pull request is merged to clarify that the change is only temporary for the 24.0 release. (Obviously out of scope here, but the revert could be backported to 24.(0+x) if needed, based on how the discussions progress)

NACK from me as well.

NACK for the same reasons I gave above.

Thank you @harding for helping trying to make progress here.

First let me comment on two of your points:

There's a reasonable argument that the inclusion of the option may significantly change network behavior in a way that would be harmful to some current users. Specifically, that it only takes one large mining pool with a publicly known IP to enable full RBF to make it practical to replace any pending transaction.

I don't think the argument is that reasonable. I buy the part that it would then realistically be easier / more reasonable for a mining pool to enable RBF for all transactions. Even though as Antoine notes i'm not convinced removing the option would make it substantially harder for them since they can just apply the well reviewed a7f3479 patch on top of the release.
I don't buy the part that it's substantially safer to accept unconfirmed transactions as final today, due to some magic protection mechanism in place (and apparently closed source? Has anyone seen it and could vouch for its efficiency, that non-signaling RBF would break?).

so it's possible some or all of its ACKs were based on an incomplete consideration of its consequences

Mine was based on the correctness of the code plus the consequences:

• As a software project we don't take side in the "gentleman agreement" (is there any or is it just blind trust?) between businesses relying on first-seen-safe and miners of not replacing non BIP125-signaling transactions: we just provide the option to do both.
• The default for that option is conservatively set to the previous behaviour of the node. There is a reasonable argument that setting the default to true would significantly affect the network, and i don't think it should be our call.

Therefore the PR seemed pretty reasonable to me and that's why i ACKed it.

That (and another comment) said. releasing a software that could, potentially, supposedly, play a (small) role in breaking a substantial part of users of the Bitcoin network without them being apparently aware in advance [0] seems just wrong. And opposed to the philosophy of this project.

For this reason, and even though treating unconfirmed transactions as final is insane, releasing an anti-feature that purposefully prevents a valid usage of the node (and that goes against the incentives of some users of the network) not something we should do, ACK fa1fff4 for 24.0 only (ie provided we immediately revert it on master).

[0] I don't know how we could communicate to end users better really. Antoine has been exemplary in trying to motivate, explain, and champion "full" RBF to affected stakeholders. And what we are reverting isn't even some "full" RBF activation on the network but an option to a perfectly valid usage of a Bitcoin node.

I retract my NACK from above, but I would like to elaborate my stance on full-RBF in light of the continued discussion:

1. First-seen safety for transaction not signaling replaceability relies on the unstable gentlemen's agreement that all miners continue adhering to a social convention that is technically unenforceable.
2. Treating unconfirmed transactions as reliable is a bad business practice that should be discouraged and preferably end sooner rather than later. The continued development of software that relies on this false assumption is detrimental to the network.
3. Apparently some senders publicly promising that they will not attempt replacement provides a valuable social signal to some receivers, sufficient for said receivers to treat such transaction in a different, more risky manner.
4. Propping up the illusion of non-replaceable transactions complicates development of multi-party protocols. Converging on the inevitable modus operandi sooner rather than later reduces effort spent on dead-end business practices.
5. It is inevitable that full-RBF will eventually emerge on the network. When this comes about, Bitcoin Core users should have the option to align their mempool content with the expected mined blocks.
6. Even in the presence of the mempoolfullrbf startup option and eventual adoption of full RBF on the network, some senders may continue promising non-replacement and receivers may continue extending unbacked credit on basis of such promises. This may be rational, since not paying after receiving service is theft and most people do not engage in theft.
7. Miners have had the option to enact full-RBF by running alternative clients or a one-line patch for years, yet we do not see significant hashrate that engages in full-RBF. I would similarly expect miners to shy away from turning on mempoolfullrbf, to deliberately bring about the end of the gentlemen's agreement. It is reasonable to expect the situation to remain in its unstable equilibrium for the time being until some miner perceives some significant advantage or spurious reason to doing away with it.
8. I am open to removing the mempoolfullrbf option in the 24.0 release in the interest of more explicitly providing advance notice in light of the concerns brought forth by businesses under the condition that the mempoolfullrbf option is released with 25.0.

I will continue to discourage the practice of accepting unconfirmed transactions (“0-conf”), encourage businesses to transition to full-RBF-compatible practices, advocate for all transactions to be labeled as replaceable by default at transaction creation, will run mempoolfullrbf=1 myself, but will not lobby miners to adopt mempoolfullrbf=1 in the foreseeable future.

I would also like to point out that even when a single miner starts running full-RBF, replacement of non-replaceable transactions that pay a competitive feerate would only succeed if that pool finds the next block. Presumably, services that engage in accepting unconfirmed transactions do not treat transactions that pay uncompetitive feerates as reliable in the first place, so a minority of the hashrate adopting full-RBF would only cause a graceful degradation of the first-seen-safe behavior they rely on.

NACK. Reverting a policy configuration option that (a non-trivial number of) users want to use just because other users are relying on nobody or almost nobody running this policy (which still doesn't provide very strong security guarantees) would:

1. suggest that Core developers are responsible for what policies individual node operators use, and
2. set a precedent that a new policy option (that doesn't change the default!) can be vetoed by users who rely on this option existing behind an artificial barrier (running a custom patch or an alternative implementation), which necessarily raises the question of just who exactly gets to veto or delay the release of policy options.

@glozow re: “Peter Todd reports that he has observed replacements of non-signaling transactions using opentimestamps. He notes that they are rare, and it's possible that these are due to propagation latency.” I'm skeptical that those double spends are due to propagation latency, as there is a significant minimum time between transaction broadcasts. As an experiment I'm paying significantly higher than normal fees for replacements. So it's in my interest to avoid them happening by accident. joy More likely is that the double spends at pools previously observed to not be running full rbf are due to miners changing configurations. Eg restarting nodes, starting new nodes, etc.

There are only two txs listed on https://alice.btc.calendar.opentimestamps.org/ as not paying 182sats, namely: https://mempool.space/tx/3949f06019fd6acb95184a6667309f4c5b045f296667019e1d8b421547831418 and https://mempool.space/tx/0f1c7b57b9e95bf84b6a2f5462dfeb7284e404dcfd1b49a95967f5e794808b66 . Looking at miningpool.observer's entry for the corresponding blocks, I'm not seeing any conflicting transactions reported https://miningpool.observer/template-and-block/00000000000000000003509625dbe8c7671a7810ebe2250c8327bf5b4f845d89 https://miningpool.observer/template-and-block/0000000000000000000371542c63451b800697125fa8f7f4c65e46b1c857f7d9 (and the txs aren't listed as missing/extra at all). That suggests pretty strongly to me that any earlier versions of those txs failed to propogate to both the miner (Foundry/f2pool) and miningpool.observer (and probably the rest of the network), rather than the miner using fullrbf logic to accept the replacement, or having missed txs due to a losing their mempool/restarting their node.

EDIT: can confirm the more recent of these on my node too; it doesn't have fullrbf enabled and saw and accepted the tx that got mined prior to the block:

$  bitcoin-cli getrawtransaction 3949f06019fd6acb95184a6667309f4c5b045f296667019e1d8b421547831418 1 | jq -r .hash
69fc0a0a101ef0bd64367455c8ac6a50601eb18598418fa7e5ae2c6256d7bddc

$ grep debug.log

2022-10-20T12:53:46Z UpdateTip: new best=000000000000000000043b1c8b2b3917ea87e9abca33256eb7e11431771ebc78 height=759517 version=0x244a2004 log2_work=93.795964 tx=773653879 date='2022-10-20T12:53:43Z' progress=1.000000 cache=53.0MiB(339792txo)

2022-10-20T12:54:06Z [net] got inv: wtx 69fc0a0a101ef0bd64367455c8ac6a50601eb18598418fa7e5ae2c6256d7bddc  new peer=6
2022-10-20T12:54:06Z [net] Requesting wtx 69fc0a0a101ef0bd64367455c8ac6a50601eb18598418fa7e5ae2c6256d7bddc peer=6
2022-10-20T12:54:07Z [net] got inv: wtx 69fc0a0a101ef0bd64367455c8ac6a50601eb18598418fa7e5ae2c6256d7bddc  have peer=52
2022-10-20T12:54:07Z [net] received getdata for: wtx 69fc0a0a101ef0bd64367455c8ac6a50601eb18598418fa7e5ae2c6256d7bddc peer=5
2022-10-20T12:54:08Z [net] got inv: wtx 69fc0a0a101ef0bd64367455c8ac6a50601eb18598418fa7e5ae2c6256d7bddc  have peer=14

2022-10-20T13:07:22Z UpdateTip: new best=00000000000000000002fd05a63699e748d4cfeeec730db995e7e9086f5c2c14 height=759518 version=0x3ad82004 log2_work=93.795977 tx=773656329 date='2022-10-20T13:07:11Z' progress=1.000000 cache=54.1MiB(349179txo)
2022-10-20T13:32:10Z UpdateTip: new best=0000000000000000000239358b51e91dc53cab411516d7661652d05e38ebb2c4 height=759519 version=0x20e00000 log2_work=93.795990 tx=773659863 date='2022-10-20T13:31:53Z' progress=1.000000 cache=56.0MiB(364004txo)
2022-10-20T13:38:18Z UpdateTip: new best=00000000000000000007423035b152ec41665fb297999591fbec8b34481d8a63 height=759520 version=0x20000004 log2_work=93.796002 tx=773662439 date='2022-10-20T13:38:08Z' progress=1.000000 cache=56.7MiB(370328txo)
2022-10-20T13:40:09Z UpdateTip: new best=0000000000000000000606c73e60548cefbd4e3553e76dc13d2c55be204795d7 height=759521 version=0x25584000 log2_work=93.796015 tx=773664575 date='2022-10-20T13:39:47Z' progress=1.000000 cache=57.0MiB(372367txo)

2022-10-20T13:45:51Z UpdateTip: new best=0000000000000000000371542c63451b800697125fa8f7f4c65e46b1c857f7d9 height=759522 version=0x29f3c004 log2_work=93.796028 tx=773667438 date='2022-10-20T13:45:27Z' progress=1.000000 cache=57.7MiB(377845txo)

EDIT 2: likewise for the more recent, quite high fee transaction in block 760761:

$ bitcoin-cli getrawtransaction c1f77cbe5a003c238cdb761ca1eb4f5d012598e7e3994f7454f93881bdd29541 1 | jq .hash
"3a7c9521cea462f03c08644a7ff3dfc535b4f297aec409da102c7256904709d7"
$ grep -E 'UpdateTip|3a7c9521cea462f03c08644a7ff3dfc535b4f297aec409da102c7256904709d7' debug.log
...
2022-10-29T02:45:12Z UpdateTip: new best=00000000000000000003b0a0888ee6dbc5cdd67de6ca588fb08f4fcf4b40250d height=760760 version=0x2000e000 log2_work=93.812146 tx=775909801 date='2022-10-29T02:45:16Z' progress=1.000000 cache=111.2MiB(810837txo)
2022-10-29T02:45:29Z [net] got inv: wtx 3a7c9521cea462f03c08644a7ff3dfc535b4f297aec409da102c7256904709d7  new peer=25
2022-10-29T02:45:29Z [net] Requesting wtx 3a7c9521cea462f03c08644a7ff3dfc535b4f297aec409da102c7256904709d7 peer=25
2022-10-29T02:45:29Z [net] got inv: wtx 3a7c9521cea462f03c08644a7ff3dfc535b4f297aec409da102c7256904709d7  have peer=1488
2022-10-29T02:45:29Z [net] got inv: wtx 3a7c9521cea462f03c08644a7ff3dfc535b4f297aec409da102c7256904709d7  have peer=5
2022-10-29T02:45:30Z [net] got inv: wtx 3a7c9521cea462f03c08644a7ff3dfc535b4f297aec409da102c7256904709d7  have peer=0
2022-10-29T02:53:44Z UpdateTip: new best=00000000000000000004a054fab02d25460c0470b3ac9e4bcc4c9a98818e1746 height=760761 version=0x3fffe000 log2_work=93.812159 tx=775911422 date='2022-10-29T02:53:05Z' progress=1.000000 cache=111.7MiB(815191txo)
...

@darosior

I don't buy the part that it's substantially safer to accept unconfirmed transactions as final today, due to some magic protection mechanism in place (and apparently closed source? Has anyone seen it and could vouch for its efficiency, that non-signaling RBF would break?).

It's not a closed source mechanism, it's the one described by satoshi in July 2010: https://satoshi.nakamotoinstitute.org/posts/bitcointalk/233/ (the followup is also apropos: "imagine if instead the machine dispensed small-denomination giftcards"). It's broken by txs that can be RBFed because there's no longer a race: you can send the replacement minutes or hours later, and the original will still disappear everywhere. It doesn't matter whether it's full RBF or opt-in RBF (cite); either breaks this approach. The difference is that with full RBF there's no possibility to make transactions where that system works anymore.

If this PR (or something like it) gets merged, here's some text that could be added to the 24.0 release notes: https://gist.github.com/harding/f3cd8affb10cea96d352347fed7f4a3a Of course, I'm happy to make edits.

I'm very much in favor of @MarcoFalke's suggestion:

I think it should be fine to revert on master right after this pull request is merged to clarify that the change is only temporary for the 24.0 release.

I incorporated language in the draft release note indicating that some form of full RBF would be released in 25.0 (if not sooner), although I didn't say exactly what in order to leave the options open for the specific implementation.

@darosior

I don't know how we could communicate to end users better really. Antoine has been exemplary in trying to motivate, explain, and champion "full" RBF to affected stakeholders.

I agree @ariard has done a great job communicating! As have several other people in this discussion. But I don't think we can ignore the evidence that at least one major service that's affected didn't learn about this pending change until they were reading the draft release notes. On one hand, it's a bit discouraging that posting things to the ML and writing them up in Optech and reaching out to known zero conf users directly aren't enough, but on the other hand I think it's really great that there are service operators who care enough about Bitcoin Core to check out our RCs and go and find our draft release notes. Let's use that and be better about communicating our plans for major changes in the future via the release notes.

@instagibbs @xekyo @petertodd I don't think I disagree with anything any of you said here. I very much want us to get to full RBF being the default policy. I don't think a six-month release delay on an option that wasn't supposed to really change the network anyway slows us down much---but it does give us a chance to demonstrate that we listen to feedback and allows us to better communicate our intention for the future.

Thanks @harding to pursue the conversation,

It seems to me that the email you linked only describes countermeasures against separate propagation of conflicting transactions and that those countermeasures would be ineffectual against a replacement that was sent more than a few seconds after the original transaction.

This is an open question on what is the most effective design of such countermeasures. Let's say attacker strategy is to build a knowledge of the victim's transaction-relay topology to identify the "blind spots", where to broadcast a conflicting double-spend transaction. Efficient countermeasures would be to have well-connected nodes with outbound transaction-relay peers rotation to break free from the attacker view of your tx-relay topology. As you would like to avoid local mempool state (e.g mempool min fee) to interfer with your capture of a conflict hinting an ongoing attack, the reconciliating logic should just monitor spent utxos, where it is indifferent if a transaction is a conflict or a replacement. A replacement candidate or a conflicting transaction should be processed identically by your double-spend countermeasures infra.

If removing this option will significantly delay or freeze the progress of full-RBF deployment, that seems to me to be a statement that you believe zero-conf services are at increased risk

To be clearer, the risks I'm mentioning are the ones affecting use-cases like coinjoin/dual-funded channels/splicing/etc if full-rbf is delayed ad vitame eternam. At the same time, I never denied there are increased risks encumbered by 0confs applicati0ons and I've been clear about in my communications and strongly noting the risk arbitrage we're facing [0].

[0] https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2022-October/021111.html

"For sure, contracting protocols and multi-party applications exposed by the lack of full-rbf are still young overall, though as they attract more volume they're also likely to become honeypots for any competing services providers interested to hijack economic traffic (kinda the same concern than channel jamming...) At the same time, we still have 0confs services more exposed by full-rbf, a bit stuck between Scylla and Charybdis."

Patches for enabling full RBF have been available since at least 2015 and have even previously been deployed by miners. That's fine; zero-conf services and those worried about free options have always been exposed to that risk. But just because others can break an existing system doesn't mean that we should treat it as broken now.

To restate the whole spirit of the initial approach proposed by the optional setting and automatic preferential peering, the idea was to remove from the Bitcoin Core project to have to say which of 0confs use-cases or contracting protocols use-cases is the legitimate one in face of policy rules, and defer the choice to node operators. According to this idea, there is no more "we" (the project) qualifying a use-case of broken or not. All systems are broken beyond their security models, what is hard to qualify are those security bounds, and how Core settings and policy rules can offer interfaces to fulfill
those bounds or not. (That use-case should be incentive-compatible should be considered by the designers/applications themselves, holistically of all the layers implicated IMHO)

I don't think we need to worry about people misinterpreting our actions when we can explain what we're doing, and why we're doing it, in the 24.0 release notes.

In the light of contracting protocols being exposed to the lack of full-rbf transaction-relay paths and mining hashrate, what I'm wondering about is the lack of contributors opinions convergence on a "common" action, even before to explain this action towards Core users, in release notes or elsewhere. Personally, I still think we should collect more feedback from coinjoins and collaborative transactions users/operators, to have a more complete view of the risks and stakes across the Bitcoin ecosystem, before to finalize a decision.

I read the above as you agreeing that reviewers were unaware of all the risks when they gave their ACKs. If that's the case, I don't see why it's a matter of significant debate to partially revert this unreleased PR so that it can be reconsidered with all of its consequences made explicit.

Reverting this PR doesn't answer the question I'm raising about the risk scope of the design considerations of Bitcoin Core development. W.r.t today of applications diversity and complexity, you cannot expect all the Bitcoin Core contributors to have a full-understanding of the implications on Lightning (when most of the LN devs themselves have a short understanding of the interactions of the protocol with mempool rules), a comprehensive view of the payment 0confs ecosystem and a practical experience with the inner workings of wallets. Rather, I think it would be more valuable for the project how our design process should actively collect feedback from the known class of use-cases, and how we should evaluate this feedback.

This patch makes it possible for people running unmodified Bitcoin Core to change a longstanding network policy that is used by multiple businesses serving thousands of users. Whether that's technically deprecation or not, I think it warrants giving businesses a warning about the impending change.

From my understanding, the crux of the issue is the risk arbitrage situation. Giving a longer warrant to the business is gained at the price of extending the risk DoS exposures of multi-party applications and contracting protocols users and services. I do believe the accumulated user base of all the coinjoin services ranks also in the order of thousands of people, with notable flows at stake. Moving forward, we would be better off laying out procedural principles to solve risk arbitrage situation.

It's an option, it's default false. If people don't like it, they don't have to use it. Bitcoin cannot rely on synchronized mempools or a prediction of miner preferences, we all know that, and this is not news - it's been discussed for years. There are lots of cool "instant transaction protocols" out there. Maybe as miners adopt full rbf, we'll see some incentive to adopt them?

@ariard

To restate the whole spirit of the initial approach proposed by the optional setting and automatic preferential peering, the idea was to remove from the Bitcoin Core project to have to say which of 0confs use-cases or contracting protocols use-cases is the legitimate one in face of policy rules, and defer the choice to node operators.

First, I think we should be clear that the choice of policy rules (and consensus rules) has always belonged to node operators. #25353 makes a particular policy choice more accessible to operators; that is all.

Second, what you wrote in the quoted text above is not remotely like what's in the help text or the release notes. If the goal is to get node operators to decide which use case is legitimate, why are the help text[1] and release notes[1] only understandable to experts in relay policy? Moreover, why do operator preferences for network-wide behavior need to be coupled to local mempool/relay policy? I can say "I want to be able to do full RBF in the future" without running a full RBF node today; I may even have a strong incentive to run a certain policy today even though I want a different policy in the future (e.g. a miner may want to run the dominant policy to maximize their BIP152 compact block effectiveness but prefer a non-dominant policy that would increase fee revenue long-term).

If we want to know what users want, let's maybe ask them instead of providing new options with abstruse descriptions and potentially significant network effects.

[1] "Accept transaction replace-by-fee without requiring replaceability signaling"
[2] "A new mempoolfullrbf option has been added, which enables the mempool to accept transaction replacement without enforcing BIP125 replaceability signaling. (#25353)"

Reverting this PR doesn't answer the question I'm raising about the risk scope of the design considerations of Bitcoin Core development. W.r.t today of applications diversity and complexity, you cannot expect all the Bitcoin Core contributors to have a full-understanding of the implications on Lightning (when most of the LN devs themselves have a short understanding of the interactions of the protocol with mempool rules), a comprehensive view of the payment 0confs ecosystem and a practical experience with the inner workings of wallets.

Sure, it will often be the case that people outside of Bitcoin Core have a better understanding about certain issues than this project's primary contributors. But that's part of why we have an open development process: so proposed changes can be reviewed and (if warranted) rejected or changed based on that feedback.

That process only works if we communicate about changes to the people who have that better understanding, and if we take the time to listen to their feedback. In the case of #25353, we didn't communicate the change to at least one outside expert who had useful feedback. We need to decide now whether that feedback warrants action. I think it does: I don't think we should provide an option that changes behavior in a way that might break user processes without sufficient warning (except in an emergency, which no one has claimed this is).

Giving a longer warrant to the business is gained at the price of extending the risk DoS exposures of multi-party applications and contracting protocols users and services.

In my opinion, the flowchart should look like:

• Is this an emergency?
  • Yes, it's an emergency: it's ok to make a change without prior notice even though many people made significant decisions under the expectation that the policy would continue
  • No, it's not an emergency: any change we want to make that may adversely affect many users should be communicated at least six months in advance so that people can make adjustments to their processes or try to convince contributors (or other users) to take a different approach

Do you disagree with the above flowchart? Do you think DoS exposure against multi-party applications and contracting protocols under the 23.x relay policy constitutes an emergency, justifying this change? Do you think no significant miners/pools will enable -mempoolfullrbf, so it doesn't matter (if so, can we remove it from mainnet as unnecessary)? Or am I missing something?

As a reminder, if this PR gets merged, there's a draft release note ready that says, "the Bitcoin Core project plans to release [full RBF] code by version 25.0 in mid-2023, or earlier if warranted by events." By merging this PR, we can demonstrate our willingness to listen to community feedback, level up our communication about this topic from good to great, gain additional time to refine our approach, but still potentially enable full RBF within six months.

In my opinion, the flowchart should look like:

• Is this an emergency?
  • Yes, it's an emergency: it's ok to make a change without prior notice
    even though many people made significant decisions under the expectation
    that the policy would continue

1. No actual change is being made by Core (as-is, before this PR).

2. It is a false premise that it is "without prior notice". People have been advised for YEARS that the "first seen" policy was unsafe, and could not be relied on.

3. It is okay to change mere policy without prior notice. Nobody should be assuming any given node has a well-known policy to begin with.

Quoting from @luke-jr :

No actual change is being made by Core (as-is, before this PR).

If no change was made in #25353, then reverting it is also not a change. So, since it's holding up release, it should just be reverted. [Note: this is not my main argument; it's just a reductio ad absurdum of Dashjr's reply.]

It is a false premise that it is "without prior notice". People have been advised for YEARS that the "first seen" policy was unsafe, and could not be relied on.

If it's equally unsafe under the 23.x policy rules as it is under the #25353 policy rules, doesn't that also imply that #25353 doesn't provide any benefit and so should be an easy revert? [Again, a reductio ad absurdum.]

It is okay to change mere policy without prior notice. Nobody should be assuming any given node has a well-known policy to begin with.

I don't understand what you're saying here or how it relates to the conversation.

Quoting from @petertodd:

Every change Bitcoin Core makes to relay policy breaks usages of zeroconf
because you can always exploit differences in relay policy to get double spends
mined.

See @ajtowns's comment above, "[the double spend monitoring system is] broken by txs that can be RBFed because there's no longer a race: you can send the replacement minutes or hours later, and the original will still disappear everywhere. It doesn't matter whether it's full RBF or opt-in RBF (cite); either breaks this approach. The difference is that with full RBF there's no possibility to make transactions where that system works anymore."

Concept ACK, though I would like us to go further and revert #25353. I wrote a long post to the mailing list arguing that we should not be trying to deploy fullrbf at this time, because:

1. Fullrbf doesn't appear to solve the multiparty (coinjoin) protocol problems first brought up as a justification;
2. Non-replacement of transactions seems like a theoretically useful feature (whether or not you think zeroconf services are good or bad for bitcoin, it's not the only use a non-replacement policy, which is helpful in CPFP applications);
3. Our current RBF policy can easily result in incentive incompatible replacements occurring (though its unclear to me how common this might be, it's an easily identifiable problem), so pushing our rbf policy on all transactions seems presumptuous; and
4. Breaking policies that are in use on the network for no clear win is counter to the philosophy I think this project should have, particularly because I worry that this would set a precedent for breaking future policies as well, such as the current v3 transaction policy proposal that is being worked on, and I think that would be a mistake.

Regarding the argument that miners and node operators should have choices: if we think that the number of double-spends of non-rbf-signaling transactions is not material -- which you would expect, given that RBF users can just signal using BIP 125 to issue their double-spends -- then the choice issue seems contrived. It seems instead that the real reason to deploy this is to try to break a policy that is otherwise working, in the hopes of that creating a big enough change that business will stop zeroconf services. However it's not clear to me that fullrbf will stop zeroconf as a practice, while it is clear that it could break legitimate use cases for non-signaling transactions, without any benefits to users who already are able to opt-in to RBF under BIP 125 today. So on the whole, offering this choice seems like it makes the network worse, not better.

NACK

Bitcoin Core developers should NOT be responsible for:

• policies individual node operators are using
• safety of said individual node operators

First, I think we should be clear that the choice of policy rules (and consensus rules) has always belonged to node operators. #25353 makes a particular policy choice more accessible to operators; that is all.

Second, what you wrote in the quoted text above is not remotely like what's in the help text or the release notes. If the goal is to get node operators to decide which use case is legitimate, why are the help text[1] and release notes[1] only understandable to experts in relay policy? Moreover, why do operator preferences for network-wide behavior need to be coupled to local mempool/relay policy? I can say "I want to be able to do full RBF in the future" without running a full RBF node today; I may even have a strong incentive to run a certain policy today even though I want a different policy in the future (e.g. a miner may want to run the dominant policy to maximize their BIP152 compact block effectiveness but prefer a non-dominant policy that would increase fee revenue long-term).

While I think we're in agreement that the choice of policy rules always belonged to node operators, at the same time we can't ignore that the choice of policy rules default, or even the existence of new optional policy mechanism present potentially significant network effects, as you're underscoring. In that light, when we're facing a situation where the most widely policy rules determine which use-case is encumbering security risks, as the Bitcoin Core project we should evaluate which use-case to support (and we don't have such formal process today, and I'm not even sure we should, unless in case of emergency), or defer the decisions to the wider crowd of node operators.

Of course, deferring the decision to the wider crowd of node operators should be done in a fashion explaining well the trade-offs, in terms that a non-expert in relay policy can understand, and within a sufficient timeframe for the average Bitcoin user or service operator to build a grounded position. About the process I've carried on with mempoolfullrlbf, while I agree this quoted reasoning wasn't exposed in the help text, neither in the releases notes, I think it's reasonable to say that explaining the full-context of a sophisticated subject like the implications of full-rbf, won't hold in a command-line help text, neither we're sure the release notes are the appropriate place. If there is one loose standard communication in term of policy changes established by years of Bitcoin Core development practices, I think this is to communicate the rational on the mailiing list, which in the present case I think I've done. That the communication and affected users collection process could have been better, this is something I'm fully conceding, though note we're moving forward in an area without clear "tickbox".

Sure, it will often be the case that people outside of Bitcoin Core have a better understanding about certain issues than this project's primary contributors. But that's part of why we have an open development process: so proposed changes can be reviewed and (if warranted) rejected or changed based on that feedback.

That process only works if we communicate about changes to the people who have that better understanding, and if we take the time to listen to their feedback. In the case of #25353, we didn't communicate the change to at least one outside expert who had useful feedback. We need to decide now whether that feedback warrants action. I think it does: I don't think we should provide an option that changes behavior in a way that might break user processes without sufficient warning (except in an emergency, which no one has claimed this is).

I think we're in agreement that an ideal policy rules design process should communicate changes to a set of qualified people, and that we should offer clear timeframe to collect and listen to their feedback. Beyond, there are few points where I might diverge, firstly I believe we should keep the Bitcoin development process permissionless and decentralized, in the sense not formalizing process where some "authority experts" must be listen to before to proceed with any change. Rather I think we should abstract out such feedback collection process and designate communication endpoints like the mailiing list or the BOLT repository or whatever. Secondly, in term of feedback warranting action, I think we would be better off to have some sense of proportionality in the risk exposure severity of any change, but also in the economic weight of the use-cases. What we could be concern of is the alleged existence of a use-case, claiming some marginal use of a subset of policy rules, and building a justification that could not be argued against or proven against, completely stalling the process. I know coming up with a development process which offer both guarantees of efficiency, decentralization and conservatism is a high bar, though I believe something worthy of for a project unique like Bitcoin Core.

In my opinion, the flowchart should look like:

Is this an emergency?
    Yes, it's an emergency: it's ok to make a change without prior notice even though many people made significant decisions under the expectation that the policy would continue
    No, it's not an emergency: any change we want to make that may adversely affect many users should be communicated at least six months in advance so that people can make adjustments to their processes or try to convince contributors (or other users) to take a different approach

Do you disagree with the above flowchart? Do you think DoS exposure against multi-party applications and contracting protocols under the 23.x relay policy constitutes an emergency, justifying this change? Do you think no significant miners/pools will enable -mempoolfullrbf, so it doesn't matter (if so, can we remove it from mainnet as unnecessary)? Or am I missing something?

I don't think I disagree in the lineaments of this flowchart, what is unclear to me is when a situation of emergency is qualified, and how we should proceed on when a known security risk exists, presents a low-technical bar, and could provoke havoc on a subset of the ecosystem as soon as some script kiddie starts to play with it (note, sadly I think the present statement works for both 0confs and contracting protocols use-cases). Personally, I won't communicate towards the miners to enable -mempoolfullrbf until we have more consensus within the project on how we move forward, however I could understand a coinjoin/dual-funded operators concerned with the risk. Like I said, somehow we're offering a safety options to 0confs service at the price of downgrading the security balance sheet of contracting protocols.

All that said, while I still hold the belief that multi-party applications and contracting protocols are exposed, I think the best way forward is actually address all the feedback we've collected during the last week on mempool policy rules design philosophy, affected use-cases, risks scope of the Bitcoin Core project and respect of a policy rules change communication process. Personally, I'll allocate my time and energy on working on those feedback, however if those contracting protocols users have a higher sensibility to the DoS concern, we have a well-reviewed patchset for them to protect better their services (#25353 + #25600). They're free to reach out to miners, and ask fro mempoolfullrbf=1 adoption.

Among all the feedbacks collected, I think they're few ones really relevant to address.

One is the formalization of the policy rules design process, which communication endpoints we should reach out for (e.g the BOLT community, the LSP process, business/merchants, etc), what types of design feedback we should ask for (economic soundness, security risk, operational risk), what should be left beyond the scope of the Bitcoin Core process (is FX risk really something we should solve at that layer ?). what are reasonable timeline, allowing multiple rounds of back-and-forth, what should be the communication venues notified of significant changes.

Beyond, I think Suhas's post raised an interesting position in terms of philosophy design of transaction-relay rules. From my understanding, the position asserted is the following "to each use-case belongs a set of policy rules", and that as long there is no harm towards the networks, the other protocols or node operators/miners we should aim to conciliate everything. While, I don't this position is sustainable in face of the mining dynamics and the long-term incentives, I had to confesss I don't have a strong model of the miners ecosystem, and how they interfere with use-cases/transaction-relay to demonstrate well my point. I think it would be very valuable to work on such model, be able to predict few income scenario in function of the strategy, and encompass MEV-style security risks towards second-layers. I would be really interested to evaluate against some long-term theoretical attacks against LN like miner harvesting or tx-withold things. Developing some sound model would take time, better to have something maintainable we can re-use to evaluate future policy changes, and other Bitcoin protocol features.

Moreover, if we can build consensus on fullrbf itself, assuming we can address the concern above, I think it's highly valuable and in compliance with the conservative stand of the project to coordinate a timeline with 0confs services, working with them on deploying adequate counter-measures, if we can come up with robust schemes. Offering a predictable and visible timeline solidifies the development practice of Bitcoin Core in term of policy changes. There is not only the question of fullrbf, but also improving our process for the future. During the coming years, I still think we'll have more and more second-layers and advanced use-cases showing up, complexifying how we have to think and develop transation-relay policy changes.

Concept ACK on #26287.

@harding

Quoting from @petertodd:

Every change Bitcoin Core makes to relay policy breaks usages of zeroconf
because you can always exploit differences in relay policy to get double spends
mined.

See @ajtowns's comment above, "[the double spend monitoring system is] broken by txs that can be RBFed because there's no longer a race: you can send the replacement minutes or hours later, and the original will still disappear everywhere. It doesn't matter whether it's full RBF or opt-in RBF (cite); either breaks this approach. The difference is that with full RBF there's no possibility to make transactions where that system works anymore."

This viewpoint of how zeroconf double-spends work is naive.

The reason why any difference in relay policy can be exploited to double-spend comes from the fact that we don't - and can't - relay double spends. This gives us two scenarios:

1. Broadcast a transaction to the merchant that some miners do not accept, followed by a double spend that some miners do accept.

2. Target a transaction to some miners that nodes broadly do not accept, followed by a transaction to the merchant

In both cases there can be a significant delay between the two transactions.

The latter case is the main reason why zeroconf "risk management" services inevitably end up sybil attacking the network: they need to connect to as many nodes as possible to try to pick up transactions that miners accepted that most nodes did not. It's also one reason why having merchants be vulnerable to zeroconf creates incentives for zeroconf double-spenders to attack the P2P network, as well as miners.

The former case is exactly what I successfully exploited over and over again during mempool congestion. Miners often had much higher minimum fee policies than merchants expected, so double-spending by simply sending a lowish fee transaction was successful as the miners simply didn't have the tx in their mempool. Aided of course, by the fact that there are a fair number of full-rbf nodes out there.

The fact is minrelaytxfee and maxmempool options are just as dangerous to zeroconf as mempoolfullrbf.

Note that you can repeat 1 or 2 over and over until you succeed, and then only use the output from that success case for the actual merchant transaction. This ensures you succeed every time.

Closing for now. Let me know if this should be reopened.