-
Notifications
You must be signed in to change notification settings - Fork 38.7k
refactor: Replace chain relayTransactions/submitMemoryPool by higher method #15713
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
refactor: Replace chain relayTransactions/submitMemoryPool by higher method #15713
Conversation
|
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers. ConflictsReviewers, this pull request conflicts with the following ones:
If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first. |
src/interfaces/chain.cpp
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for consolidating the code from relayTransaction and submitToMemoryPool in interfaces/chain.cpp and from BroadcastTransaction in node/transaction.cpp.
I do think the right place for all this code to live is in node/transaction.cpp, though. interfaces/chain.cpp is really just meant to be a simple glue file with lines like:
bool broadcastTransaction(...) override { return BroadcastTransaction(...); }Also, I didn't look closely yet, but it seems like this refactoring might be changing wallet behavior (this could explain travis test failures). It would probably be better if this PR avoided changing behavior to the extent possible. This might require adding some extra options to the existing BroadcastTransaction function in node/transaction.cpp, or adding a second broadcast function there that can hopefully share some code with the current one.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Okay was thinking than node/transaction.cpp was temporary and long term goal was to aggregate it to interfaces/node.cpp.
Didn't want to change wallet behavior, fixed it by adding options in the existing BroadcastTransaction (but still not sure of ReacceptWalletTransactions, should we broadcast txn there ?), functional tests should be good now. Also rebased
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
re: #15713 (comment)
Thanks! This looks good and now net removes code (+63/-83).
Okay was thinking than node/transaction.cpp was temporary
Yeah, that may have been the original thought. But I think it could be a reasonable long term location if more things are moved around it. In any case it's where the BroadcastTransaction function currently lives, so it's easier not to move in this PR, which is more about simplifying wallet code.
and long term goal was to aggregate it to interfaces/node.cpp
Everything in interfaces/, including interfaces/node.cpp is just supposed to be very short glue code (the files would be huge otherwise). The interfaces are a way of letting wallet, node, and gui code run in different processes, and a way of preventing them from accessing each others internal state.
abae703 to
322638c
Compare
|
Thanks for looking at this @ariard. My view is different from @ryanofsky's. I think rather than trying to maintain existing behaviour, we should use this interface tidyup to better define the relationship between the wallet and the node, ie to answer the question "What services should the node offer to the wallet?" Currently, the node exposes two methods to the wallet for sending a transaction:
Note that to relay an INV for the transaction to peers (2), the transaction must already be in the mempool. See: bitcoin/src/net_processing.cpp Line 3804 in ba54342
The wallet currently calls 1a. In The wallet currently calls 2a. In I've been thinking about this in the context of the wallet rebroadcasting transactions (see discussion here: http://www.erisian.com.au/meetbot/bitcoin-core-dev/2019/bitcoin-core-dev.2019-03-29-19.10.log.html#l-9). My view now is that the wallet should not have any access to relay functionality. It should be able to submit transactions to the node's mempool, and it should then be the mempool's responsibility to broadcast/rebroadcast the transaction (see comment here: http://www.erisian.com.au/meetbot/bitcoin-core-dev/2019/bitcoin-core-dev.2019-03-29-19.10.log.html#l-92). That seems like a much cleaner divide of responsibility. This PR seems like a simplification in terms of combining |
|
John are you just saying that this PR does not go far enough, or that it does something bad or makes things more complicated? I haven't looked closely enough at this yet but if it exposes one node method node instead of two, and reduces the amount of code without changing behavior, it would seem like a strict improvement. Is your rebroadcast proposal harder to implement after this PR? |
It does reduce the number of node methods exposed, but does so by adding a flag to one of them which toggles the behaviour. I'm not sure if that counts as a simplification, or it's just moving the complexity around. I'm also not sure if this maintains existing behaviour. For example, the |
322638c to
56e3997
Compare
|
I kinda agree with @jnewbery here, wallet shouldn't have access to relay, because if so it may add complexity of proper tx rebroadcasting tracking, notably in privacy-enhancing context in referenced discussions ? |
56e3997 to
468116b
Compare
|
Ariard or John, what are next steps? Should this PR be closed and revisited in the future if rebroadcasting is improved? Should the current code comment about this be removed or clarified? bitcoin/src/interfaces/chain.h Lines 52 to 54 in f3ecf30
|
|
I don't want to discourage @ariard from making this change if you both think it's an improvement over the existing code. It certainly doesn't preclude us from makeing the changes that I proposed in #15713 (comment) at a later time. @ariard - if you think this should still go in, can you perhaps break up the PR into commits and justify any changes (eg why remove Note that this conflicts with my #15728 which I think is a useful tidy-up in itself. |
|
Okay will keep working on it by breaking up the PR and justify changes. Had a look on #15728, IMO conflicts should be solved smoothly once yours get into it. |
468116b to
5628623
Compare
|
Rebased, splitted the commits, added justifications. Note that I removed relay flag in BroadcastTransaction, if I get John opinion submitting tx to memory pool should imply relay if it's a valid one. |
src/wallet/wallet.cpp
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I still have a doubt on BroadcastTransaction HaveChain check behavior. HaveChain = !existingCoin.IsSpent() doesn't mean if tx is in chain, but all of its outputs has been spent, we accept it again to mempool/relay ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@ariard wrote:
I still have a doubt on BroadcastTransaction HaveChain check behavior.
HaveChain = !existingCoin.IsSpent()doesn't mean if tx is in chain, but all of its outputs has been spent, we accept it again to mempool/relay ?
IsSpent() is "misleadingly" named. Because we use AccessCoin: Return a reference to Coin in the cache, or a pruned one if not found.. Pruned in this case means all its CTxOut are IsNull(), and IsSpent() just calls IsNull().
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @Sjors . I agree with you that IsSpent() is a misleading name. A more accurate name would be DoesNotExist(). Negating the function here determines whether the coin exists in the current UTXO set.
The check in BroadcastTransaction() is an optimization: if any of the outputs from the tx still exist in the UTXO set then the tx must be confirmed. If the coin has been confirmed but it's outputs don't exist, then it'll fail later (because its inputs can't exist if it has already been spent).
This check in RelayWalletTransaction() is also an optimization. If the transaction has been confirmed (or a conflicting transaction has been confirmed), then the block hash will be recorded in the CMerkleTx, and we can check in the chain whether that block exists.
An alternative to removing this call to GetDepthInMainChain() would be to merge #15931 first, which removes GetDepthInMainChain's requirement on locking the chain.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @Sjors . I agree with you that
IsSpent()is a misleading name. A more accurate name would beDoesNotExist(). Negating the function here determines whether the coin exists in the current UTXO set.The check in
BroadcastTransaction()is an optimization: if any of the outputs from the tx still exist in the UTXO set then the tx must be confirmed. If the coin has been confirmed but it's outputs don't exist, then it'll fail later (because its inputs can't exist if it has already been spent).
It might be useful to enhance the code documentation added in 976d1e485 with this information (if it isn't expected to change soon). I found it v helpful to understanding the !existingCoin.IsSpent()) optimisation.
jnewbery
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
opinion submitting tx to memory pool should imply relay if it's a valid one
No, this would cause the wallet to rebroadcast all of its unconfirmed transactions at startup. That's a change in behaviour and a potentially bad privacy leak.
bcbc316 to
94623bd
Compare
jnewbery
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A few comments inline. The intermediate commit 07d5787041461bcfb8b5777809d874869710aa1e does not build because locked_chain is not defined in SubmitMemoryPoolAndRelay().
src/wallet/wallet.cpp
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Review note: This is a minor behaviour change. chain().broadcastTransaction() could return false if BroadcastTransaction() returned TransactionError::P2P_DISABLED (ie the transaction is added to the mempool but not relayed).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What's your take on returning a pair of boolean from chain().broadcastTransaction() to encode both mempool/relay results ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In the existing code there's not a return code for chain::relayTransaction(). I think it's fine for the new broadcastTransaction interface method to only return false if the transaction failed to get into the mempool, and treat TransactionError::P2P_DISABLED as success. That way, fInMempool will get updated in exactly the same way as the existing code.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good, added TransactionError::P2P_DISABLED == err check in broadcastTransaction interface
94623bd to
602e38d
Compare
|
Thanks @jnewbery for the review and sorry for the ugly commit split. I think I've addressed all your points except this one : #15713 (comment) ? |
…MemoryPool by higher method fb62f12 Tidy up BroadcastTransaction() (John Newbery) b8eecf8 Remove unused submitToMemoryPool and relayTransactions Chain interfaces (Antoine Riard) 8753f56 Remove duplicate checks in SubmitMemoryPoolAndRelay (Antoine Riard) 611291c Introduce CWalletTx::SubmitMemoryPoolAndRelay (Antoine Riard) 8c8aa19 Add BroadcastTransaction utility usage in Chain interface (Antoine Riard) Pull request description: Remove CWalletTx::AcceptToMemoryPool Replace CWalletTx::RelayWalletTransaction by SubmitMemoryPoolAndRelay Add a relay flag to broadcastTransaction because wasn't sure of ReacceptWalletTransactions semantic. Obviously, working on implementing bitcoin#14978 (comment) to add the new higher-method in Node interface, will add a commit, just need more thought to do it cleanly ACKs for top commit: MarcoFalke: re-ACK fb62f12 Sjors: re-ACK fb62f12 Tree-SHA512: a7ee48b0545f537fa65cac8ed4cb24e777ab90b877d4eefb87971fa93c6a59bd555b62ad8940c6ffb40592a0bd50787d27587af99f20b56af72b415b6394251f
jnewbery
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
post-merge ACK fb62f12.
A couple of nits inline that could be cleaned up in a subsequent PR.
| { | ||
| RelayTransaction(txid, *g_connman); | ||
| const TransactionError err = BroadcastTransaction(tx, err_string, max_tx_fee, relay, /*wait_callback*/ false); | ||
| // Chain clients only care about failures to accept the tx to the mempool. Disregard non-mempool related failures. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this line can be removed now. The only failures that BroadcastTransaction() can return are mempool-related failures.
| TransactionError BroadcastTransaction(const CTransactionRef tx, uint256& hashTx, std::string& err_string, const CAmount& highfee) | ||
| TransactionError BroadcastTransaction(const CTransactionRef tx, std::string& err_string, const CAmount& max_tx_fee, bool relay, bool wait_callback) | ||
| { | ||
| assert(g_connman); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: I'd prefer to see a comment next to this assert to explain why g_connman must always be assigned here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should be part of #16503, maybe
|
I was wrong about this: #15713 (comment). Apparently removing that check causes log spam, particularly for test nodes. |
b7b9f6e Remove p2pEnabled from Chain interface (Antoine Riard) Pull request description: RPC server starts in warmup mode, it can't process yet calls, then follows connection manager initialization and finally RPC server get out of warmup mode. RPC calls shouldn't be able to get P2P disabled errors because once we initialize g_connman it's not unset until shutdown, after RPC server has been stopped. @mzumsande comment in bitcoin#15713 let me thought that `p2pEnabled` was maybe useless, `g_connman` is always initialized before RPC server is getting out of warmup. These checks against P2P state were introduced in bitcoin@5b446dd. ACKs for top commit: promag: ACK b7b9f6e jnewbery: ACK b7b9f6e Tree-SHA512: 4de2b9fc496bf8347ff5cc645848a5a44c8ca7596cd134f17f3088f5f8262d1d88b8e2a052df93e309ec9a81956a808df17a9eb9f10d4f4d693c95d607fe3561
…rface Summary: Access through a broadcastTransaction method. Add a wait_callback flag to turn off race protection when wallet already track its tx being in mempool Standardise highfee, absurdfee variable name to max_tx_fee We drop the P2P check in BroadcastTransaction as g_connman is only called by RPCs and the wallet scheduler, both of which are initialized after g_connman is assigned and stopped before g_connman is reset. bitcoin/bitcoin@8c8aa19 --- Depends on D6272 Partial backport of Core [[bitcoin/bitcoin#15713 | PR15713]] Test Plan: ninja check-all Reviewers: #bitcoin_abc, Fabien Reviewed By: #bitcoin_abc, Fabien Differential Revision: https://reviews.bitcoinabc.org/D6273
Summary: Higher wallet-tx method combining RelayWalletTransactions and AcceptToMemoryPool, using new Chain::broadcastTransaction bitcoin/bitcoin@611291c --- Partial backport of Core [[bitcoin/bitcoin#15713 | PR15713]] Test Plan: ninja check-all Reviewers: #bitcoin_abc, deadalnix Reviewed By: #bitcoin_abc, deadalnix Differential Revision: https://reviews.bitcoinabc.org/D6420
Summary: IsCoinBase check is already performed early by AcceptToMemoryPoolWorker GetDepthInMainChain check is already perfomed by BroadcastTransaction To avoid deadlock we MUST keep lock order in ResendWalletTransactions and CommitTransaction, even if we lock cs_main again further. in BroadcastTransaction. Lock order will need to be clean at once in a future refactoring bitcoin/bitcoin@8753f56 --- Depends on D6420 Partial backport of Core [[bitcoin/bitcoin#15713 | PR15713]] Test Plan: ninja check-all Reviewers: #bitcoin_abc, deadalnix Reviewed By: #bitcoin_abc, deadalnix Subscribers: deadalnix Differential Revision: https://reviews.bitcoinabc.org/D6421
…ctions Chain interfaces Summary: bitcoin/bitcoin@b8eecf8 --- Depends on D6421 Partial backport of Core [[bitcoin/bitcoin#15713 | PR15713]] Test Plan: ninja check-all Reviewers: #bitcoin_abc, deadalnix Reviewed By: #bitcoin_abc, deadalnix Differential Revision: https://reviews.bitcoinabc.org/D6422
Summary: bitcoin/bitcoin@fb62f12 --- Concludes backport of Core [[bitcoin/bitcoin#15713 | PR15713]] Test Plan: ninja check-all Reviewers: #bitcoin_abc, deadalnix Reviewed By: #bitcoin_abc, deadalnix Differential Revision: https://reviews.bitcoinabc.org/D6427
…utility 3fef7d3 refactor : use RelayTransaction in BroadcastTransaction utility (Antoine Riard) Pull request description: Implementing suggestion in bitcoin/bitcoin#15713 (comment). Seems a reason of these node utilities is to glue with already there functions, so we should reuse them. ACKs for top commit: MarcoFalke: ACK 3fef7d3 promag: ACK 3fef7d3, verified there are no more `PushInventory(CInv(MSG_TX, ...`, nice refactor, 👍 @amitiuttarwar. jnewbery: ACK 3fef7d3 jonatack: ACK 3fef7d3, second @jnewbery's suggestions, my guess is they could be added without risking delaying this PR. Tree-SHA512: 841c65d5f0d9ead5380814bb2260d7ebf03f2a9bfa58a1025785d353bdb42f9122cc257993e6a7bd2bd3f2c74db19c5978cc14be0d83258124ca22e33d6d0164
…utility 9bc8b28 refactor : use RelayTransaction in BroadcastTransaction utility (Antoine Riard) Pull request description: Implementing suggestion in bitcoin/bitcoin#15713 (comment). Seems a reason of these node utilities is to glue with already there functions, so we should reuse them. ACKs for top commit: MarcoFalke: ACK 9bc8b28 promag: ACK 9bc8b28, verified there are no more `PushInventory(CInv(MSG_TX, ...`, nice refactor, 👍 @amitiuttarwar. jnewbery: ACK 9bc8b28 jonatack: ACK 9bc8b28, second @jnewbery's suggestions, my guess is they could be added without risking delaying this PR. Tree-SHA512: 841c65d5f0d9ead5380814bb2260d7ebf03f2a9bfa58a1025785d353bdb42f9122cc257993e6a7bd2bd3f2c74db19c5978cc14be0d83258124ca22e33d6d0164
Remove CWalletTx::AcceptToMemoryPool
Replace CWalletTx::RelayWalletTransaction by SubmitMemoryPoolAndRelay
Add a relay flag to broadcastTransaction because wasn't sure of ReacceptWalletTransactions semantic.
Obviously, working on implementing #14978 (comment) to add the new higher-method in Node interface, will add a commit, just need more thought to do it cleanly