-
Notifications
You must be signed in to change notification settings - Fork 38.7k
wallet: add cached m_is_ibd to remove Chain::isInitialBlockDownload #17484
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
The reason that you had to be out of IBD in order to set an HD seed is because by setting a new HD seed, no new keys would ever be generated from the old seed. So if there were a transaction that involved a key that hadn't been generated yet and was in a block after where you were currently synced, setting a new HD seed would mean that you don't detect that transaction. That would be bad. With |
|
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers. ConflictsReviewers, this pull request conflicts with the following ones:
If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first. |
You're talking about a key derived from the old seed here ? I didn't dig in code but if I understand when we set a new seed we discard checking keys for old one right ? Node is getting out of IBD at tip N - DEFAULT_TIP_AGE(~144), which means you can set a new seed at N - 144, but you're not going to see a tx with an old key in block N - 100 for example, so is this mechanism reliable ? |
Yes, I'm talking about keys derived from the old seed. Any key that has already been derived and added to the wallet (i.e. it was in the keypool or already marked as used) will still be checked for. Setting a new seed doesn't change that. Setting a new seed only makes it so that no new keys will be derived from the old seed.
I think it's reasonably reliable as I think it is unlikely that someone used the entire keypool in 144 blocks. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Concept ACK.
First review pass to get started before digging deeper into the approach. Built, ran tests and bitcoind.
Some implementation thoughts below; please ignore them until the PR has more Concept/Approach ACKs. Additional thoughts:
- It might be clearer to merge the last two commits which appear to make overlapping changes
- Consider if you can use
const - I'd like to test this further with some embedded printing or logging
- Does the current test coverage suffice for these changes?
src/wallet/wallet.h
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
perhaps: "Whether or not the chain is in initial block download."
src/wallet/wallet.h
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for adding this documentation. Suggestion (if I understand correctly that othersaid means autrement dit, which translates to in other words):
- /* Flag to indicate chain is still in initial block download. Set it to
- * false to let wallet generating new seed without being connected to a node.
- * Othersaid, a wallet evaluates chain being out of initial block download
- * until it's proven false.
+ /* Flag to indicate whether or not the chain is still in initial block
+ * download. Defaults to false to allow the wallet to generate new seeds
+ * without being connected to a node. In other words, the wallet assumes the
+ * chain is out of initial block download unless shown otherwise.
src/wallet/rpcwallet.cpp
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Perhaps make this change in the last commit where the other removals of isInitialBlockDownload take place?
IIUC, the commit is misnamed and could better be called something like "Remove Chain::isInitialBlockDownload`
src/wallet/wallet.cpp
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Perhaps make this change in the commit "Add m_is_ibd in CWallet` (or merge the two commits, IIUC)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
re: #17484 (comment)
In commit "Remove Chain::isReadyToBroadcast method" (a5a674857d0ae6d535e4725337968492c821c635)
Perhaps make this change in the commit "Add m_is_ibd in CWallet` (or merge the two commits, IIUC)
Change does make sense (and is necessary) in this commit due to isReadyToBroadcast call a few lines up. But it would be good to add a comment here mentioning this is also needed to avoid spamming nodes with old transactions.
|
Thanks @achow101 for your insights, I'm going to hold-on this PR once the wallet init its state asynchronously like thanks to change here https://github.com/ariard/bitcoin/commits/2019-08-rescan-index-refactor so it would avoid behavior change on the point you raised. Thanks for your review @jonatack, will fix comments once PR gets Concept ACK. |
|
Concept ACK. @achow101 to be clear, is the failure scenario as follows:
(1) must be an HD wallet because restoring a non-HD wallet backup that has drained its initial keypool keys always results in funds loss Not allowing @ariard - is it possible to change this so |
|
For some context about the IBD and
Yes. |
|
@achow101 Thanks for the cross-reference 🙏
Keeping the old key for key derivation definitely seems like the correct fix. How complex is that (and does descriptor wallets do that automatically?) |
I don't think it would be too complex. Although I think it would be a fairly large change. We would need to keep track of the old hd seeds and keep track of where we've derived up to for a particular seed. And a bunch of things that do things on the seed would need to be changed to do things on any seed. We would have to add several more records to the wallet. Alternatively we could have
Not the current implementation. But it also would be pretty trivial to do it. |
Yes but that removes PR interest of drying up the Chain interface. I think best it to wait wallet registering its sync state with the Chain::interface (https://github.com/ariard/bitcoin/commits/2019-08-rescan-index-refactor) at start-up so after rescan done wallet knows it's out of IBD.
I think by robustness we should go for the second option. Reading https://github.com/bitcoin-core/bitcoin-devwiki/wiki/Wallet-Class-Structure-Changes, best option is to have multiple |
I think you could have separate LegacyScriptPubKeyMan instances for each hdseed, but this would probably be more work than necessary, and not very related to what this PR is trying to do. I think the minimal fix to prevent running out of keys for old hd seeds would be to do some extra checking and topping up inside Actually, I think you would need to do what I just described even if you did want to instantiate different If it is desirable to have separate |
|
I've dealt with the |
|
Whoa awesome, will review #17681 ! |
ryanofsky
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code review ACK a5a674857d0ae6d535e4725337968492c821c635 with caveat that I only did cursory check to verify that CMainSignals::UpdatedBlockTip fInitialDownload value is the same value given by ChainstateActive().IsInitialBlockDownload()
Would be great to see this PR and #17443 rebased and merged as followup to #16426!
src/wallet/wallet.cpp
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In commit "Add m_is_ibd in CWallet" (dca4246dac0f4fa6369a71ee787b2bf1f6415c6d)
Calling chain lock not unnecessary here (but harmless)
src/interfaces/chain.h
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In commit "Remove Chain::isReadyToBroadcast method" (a5a674857d0ae6d535e4725337968492c821c635)
Implementation and meaning of isReadyToBroadcast method is changing a lot, so I would rename it to something like "isInitializing" or "isLoading" and update the comment.
Also commit title could be changed to mention removing isInitialBlockDownload
src/wallet/wallet.cpp
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
re: #17484 (comment)
In commit "Remove Chain::isReadyToBroadcast method" (a5a674857d0ae6d535e4725337968492c821c635)
Perhaps make this change in the commit "Add m_is_ibd in CWallet` (or merge the two commits, IIUC)
Change does make sense (and is necessary) in this commit due to isReadyToBroadcast call a few lines up. But it would be good to add a comment here mentioning this is also needed to avoid spamming nodes with old transactions.
src/wallet/rpcwallet.cpp
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In commit "Add m_is_ibd in CWallet" (dca4246dac0f4fa6369a71ee787b2bf1f6415c6d)
Is the commit message correct? It says there is a behavior change here, but there doesn't seem to be one if this is still raising an error.
|
Thanks @ryanofsky for call-up on this and #17443! I will rebase them tomorrow-ish but I think for this one it would be better for achow one to get first. |
…keys from them as needed 1ed52fb Remove IBD check in sethdseed (Andrew Chow) b1810a1 Test that keys from inactive seeds are generated (Andrew Chow) c93082e Generate new keys for inactive seeds after marking used (Andrew Chow) 45f2f6a Determine inactive HD seeds from key metadata and track them in LegacyScriptPubKeyMan (Andrew Chow) b59b450 have GenerateNewKey and DeriveNewChildKey take a CHDChain as an argument (Andrew Chow) Pull request description: Largely implements the suggestion from #17484 (comment). After `sethdseed` is called, the CHDChain for the old seed is kept in the wallet. It is kept on the file as a new `inactivehdseed` record and in memory in a map `m_inactive_hd_seeds`. In `LegacyScriptPubKeyMan::MarkUnusedAddresses` we check each used key's metadata for whether it was derived from an inactive seed. If it is, we then check to see how many keys after that key were derived from the inactive seed. If that number does not match the keypool parameter, we derive more keys from the inactive seed until it does match. This way we won't miss transactions belonging to keys outside of the range of the keypool initially. The indexes and internal-ness of a key is gotten by checking it's key origin data. Because of this change, we no longer need to wait for IBD to finish before `sethdseed` can work so that check is also removed. A test case for this is added as well which fails on master. ACKs for top commit: ryanofsky: Code review ACK 1ed52fb. Changes since last review: various commit message, code comment, log message, error checking improvements, and fix for topping up inactive seeds if wallet isn't reloaded after calling sethdseed and test for this ariard: Code Review ACK 1ed52fb jonatack: ACK 1ed52fb thanks for addressing the previous review feedback; would be happy to see the new review questions answered and feedback addressed and re-ack. Tree-SHA512: e658ae0e1dab94be55d2b62cdda506c94815e73a6881533fd30d41cc77477f82fee2095144957a3a1df0c129e256bdd7b7abe3737d515f393610446cae4edf1c
…derive keys from them as needed 1ed52fb Remove IBD check in sethdseed (Andrew Chow) b1810a1 Test that keys from inactive seeds are generated (Andrew Chow) c93082e Generate new keys for inactive seeds after marking used (Andrew Chow) 45f2f6a Determine inactive HD seeds from key metadata and track them in LegacyScriptPubKeyMan (Andrew Chow) b59b450 have GenerateNewKey and DeriveNewChildKey take a CHDChain as an argument (Andrew Chow) Pull request description: Largely implements the suggestion from bitcoin#17484 (comment). After `sethdseed` is called, the CHDChain for the old seed is kept in the wallet. It is kept on the file as a new `inactivehdseed` record and in memory in a map `m_inactive_hd_seeds`. In `LegacyScriptPubKeyMan::MarkUnusedAddresses` we check each used key's metadata for whether it was derived from an inactive seed. If it is, we then check to see how many keys after that key were derived from the inactive seed. If that number does not match the keypool parameter, we derive more keys from the inactive seed until it does match. This way we won't miss transactions belonging to keys outside of the range of the keypool initially. The indexes and internal-ness of a key is gotten by checking it's key origin data. Because of this change, we no longer need to wait for IBD to finish before `sethdseed` can work so that check is also removed. A test case for this is added as well which fails on master. ACKs for top commit: ryanofsky: Code review ACK 1ed52fb. Changes since last review: various commit message, code comment, log message, error checking improvements, and fix for topping up inactive seeds if wallet isn't reloaded after calling sethdseed and test for this ariard: Code Review ACK 1ed52fb jonatack: ACK 1ed52fb thanks for addressing the previous review feedback; would be happy to see the new review questions answered and feedback addressed and re-ack. Tree-SHA512: e658ae0e1dab94be55d2b62cdda506c94815e73a6881533fd30d41cc77477f82fee2095144957a3a1df0c129e256bdd7b7abe3737d515f393610446cae4edf1c
|
Concept ACK, will do full review when this is rebased. |
…s_ibd This is only used in next commit.
a5a6748 to
20eb0da
Compare
|
Rebased the PR but this is not worthy to review now, after thinking a bit more I want to avoid behavior changes and for so, it would be better to get #15719 and its follow-up first to always ensure we are in-sync with node wrt to IBD after wallet startup. This current PR may provoke issue if a privacy leak if user broadcast a transaction without being at tip. Updated PR description in consequence. |
ryanofsky
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
re: #17484 (comment)
Seems reasonable to delay this PR until wallet startup works better. In case it's not clear to others, the change in behavior here is that with this PR the wallet may now incorrectly assume the node is in IBD when it's loaded (before it gets an updatedBlockTip notification indicating otherwise). This can cause it to choose transaction lock times differently and be a privacy leak.
But code review ACK 20eb0da aside from this concern. Changes since last review: rebase, adding m_is_ibd comment, moving resend ibd check to earlier commit.
| { // cs_wallet scope | ||
| LOCK(cs_wallet); | ||
|
|
||
| if (m_is_ibd) return; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In commit "[wallet] Add m_is_ibd in CWallet" (797ec9d)
I do think it would be helpful to have comment here saying this is skipped during ibd to avoid spam.
Also the comment above for isReadyToBroadcast() is incorrect after commit "[interfaces] Remove Chain::isReadyToBroadcast" (20eb0da), and should be updated to not mention IBD since the IBD check isn't there anymore
|
🐙 This pull request conflicts with the target branch and needs rebase. Want to unsubscribe from rebase notifications on this pull request? Just convert this pull request to a "draft". |
…ve keys from them as needed Summary: 1ed52fbb4d81f7b7634fd4fb6d1d00e1478129dc Remove IBD check in sethdseed (Andrew Chow) b1810a145a601a8064e4094350cfb6ddafbdb4d8 Test that keys from inactive seeds are generated (Andrew Chow) c93082ece40b1c72f05b3e2085c022c09eaa4d65 Generate new keys for inactive seeds after marking used (Andrew Chow) 45f2f6a0e8514a0438a87554400bf73cbb90707f Determine inactive HD seeds from key metadata and track them in LegacyScriptPubKeyMan (Andrew Chow) b59b4504abf96cec860badfed2ac793ae5d40ced have GenerateNewKey and DeriveNewChildKey take a CHDChain as an argument (Andrew Chow) Pull request description: Largely implements the suggestion from bitcoin/bitcoin#17484 (comment). After `sethdseed` is called, the CHDChain for the old seed is kept in the wallet. It is kept on the file as a new `inactivehdseed` record and in memory in a map `m_inactive_hd_seeds`. In `LegacyScriptPubKeyMan::MarkUnusedAddresses` we check each used key's metadata for whether it was derived from an inactive seed. If it is, we then check to see how many keys after that key were derived from the inactive seed. If that number does not match the keypool parameter, we derive more keys from the inactive seed until it does match. This way we won't miss transactions belonging to keys outside of the range of the keypool initially. The indexes and internal-ness of a key is gotten by checking it's key origin data. Because of this change, we no longer need to wait for IBD to finish before `sethdseed` can work so that check is also removed. A test case for this is added as well which fails on master. --- Backport of Core [[bitcoin/bitcoin#17681 | PR17681]] Test Plan: CC=clang CXX=clang++ cmake .. -GNinja -DENABLE_SANITIZERS=thread ninja all check check-functional cmake .. -GNinja -DCMAKE_BUILD_TYPE=Debug ninja all check check-functional Reviewers: #bitcoin_abc, PiRK Reviewed By: #bitcoin_abc, PiRK Subscribers: PiRK Differential Revision: https://reviews.bitcoinabc.org/D9196
There hasn't been much activity lately and the patch still needs rebase. What is the status here?
|
|
Looks like this depends on #15719 |
There hasn't been much activity lately and the patch still needs rebase. What is the status here?
|
Dependency on #15719
Actually we lock the chain every-time a resend or transaction is broadcast to evaluate if node is currently in-or-out of IBD. This PR proposes instead to rely on
updatedBlockTipto refresh IBD status, therefore increasing asynchronicity between wallet-node and should be a slight performance improvement.