-
Notifications
You must be signed in to change notification settings - Fork 38.7k
Use internal HD chain for change outputs (hd split) #9294
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
Interesting assertion error in CWallet::CreateRawTransaction |
|
Travis is failing because it did what it is supposed to: finding bugs. |
qa/rpc-tests/keypool.py
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Still three?
qa/rpc-tests/keypool.py
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
three?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
also, this is 100%, not 80% of 6. 100% vs 20%, not 80% vs 20%.
Shall we find out? |
324c7f5 to
1928302
Compare
|
Rebased. |
1928302 to
0971c35
Compare
|
Travis succeeds now. |
luke-jr
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't see code for upgrading existing HD wallets.
Does this implement a BIP that should be mentioned?
src/wallet/wallet.cpp
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Prefer to explicitly add (internal ? 1 : 0)
src/wallet/wallet.cpp
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Seems like it would be better to just access the specific counter once with a post-increment, and use variables here.
src/wallet/wallet.cpp
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
IMO this is unintuitive and can lead to loss. If users set a keypool size of 100, they expect to be able to safely generate 100 addresses between backups. I usually round down to 90 in advice to give some breathing room, but 80% here will break even that.
Therefore, the full keypool size should be ensured on both chains.
Edit: Forgot this was HD. Less critical in that case.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes. This is a discussion point and I'd like to get others feedback (maybe @gmaxwell and @sipa).
The question is, if you want a keypool of 100 external + 50 (TBD) internals = total 150 keys, if you set the keypool to 100.
IMO user given values should be respected. If the user chooses keypool=100, the keypool should contain 100 keys. But as said, no strong opinion on that.
src/wallet/wallet.cpp
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
IMO it would be better to use an enum for internal vs external rather than a bool.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
IMO it would be better to use an enum for internal vs external rather than a bool.
Why? Either its internal or external. Once we have more configurable HD key derivation (ext-pubkey-derivation, hd-multisig) we could switch to a enum or a more flexible design.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
To avoid implicit casting when/if the type changes for this parameter. Also makes it clearer at the call-locations what the parameter is specifying.
409535d to
3817bbe
Compare
|
Overhauled after recommendation from @sipa and @luke-jr.
... this now results in always have 120% keys pre-generated (20% internal key) |
3817bbe to
034ee19
Compare
|
Updated with a small nit-fix (#9294 (comment)) and a fix for wallet-dump.py RPC test. Travis should succeed now. |
034ee19 to
76c883c
Compare
instagibbs
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You might also want to have a test that ensures the ceil behavior is correct for the 100vs20 split as I think the tests only check evenly divisible or the min value of 2.
src/wallet/wallet.cpp
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's called externalChainChildKey but it could be either.
src/wallet/wallet.cpp
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
update comment
src/wallet/wallet.cpp
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
/HD/HD split/?
src/wallet/wallet.cpp
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I find it a bit confusing that this value goes from 0(non-existant) to -keypool then to -keypool*.2. Perhaps have it start at 0 on previous commit, then set it to non-0 here (unless there is reasoning).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I find it a bit confusing that this value goes from 0(non-existant) to -keypool then to -keypool*.2. Perhaps have it start at 0 on previous commit, then set it to non-0 here (unless there is reasoning).
Fixed. I decided to squash the two commit into a single one because the current PR is a non-splittable change.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we just set the target for both internal and external to nTargetSize? It seems super non-user-friendly that the setting which used to mean "can create this many keys/transactions" now means "can create this many keys, or 1/5th as many transactions". I dont think we care all that much about the performance hit, do we?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Some weeks back we discusses that and we came up with something that we should create less internal keys. But I'm fine with 100%/100%. Any objections?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd be fine with reducing the internal keypool significantly once we have rescan logic that can regenerate keys, but until then I think we absolutely need to have it be 100%.
src/wallet/wallet.cpp
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Seems like a separate optimization?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Right. This is no longer required because we always derive at minimum 2 internal keys. And ReserveKeyFromKeyPool will topup the keypool.
Will remove that part.
src/wallet/rpcwallet.cpp
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Might make sense to explicitly count both types and use those counts if we foresee a future with more possible types rather than implicitly count total - external. You'd know better than me how likely that is.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it make sense to keep this as it is: keypoolsize should be alined with the -keypool conf. That's why keypoolsize in getwalletinfo responses only the external key-count in the keypool (-keypool conf arg also defines the external key count).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry, I agree with what you're saying I just wasn't clear enough. I suppose it's along the lines of @luke-jr 's comment about enum, keeping flexible for the future. Not a blocker, just a suggestion. That can be deferred.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Right. This is generally a good idea. IMO the next possible HD extensions (far away) are maybe xpub watch-only-wallets, flexible keypath, hd-multisig.
I can't imagine any change the would require a third (or more then two) type(s) during key derivation as well as in the keypool.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"keypoolsizes": {"internal": N, "external": N} sounds like a good idea here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I though about that, but this would break the API while the current PRs change does not.
qa/rpc-tests/keypool.py
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
also, this is 100%, not 80% of 6. 100% vs 20%, not 80% vs 20%.
76c883c to
4065d5f
Compare
|
Fixed @instagibbs points. Squashed into a single commit. |
|
utACK 4065d5fd44b9ce68a688c82353822a1cf49efe53 |
4065d5f to
0a7b70a
Compare
|
Rebased. |
src/wallet/walletdb.h
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This should set nInternalChainCounter for old versions - maybe use std::numeric_limits<uint32_t>::max() (and make it private, with all accessors throwing if it's not a sufficient version)?
Also, braces, please.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Where do you see the need to set the nInternalChainCounter for old versions here? Old Versions will always have the nInternalChainCounter set the 0 over SetNull() during the constructor call.
A new chain (which then will enable VERSION_HD_CHAIN_SPLIT) can only be created when a new HD master key will be set which again will enable FEATURE_HD_SPLIT.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It should be setup to prevent accidental usage. If any code were to attempt to use it without a wallet that supports it, we want to throw an error/exception, not silently generate keys we may not be able to recover.
src/wallet/wallet.cpp
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
To avoid implicit casting when/if the type changes for this parameter. Also makes it clearer at the call-locations what the parameter is specifying.
src/wallet/wallet.cpp
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is there a reason to do this check here, rather than inside DeriveNewChildKey?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A design question, but IMO DeriveNewChildKey should execute the derivation and not care about wallet versions and supported features.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe move it to private: then?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yea, really should be private, I think.
src/wallet/wallet.cpp
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Seems like this could be significantly simplified with:
uint32_t &nChainCounter = (internal ? hdChain.nInternalChainCounter : hdChain.nExternalChainCounter);There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since almost everything here is different depending on internal/external, I think it'd be clearer to just make it if (internal) and separate the behaviors
src/wallet/wallet.cpp
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this won't work if the user specifies a HD-but-not-split wallet version for -walletupgrade?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we should no longer allow non split HD versions.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This also forcibly upgrades the user's wallet (if from pre-split HD version) if they encrypt their wallet, which I think is unaccptable.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Note that if you change this need to set the newHdChain's version to the appropriate value prior to SetHDChain.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We could... one question is: if the users encrypt his 0.13 wallet (HD enabled but no chain split) in 0.14. Do we want to keep the non-hd split or do we want to enforce an upgrade to HD_Split. The later seems preferable from the users perspective (Is there a reason to not split the HD chains if we can?).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I believe it is policy to not upgrade a user's wallet unless they have specifically requested we do so, so, no, I dont think we should use HD_Split at that point.
src/wallet/wallet.h
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we want internal to have a default? Should it be false? IIRC currently new keys default to change unless added to the address book, so this default seems to contradict the status quo.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have set the default to false to keep the exiting behaviour.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
My point is that the existing behaviour is closer to true than false.
src/wallet/walletdb.h
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Missing logic to initialise this version.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
VERSION_HD_BASE is only there to identify versions that do not support HD SPLIT (first HD version introduced in 0.13).
src/wallet/rpcwallet.cpp
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
keypoolsize should probably be the lower of the two and deprecated. There is no reason to assume prior use didn't look at it for info on change keys.
src/wallet/rpcwallet.cpp
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"keypoolsizes": {"internal": N, "external": N} sounds like a good idea here.
src/wallet/rpcwallet.cpp
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Shouldn't this use CanSupportFeature(FEATURE_HD_SPLIT)?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think always reporting keypoolsize_hd_internal (with 0 in non HD-SPLIT case) can make parsing simpler.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Then always report it? Right now it's conditional on something more or less irrelevant.
But I'm not sure it makes sense to ever report 0 when there are indeed keys that will be used for internal outputs...
|
Added a commit that addresses some of @luke-jr points. |
src/wallet/rpcwallet.cpp
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think the 10% number is wrong? current code is 20, I believe? (and I think it should be changed to 100).
src/wallet/rpcwallet.cpp
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we really need to update this - keypoololdest is used to know when your backup has been invalidated. I know technically now that we have HD wallets arent invalidated, but until we have rebuilding-from-chain implemented I think we really need to make sure keypoololdest is still useable as "if your backup is older than this date, backup again".
This means keypoololdest needs to be max(oldest internal keypool, oldest external keypool) isntead.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agree. But probably in a separate PR.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not doing this would break some uses of walletbackup, I think, so I wouldnt be too happy with this slipping into a different release.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe add a TODO comment (though this does seem to me like a desirable change to include in this PR).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe add a TODO comment (though this does seem to me like a desirable change to include in this PR).
This is fixed in the current version.
…(oldest-internal-key, oldest-external-key)
…to getrawchangeaddress())
78f2cb9 to
1b3b5c6
Compare
|
Addressed @TheBlueMatt's nits (except #9294 (comment) which I'm not sure if this is a concern or not) |
|
re utACK 1df08d1 |
|
Added a commit to address the backward compatibility issue during wallet encryption mentioned by @TheBlueMatt (#9294 (comment)). |
src/wallet/rpcwallet.cpp
Outdated
|
|
||
| UniValue obj(UniValue::VOBJ); | ||
|
|
||
| size_t kpExternalSize = pwalletMain->KeypoolCountExternalKeys(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oops, should be pwallet, not pwalletMain now :).
| READWRITE(this->nVersion); | ||
| READWRITE(nExternalChainCounter); | ||
| if (this->nVersion >= VERSION_HD_CHAIN_SPLIT) | ||
| READWRITE(nInternalChainCounter); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In addition to the fix for backward compat where you set the version, can we make this more robust and flip the order? Generally adding new fields for (de-)serialization we should be adding them to the end.
| if (!SetHDMasterKey(masterPubKey)) | ||
| // preserve the old chains version to not break backward compatibility | ||
| CHDChain oldChain = GetHDChain(); | ||
| if (!SetHDMasterKey(masterPubKey, &oldChain)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why not SetHDMasterKey(int nHDChainVersion)? Then you can just do CanSupportFeature(FEATURE_HD_SPLIT) ? CHDChain::VERSION_HD_CHAIN_SPLIT : CHDChain::VERSION_HD_BASE.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I though hiding the internals at this point would be preferable. IMO CWallet::encryptWallet should not have direct knowhow of the CHDChain version handling. I'd prefer passing in the old chain and let it be handled through SetHDMasterKey. Also, in future maybe other attributes need to be preserved.
Not sure although.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It kinda feels like mixing models, then. Wallet checks for FEATURE_HD_SPLIT everywhere before it updates the nInternalCounts, ie it seems to think its responsible for doing the checks, only to have CHDChain hide some stuff like version, but then you have to copy the version? Either we move logic into CHDChain (as @luke-jr suggested previously, have CHDChain know what version means and assert if you try to use the internal counter with a version that doesnt support it, which I assume some folks would object to), or all the logic should be in CWallet.
| for(const int64_t& id : setKeyPool) | ||
| { | ||
| if (!walletdb.ReadPool(id, keypool)) { | ||
| throw std::runtime_error(std::string(__func__) + ": read failed"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: the old version does an assert(keypool.vchPubKey.IsValid()); which is nice to have...not necessary to add, but would be cool.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For the keypool oldest it's further down L3026 (at https://github.com/bitcoin/bitcoin/pull/9294/files#diff-b2bb174788c7409b671c46ccc86034bdL3026)
Ser./Deser. nInternalChainCounter as last element
|
Fixed two points reported by @TheBlueMatt:
|
|
utACK 4115af7 |
4115af7 Fix rebase issue where pwalletMain was used instead of pwallet Ser./Deser. nInternalChainCounter as last element (Jonas Schnelli) 9382f04 Do not break backward compatibility during wallet encryption (Jonas Schnelli) 1df08d1 Add assertion for CanSupportFeature(FEATURE_HD_SPLIT) (Jonas Schnelli) cd468d0 Define CWallet::DeriveNewChildKey() as private (Jonas Schnelli) ed79e4f Optimize GetOldestKeyPoolTime(), return as soon as we have both oldest keys (Jonas Schnelli) 771a304 Make sure we set the wallets min version to FEATURE_HD_SPLIT at the very first point (Jonas Schnelli) 1b3b5c6 Slightly modify fundrawtransaction.py test (change getnewaddress() into getrawchangeaddress()) (Jonas Schnelli) 003e197 Remove FEATURE_HD_SPLIT bump TODO (Jonas Schnelli) d9638e5 Overhaul the internal/external key derive switch (Jonas Schnelli) 1090502 Fix superfluous cast and code style nits in RPC wallet-hd.py test (Jonas Schnelli) 58e1483 CKeyPool avoid "catch (...)" in SerializationOp (Jonas Schnelli) e138876 Only show keypoolsize_hd_internal if HD split is enabled (Jonas Schnelli) add38d9 GetOldestKeyPoolTime: if HD & HD Chain Split is enabled, response max(oldest-internal-key, oldest-external-key) (Jonas Schnelli) dd526c2 Don't switch to HD-chain-split during wallet encryption of non HD-chain-split wallets (Jonas Schnelli) 79df9df Switch to 100% for the HD internal keypool size (Jonas Schnelli) bcafca1 Make sure we always generate one keypool key at minimum (Jonas Schnelli) d0a627a Fix issue where CDataStream->nVersion was taken a CKeyPool record version (Jonas Schnelli) 9af8f00 Make sure we hand out keypool keys if HD_SPLIT is not enabled (Jonas Schnelli) 469a47b Make sure ReserveKeyFromKeyPool only hands out internal keys if HD_SPLIT is supported (Jonas Schnelli) 05a9b49 Fix wrong keypool internal size in RPC getwalletinfo help (Jonas Schnelli) 01de822 Removed redundant IsLocked() check in NewKeyPool() (Jonas Schnelli) d59531d Immediately return setKeyPool's size if HD or HD_SPLIT is disabled or not supported (Jonas Schnelli) 02592f4 [Wallet] split the keypool in an internal and external part (Jonas Schnelli) Tree-SHA512: 80d355d5e844b48c3163b56c788ab8b5b5285db0ceeb19858a3ef517d5a702afeca21dbae526d7b8fb4101c2a745af1d92bf557c40cf516780f17992bf678c1a
Main changes
This pull request will result in using
m/0'/1'/k'for internal keys (change outputs) while keepingm/0'/0'/k'for external keys (getnewaddress).CKeyPool) have now a flag (fInternal).The keypool will be filled with 80% external keys (round ceil).getwalletinfoadditionally reportskeypoolsize_hd_internal(amount of internal key in theCompatibility:
m/0'/0'/k').This change also fixes the keypool +1 offset.