-
Notifications
You must be signed in to change notification settings - Fork 38.7k
wallet: cache descriptor ID to avoid repeated descriptor string creation #28799
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
wallet: cache descriptor ID to avoid repeated descriptor string creation #28799
Conversation
Right now a wallet descriptor is converted to it's string representation (via `Descriptor::ToString`) repeatedly at different instances: - on finding a `DescriptorScriptPubKeyMan` for a given descriptor (`CWallet::GetDescriptorScriptPubKeyMan`, e.g. used by the `importdescriptors` RPC); the string representation is created once for each spkm in the wallet and at each iteration again for the searched descriptor (`DescriptorScriptPubKeyMan::HasWalletDescriptor`) - whenever `DescriptorScriptPubKeyMan::GetID()` is called, e.g. in `TopUp` or any instances where a descriptor is written to the DB to determine the database key etc. As there is no good reason to calculate a fixed descriptor's string/ID more than once, add the ID as a field to `WalletDescriptor` and calculate it immediately at initialization (or deserialization). `HasWalletDescriptor` is changed to compare the spkm's and searched descriptor's ID instead of the string to take use of that. This speeds up the functional test `wallet_miniscript.py` by a factor of 5-6x on my machine (3m30.95s on master vs. 0m38.02s on PR). The recently introduced "max-size TapMiniscript" test-case introduced a descriptor that takes 2-3 seconds to create a string representation, so the repeated calls to that were significantly hurting the performance.
|
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers. Code CoverageFor detailed information about the code coverage, see the test coverage report. ReviewsSee the guideline for information on the review process.
If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update. ConflictsReviewers, this pull request conflicts with the following ones:
If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first. |
|
Code Review ACK 5e6bc6d The change looks correct. I also verified that when we update descriptors with |
BrandonOdiwuor
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ACK 5e6bc6d
looks good to me
|
ACK 5e6bc6d |
…eated descriptor string creation BACKPORT NOTICE: It doesn't include changes related to wallet_miniscript.py 5e6bc6d test: remove custom rpc timeout for `wallet_miniscript.py`, reorder in test_runner (Sebastian Falbesoner) f811a24 wallet: cache descriptor ID to avoid repeated descriptor string creation (Sebastian Falbesoner) Pull request description: Right now a wallet descriptor is converted to its string representation (via `Descriptor::ToString`) repeatedly at different instances: - on finding a `DescriptorScriptPubKeyMan` for a given descriptor (`CWallet::GetDescriptorScriptPubKeyMan`, e.g. used by the `importdescriptors` RPC); the string representation is created once for each spkm in the wallet and at each iteration again for the searched descriptor (`DescriptorScriptPubKeyMan::HasWalletDescriptor`) - whenever `DescriptorScriptPubKeyMan::GetID()` is called, e.g. in `TopUp` or any instances where a descriptor is written to the DB to determine the database key, also at less obvious places like `FastWalletRescanFilter` etc. As there is no good reason to calculate a fixed descriptor's string/ID more than once, add the ID as a field to `WalletDescriptor` and calculate it immediately at initialization (or deserialization). `HasWalletDescriptor` is changed to compare the spkm's and searched descriptor's ID instead of the string to take use of that. This speeds up the functional test `wallet_miniscript.py` by a factor of 5-6x on my machine (3m30.95s on master vs. 0m38.02s on PR). The recently introduced "max-size TapMiniscript" test-case introduced a descriptor that takes 2-3 seconds to create a string representation, so the repeated calls to that were significantly hurting the performance. Fixes bitcoin#28800. ACKs for top commit: Sjors: ACK 5e6bc6d S3RK: Code Review ACK 5e6bc6d achow101: ACK 5e6bc6d BrandonOdiwuor: ACK 5e6bc6d Tree-SHA512: 98b43963a5dde6055bb26cecd3b878dadd837d6226af4c84142383310495da80b3c4bd552e73b9107f2f2ff1c11f5e18060c6fd3d9e44bbd5224114c4d245c1c
…29007 - to improve CI experience 2eadcf2 partial Merge bitcoin#29007: test: create deterministic addrman in the functional tests (stratospher) f95ca4e fix: unify with bitcoin: removed requirement of txindex=0 (Konstantin Akimov) bf46e7a partial Merge bitcoin#28799: wallet: cache descriptor ID to avoid repeated descriptor string creation (Andrew Chow) 05e5966 partial Merge bitcoin#27920: wallet: bugfix, always use apostrophe for spkm descriptor ID (furszy) Pull request description: ## Issue being fixed or feature implemented Lately our CI for tsan is flapping many functional tests and take long times. This PR has several important changes backported from the latest bitcoin's version to improve CI experience ## What was done? This PR has several backports that improved CI experience drastically. **Firstly, it aims to fix flapping test p2p_node_network_limited.py** For example: https://gitlab.com/dashpay/dash/-/jobs/7692635307 <details> <summary>p2p_node_network_limited.py | ✖ Failed | 28 s</summary> ``` test 2024-08-29T02:50:53.929000Z TestFramework (ERROR): Assertion failed Traceback (most recent call last): File "/builds/dashpay/dash/build-ci/dashcore-linux64_tsan/test/functional/test_framework/test_framework.py", line 158, in main self.run_test() File "/builds/dashpay/dash/build-ci/dashcore-linux64_tsan/test/functional/p2p_node_network_limited.py", line 79, in run_test self.nodes[0].disconnect_p2ps() File "/builds/dashpay/dash/build-ci/dashcore-linux64_tsan/test/functional/test_framework/test_node.py", line 611, in disconnect_p2ps wait_until_helper(check_peers, timeout=5) File "/builds/dashpay/dash/build-ci/dashcore-linux64_tsan/test/functional/test_framework/util.py", line 262, in wait_until_helper raise AssertionError("Predicate {} not true after {} seconds".format(predicate_source, timeout)) AssertionError: Predicate '''' def check_peers(): for p in self.getpeerinfo(): for p2p in self.p2ps: if p['subver'] == p2p.strSubVer: return False return True ''' not true after 5.0 seconds ``` </details> **Secondly, it improves performance of Descriptor wallets significantly for case of `tsan` CI**. It is tiny improvement for Release build and local runs, but some fucnctional tests run as fast as twice: https://gitlab.com/dashpay/dash/-/jobs/7694458953 https://gitlab.com/dashpay/dash/-/jobs/7665132625 ``` wallet_create_tx.py --descriptors | ✓ Passed | 236 s <-- new version wallet_create_tx.py --legacy-wallet | ✓ Passed | 108 s wallet_basic.py --descriptors | ✓ Passed | 135 s <---- new version wallet_basic.py --legacy-wallet | ✓ Passed | 97 s wallet_create_tx.py --descriptors | ✓ Passed | 456 s <-- old version wallet_create_tx.py --legacy-wallet | ✓ Passed | 98 s wallet_basic.py --descriptors | ✓ Passed | 189 s <--- old version wallet_basic.py --legacy-wallet | ✓ Passed | 131 s ``` See performance investigation here: #6226 ## How Has This Been Tested? Run unit/functional tests ## Breaking Changes N/A ## Checklist: - [x] I have performed a self-review of my own code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have added or updated relevant unit/integration/functional/e2e tests - [ ] I have made corresponding changes to the documentation - [x] I have assigned this pull request to a milestone ACKs for top commit: PastaPastaPasta: utACK 2eadcf2 UdjinM6: utACK 2eadcf2 Tree-SHA512: 127fbaa65c160aa95e2145a6b40d3811f7c42e36fbee9ce98a9ac021abd9cbe6edc7791870b331a54855ba891e3804885db7936ef212647b693f50f79a60d232
Right now a wallet descriptor is converted to its string representation (via
Descriptor::ToString) repeatedly at different instances:DescriptorScriptPubKeyManfor a given descriptor (CWallet::GetDescriptorScriptPubKeyMan, e.g. used by theimportdescriptorsRPC); the string representation is created once for each spkm in the wallet and at each iteration again for the searched descriptor (DescriptorScriptPubKeyMan::HasWalletDescriptor)DescriptorScriptPubKeyMan::GetID()is called, e.g. inTopUpor any instances where a descriptor is written to the DB to determine the database key, also at less obvious places likeFastWalletRescanFilteretc.As there is no good reason to calculate a fixed descriptor's string/ID more than once, add the ID as a field to
WalletDescriptorand calculate it immediately at initialization (or deserialization).HasWalletDescriptoris changed to compare the spkm's and searched descriptor's ID instead of the string to take use of that.This speeds up the functional test
wallet_miniscript.pyby a factor of 5-6x on my machine (3m30.95s on master vs. 0m38.02s on PR). The recently introduced "max-size TapMiniscript" test-case introduced a descriptor that takes 2-3 seconds to create a string representation, so the repeated calls to that were significantly hurting the performance.Fixes #28800.