Skip to content

Conversation

@theStack
Copy link
Contributor

@theStack theStack commented Nov 5, 2023

Right now a wallet descriptor is converted to its string representation (via Descriptor::ToString) repeatedly at different instances:

  • on finding a DescriptorScriptPubKeyMan for a given descriptor (CWallet::GetDescriptorScriptPubKeyMan, e.g. used by the importdescriptors RPC); the string representation is created once for each spkm in the wallet and at each iteration again for the searched descriptor (DescriptorScriptPubKeyMan::HasWalletDescriptor)
  • whenever DescriptorScriptPubKeyMan::GetID() is called, e.g. in TopUp or any instances where a descriptor is written to the DB to determine the database key, also at less obvious places like FastWalletRescanFilter etc.

As there is no good reason to calculate a fixed descriptor's string/ID more than once, add the ID as a field to WalletDescriptor and calculate it immediately at initialization (or deserialization). HasWalletDescriptor is changed to compare the spkm's and searched descriptor's ID instead of the string to take use of that.

This speeds up the functional test wallet_miniscript.py by a factor of 5-6x on my machine (3m30.95s on master vs. 0m38.02s on PR). The recently introduced "max-size TapMiniscript" test-case introduced a descriptor that takes 2-3 seconds to create a string representation, so the repeated calls to that were significantly hurting the performance.

Fixes #28800.

Right now a wallet descriptor is converted to it's string representation
(via `Descriptor::ToString`) repeatedly at different instances:
- on finding a `DescriptorScriptPubKeyMan` for a given descriptor
  (`CWallet::GetDescriptorScriptPubKeyMan`, e.g. used by the
  `importdescriptors` RPC); the string representation is created once
  for each spkm in the wallet and at each iteration again for
  the searched descriptor (`DescriptorScriptPubKeyMan::HasWalletDescriptor`)
- whenever `DescriptorScriptPubKeyMan::GetID()` is called, e.g. in
  `TopUp` or any instances where a descriptor is written to the DB
  to determine the database key etc.

As there is no good reason to calculate a fixed descriptor's string/ID
more than once, add the ID as a field to `WalletDescriptor` and
calculate it immediately at initialization (or deserialization).
`HasWalletDescriptor` is changed to compare the spkm's and searched
descriptor's ID instead of the string to take use of that.

This speeds up the functional test `wallet_miniscript.py` by a factor of
5-6x on my machine (3m30.95s on master vs. 0m38.02s on PR). The recently
introduced "max-size TapMiniscript" test-case introduced a descriptor
that takes 2-3 seconds to create a string representation, so the
repeated calls to that were significantly hurting the performance.
@DrahtBot
Copy link
Contributor

DrahtBot commented Nov 5, 2023

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage

For detailed information about the code coverage, see the test coverage report.

Reviews

See the guideline for information on the review process.

Type Reviewers
ACK Sjors, S3RK, BrandonOdiwuor, achow101

If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

Conflicts

Reviewers, this pull request conflicts with the following ones:

  • #28710 (Remove the legacy wallet and BDB dependency by achow101)
  • #22838 (descriptors: Be able to specify change and receiving in a single descriptor string by achow101)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

@Sjors
Copy link
Member

Sjors commented Nov 6, 2023

This fixes #28800 for me and runs way faster under --with-sanitizers=thread.

ACK 5e6bc6d

@S3RK
Copy link
Contributor

S3RK commented Nov 6, 2023

Code Review ACK 5e6bc6d

The change looks correct. I also verified that when we update descriptors with importdescriptor command we replace the whole WalletDescriptor object, so ID remains correct in that case.

Copy link
Contributor

@BrandonOdiwuor BrandonOdiwuor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK 5e6bc6d

looks good to me

@achow101
Copy link
Member

achow101 commented Nov 6, 2023

ACK 5e6bc6d

@achow101 achow101 merged commit 0f5e31c into bitcoin:master Nov 6, 2023
@theStack theStack deleted the 202311-wallet-avoid_repeated_desc_str_id_calculation branch November 7, 2023 18:18
knst pushed a commit to knst/dash that referenced this pull request Aug 29, 2024
…eated descriptor string creation

BACKPORT NOTICE:
It doesn't include changes related to wallet_miniscript.py

5e6bc6d test: remove custom rpc timeout for `wallet_miniscript.py`, reorder in test_runner (Sebastian Falbesoner)
f811a24 wallet: cache descriptor ID to avoid repeated descriptor string creation (Sebastian Falbesoner)

Pull request description:

  Right now a wallet descriptor is converted to its string representation (via `Descriptor::ToString`) repeatedly at different instances:
  - on finding a `DescriptorScriptPubKeyMan` for a given descriptor (`CWallet::GetDescriptorScriptPubKeyMan`, e.g. used by the `importdescriptors` RPC); the string representation is created once for each spkm in the wallet and at each iteration again for the searched descriptor (`DescriptorScriptPubKeyMan::HasWalletDescriptor`)
  - whenever `DescriptorScriptPubKeyMan::GetID()` is called, e.g. in `TopUp` or any instances where a descriptor is written to the DB to determine the database key, also at less obvious places like `FastWalletRescanFilter` etc.

  As there is no good reason to calculate a fixed descriptor's string/ID more than once, add the ID as a field to `WalletDescriptor` and calculate it immediately at initialization (or deserialization). `HasWalletDescriptor` is changed to compare the spkm's and searched descriptor's ID instead of the string to take use of that.

  This speeds up the functional test `wallet_miniscript.py` by a factor of 5-6x on my machine (3m30.95s on master vs. 0m38.02s on PR). The recently introduced "max-size TapMiniscript" test-case introduced a descriptor that takes 2-3 seconds to create a string representation, so the repeated calls to that were significantly hurting the performance.

  Fixes bitcoin#28800.

ACKs for top commit:
  Sjors:
    ACK 5e6bc6d
  S3RK:
    Code Review ACK 5e6bc6d
  achow101:
    ACK 5e6bc6d
  BrandonOdiwuor:
    ACK 5e6bc6d

Tree-SHA512: 98b43963a5dde6055bb26cecd3b878dadd837d6226af4c84142383310495da80b3c4bd552e73b9107f2f2ff1c11f5e18060c6fd3d9e44bbd5224114c4d245c1c
PastaPastaPasta added a commit to dashpay/dash that referenced this pull request Aug 30, 2024
…29007 - to improve CI experience

2eadcf2 partial Merge bitcoin#29007: test: create deterministic addrman in the functional tests (stratospher)
f95ca4e fix: unify with bitcoin: removed requirement of txindex=0 (Konstantin Akimov)
bf46e7a partial Merge bitcoin#28799: wallet: cache descriptor ID to avoid repeated descriptor string creation (Andrew Chow)
05e5966 partial Merge bitcoin#27920: wallet: bugfix, always use apostrophe for spkm descriptor ID (furszy)

Pull request description:

  ## Issue being fixed or feature implemented
  Lately our CI for tsan is flapping many functional tests and take long times.

  This PR has several important changes backported from the latest bitcoin's version to improve CI experience

  ## What was done?
  This PR has several backports that improved CI experience drastically.

  **Firstly, it aims to fix flapping test p2p_node_network_limited.py**

  For example: https://gitlab.com/dashpay/dash/-/jobs/7692635307

  <details>

  <summary>p2p_node_network_limited.py                        | ✖ Failed  | 28 s</summary>

  ```
   test  2024-08-29T02:50:53.929000Z TestFramework (ERROR): Assertion failed
  Traceback (most recent call last):
  File "/builds/dashpay/dash/build-ci/dashcore-linux64_tsan/test/functional/test_framework/test_framework.py", line 158, in main
    self.run_test()
  File "/builds/dashpay/dash/build-ci/dashcore-linux64_tsan/test/functional/p2p_node_network_limited.py", line 79, in run_test
    self.nodes[0].disconnect_p2ps()
  File "/builds/dashpay/dash/build-ci/dashcore-linux64_tsan/test/functional/test_framework/test_node.py", line 611, in disconnect_p2ps
    wait_until_helper(check_peers, timeout=5)
  File "/builds/dashpay/dash/build-ci/dashcore-linux64_tsan/test/functional/test_framework/util.py", line 262, in wait_until_helper
    raise AssertionError("Predicate {} not true after {} seconds".format(predicate_source, timeout))
    AssertionError: Predicate ''''
         def check_peers():
             for p in self.getpeerinfo():
                 for p2p in self.p2ps:
                     if p['subver'] == p2p.strSubVer:
                         return False
             return True
  ''' not true after 5.0 seconds
  ```
  </details>

  **Secondly, it improves performance of Descriptor wallets significantly for case of `tsan` CI**. It is tiny improvement for Release build and local runs, but some fucnctional tests run as fast as twice:

  https://gitlab.com/dashpay/dash/-/jobs/7694458953
  https://gitlab.com/dashpay/dash/-/jobs/7665132625

  ```
  wallet_create_tx.py --descriptors                  | ✓ Passed  | 236 s <-- new version
  wallet_create_tx.py --legacy-wallet                | ✓ Passed  | 108 s
  wallet_basic.py --descriptors                      | ✓ Passed  | 135 s <---- new version
  wallet_basic.py --legacy-wallet                    | ✓ Passed  | 97 s

  wallet_create_tx.py --descriptors                  | ✓ Passed  | 456 s <-- old version
  wallet_create_tx.py --legacy-wallet                | ✓ Passed  | 98 s
  wallet_basic.py --descriptors                      | ✓ Passed  | 189 s <--- old version
  wallet_basic.py --legacy-wallet                    | ✓ Passed  | 131 s
  ```
  See performance investigation here: #6226

  ## How Has This Been Tested?
  Run unit/functional tests

  ## Breaking Changes
  N/A

  ## Checklist:
  - [x] I have performed a self-review of my own code
  - [ ] I have commented my code, particularly in hard-to-understand areas
  - [ ] I have added or updated relevant unit/integration/functional/e2e tests
  - [ ] I have made corresponding changes to the documentation
  - [x] I have assigned this pull request to a milestone

ACKs for top commit:
  PastaPastaPasta:
    utACK 2eadcf2
  UdjinM6:
    utACK 2eadcf2

Tree-SHA512: 127fbaa65c160aa95e2145a6b40d3811f7c42e36fbee9ce98a9ac021abd9cbe6edc7791870b331a54855ba891e3804885db7936ef212647b693f50f79a60d232
@bitcoin bitcoin locked and limited conversation to collaborators Nov 6, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test: wallet_miniscript.py fails with thread sanitizer

6 participants