Skip to content

Conversation

@achow101
Copy link
Member

Currently, various tests use LegacyScriptPubKeyMan because it was convenient for the refactor that introduced the ScriptPubKeyMan interface. However, with the legacy wallet slated to be removed, these tests should not continue to use LegacyScriptPubKeyMan as they are not testing any specific legacy wallet behavior. These tests are changed to use DescriptorScriptPubKeyMans.

Some of the coin selection tests and benchmarks had a global testWallet, but this seemed to cause some issues with ensuring that descriptors were set up in that wallet for each test. Those have been restructured to not have any global variables that may be modified between tests.

The tests which test specific legacy wallet behavior remain unchanged.

To avoid issues with test data leaking across tests cases, the global
vCoins and testWallet are removed from coinselector_tests and all of the
relevant functions reworked to not need them.
For wallet related benchmarks that need a ScriptPubKeyMan for operation,
use a DescriptorScriptPubKeyMan
@DrahtBot
Copy link
Contributor

DrahtBot commented Oct 15, 2021

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

  • #22981 (doc: Fix incorrect C++ named args by MarcoFalke)
  • #22928 (refactor: Remove gArgs from wallet.h and wallet.cpp (2) by kiminuo)
  • #22805 (refactor: use CWallet const shared pointers in dump{privkey,wallet} by theStack)
  • #22787 (refactor: actual immutable pointing by kallewoof)
  • #22260 (Make bech32m the default for RPC, opt-in for GUI by Sjors)
  • #22019 (wallet: Introduce SelectionResult for encapsulating a coin selection solution by achow101)
  • #21206 (refactor: Make CWalletTx sync state type-safe by ryanofsky)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

The subtract fee from outputs assumes that the leftover input amount
will be dropped to fees. However this only happens if that amount is
less than the cost of change. In the event that it is higher than the
cost of change, the leftover amount will actually become a change
output. To avoid this scenario, force a change type which has a high
cost of change.
For the generic wallet tests, make DescriptorScriptPubKeyMans. There are
still some wallet tests that test legacy wallet things. Those remain
unchanged.
Copy link
Contributor

@brunoerg brunoerg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tACK 2d2edc1

Ran benchs and tests successfully

@laanwj
Copy link
Member

laanwj commented Oct 22, 2021

Code review ACK 2d2edc1

@laanwj laanwj merged commit 12eda27 into bitcoin:master Oct 22, 2021
sidhujag pushed a commit to syscoin/syscoin that referenced this pull request Oct 22, 2021
maflcko pushed a commit that referenced this pull request Oct 25, 2021
a52f1d1 walletdb: Use SQLiteDatabase for mock wallet databases (Andrew Chow)
a78c229 tests: Place into mapWallet in coinselector_tests (Andrew Chow)

Pull request description:

  #23288 changed coinselector_tests to use `DescriptorScriptPubKeyMan`, but it also ended up significantly slowing down the test, from 4 seconds to over 1 minute. It appears that the source of this slow down is with `CWallet::AddToWallet`, and primarily due to writing data to the mock wallet database. Because the only thing that is actually needed is for the created transaction to be placed into `CWallet::mapWallet`, this PR removes the call to `AddToWallet` and just places the transaction into `mapWallet` directly. This reduces the test time to 5 seconds.

  To speed things up further, `CreateMockWalletDatabase` is changed to make a `SQLiteDatabase` instead of a `BerkeleyDatabase`. This is safe because there are no tests that require a specific mock database type.

ACKs for top commit:
  brunoerg:
    tACK a52f1d1
  lsilva01:
    tACK a52f1d1. Performed 74.36% better on Ubuntu 20.04 (VM, 12 MB, 8vCPU).
  glozow:
    utACK a52f1d1

Tree-SHA512: da77936bfd2e816d2e71703567b9389d0ee79f3a4a690802ffe3469df5bed371b296cb822b897f625654dab9436d91fd6bc02364a518a47d746e487d70a72595
sidhujag pushed a commit to syscoin/syscoin that referenced this pull request Oct 25, 2021
a52f1d1 walletdb: Use SQLiteDatabase for mock wallet databases (Andrew Chow)
a78c229 tests: Place into mapWallet in coinselector_tests (Andrew Chow)

Pull request description:

  bitcoin#23288 changed coinselector_tests to use `DescriptorScriptPubKeyMan`, but it also ended up significantly slowing down the test, from 4 seconds to over 1 minute. It appears that the source of this slow down is with `CWallet::AddToWallet`, and primarily due to writing data to the mock wallet database. Because the only thing that is actually needed is for the created transaction to be placed into `CWallet::mapWallet`, this PR removes the call to `AddToWallet` and just places the transaction into `mapWallet` directly. This reduces the test time to 5 seconds.

  To speed things up further, `CreateMockWalletDatabase` is changed to make a `SQLiteDatabase` instead of a `BerkeleyDatabase`. This is safe because there are no tests that require a specific mock database type.

ACKs for top commit:
  brunoerg:
    tACK a52f1d1
  lsilva01:
    tACK a52f1d1. Performed 74.36% better on Ubuntu 20.04 (VM, 12 MB, 8vCPU).
  glozow:
    utACK a52f1d1

Tree-SHA512: da77936bfd2e816d2e71703567b9389d0ee79f3a4a690802ffe3469df5bed371b296cb822b897f625654dab9436d91fd6bc02364a518a47d746e487d70a72595
@hebasto
Copy link
Member

hebasto commented Oct 31, 2021

The dcd6eeb commit introduced an intermittent failure in the psbt_wallet_tests/psbt_updater_test unit test.

See:

SignatureData sigdata;
BOOST_CHECK(spk_man->FillPSBT(psbtx, PrecomputePSBTData(psbtx), SIGHASH_ALL, true, true) != TransactionError::OK);
BOOST_CHECK(m_wallet.FillPSBT(psbtx, complete, SIGHASH_ALL, true, true) != TransactionError::OK);
//BOOST_CHECK(spk_man->FillPSBT(psbtx, PrecomputePSBTData(psbtx), SIGHASH_ALL, true, true) != TransactionError::OK);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is this comment for?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removal proposed in #24592.

maflcko pushed a commit to bitcoin-core/gui that referenced this pull request Nov 1, 2021
…sts/psbt_updater_test

68018e4 test: Avoid excessive locking of `cs_wallet` (Hennadii Stepanov)
7986faf test: Fix segfault in the psbt_wallet_tests/psbt_updater_test (Hennadii Stepanov)

Pull request description:

  The dcd6eeb commit (bitcoin/bitcoin#23288) introduced an intermittent failure in the `psbt_wallet_tests/psbt_updater_test` unit test. See bitcoin/bitcoin#23368.

  The test failure can be easily made reproducible with the following patch:
  ```diff
  --- a/src/scheduler.cpp
  +++ b/src/scheduler.cpp
  @@ -57,6 +57,8 @@ void CScheduler::serviceQueue()
               Function f = taskQueue.begin()->second;
               taskQueue.erase(taskQueue.begin());

  +            UninterruptibleSleep(100ms);
  +
               {
                   // Unlock before calling f, so it can reschedule itself or another task
                   // without deadlocking:
  ```

  This PR implements an idea which was mentioned in the [comment](bitcoin/bitcoin#23368 (comment)):
  > Yes, as I said before this looks like a race where the wallet is deleted before stopping the scheduler: [#23368 (comment)](bitcoin/bitcoin#23368 (comment))
  >
  > IIRC, the order should be:
  >
  >    * stop scheduler
  >
  >    * delete wallet
  >
  >    * delete scheduler

  The second commit introduces a refactoring with no behavior change.

  Fixes bitcoin/bitcoin#23368.

ACKs for top commit:
  mjdietzx:
    Code review ACK 68018e4

Tree-SHA512: d9103f6252aab807453628159dec243bc543a2595eecaa04ec761dca3c2370085592c55d6f50967d69a4ac6e8b5827eec30dd9b025132c99b0bb9aa5911ad915
sidhujag pushed a commit to syscoin/syscoin that referenced this pull request Nov 1, 2021
…_updater_test

68018e4 test: Avoid excessive locking of `cs_wallet` (Hennadii Stepanov)
7986faf test: Fix segfault in the psbt_wallet_tests/psbt_updater_test (Hennadii Stepanov)

Pull request description:

  The dcd6eeb commit (bitcoin#23288) introduced an intermittent failure in the `psbt_wallet_tests/psbt_updater_test` unit test. See bitcoin#23368.

  The test failure can be easily made reproducible with the following patch:
  ```diff
  --- a/src/scheduler.cpp
  +++ b/src/scheduler.cpp
  @@ -57,6 +57,8 @@ void CScheduler::serviceQueue()
               Function f = taskQueue.begin()->second;
               taskQueue.erase(taskQueue.begin());

  +            UninterruptibleSleep(100ms);
  +
               {
                   // Unlock before calling f, so it can reschedule itself or another task
                   // without deadlocking:
  ```

  This PR implements an idea which was mentioned in the [comment](bitcoin#23368 (comment)):
  > Yes, as I said before this looks like a race where the wallet is deleted before stopping the scheduler: [bitcoin#23368 (comment)](bitcoin#23368 (comment))
  >
  > IIRC, the order should be:
  >
  >    * stop scheduler
  >
  >    * delete wallet
  >
  >    * delete scheduler

  The second commit introduces a refactoring with no behavior change.

  Fixes bitcoin#23368.

ACKs for top commit:
  mjdietzx:
    Code review ACK 68018e4

Tree-SHA512: d9103f6252aab807453628159dec243bc543a2595eecaa04ec761dca3c2370085592c55d6f50967d69a4ac6e8b5827eec30dd9b025132c99b0bb9aa5911ad915
Fabcien pushed a commit to Bitcoin-ABC/bitcoin-abc that referenced this pull request Jan 25, 2022
…scheduler

Summary:
> The order should be:
>  - stop scheduler
>  - delete wallet
>  - delete scheduler

This is a partial backport of [[bitcoin/bitcoin#23403 | core#23403]]
bitcoin/bitcoin@68018e4

The other commit in that pull request is not applicable to Bitcoin ABC, for now. It has a dependency on [[bitcoin/bitcoin#23288 | core#23288]], which has itself a lot of dependencies on recent descriptor wallets developments.

Test Plan:
With TSAN:
`ninja && ninja check`

Reviewers: #bitcoin_abc, Fabien

Reviewed By: #bitcoin_abc, Fabien

Differential Revision: https://reviews.bitcoinabc.org/D10883
achow101 added a commit that referenced this pull request Mar 16, 2022
9a5b4d7 doc: Delete old line of code that was commented out (Michael Folkson)

Pull request description:

  In #23288 MarcoFalke [highlighted](https://github.com/bitcoin/bitcoin/pull/23288/files#r739817055) an old BOOST_CHECK that was commented out and replaced by a different BOOST_CHECK. I think this can be deleted and wasn't deliberately left in by achow101.

ACKs for top commit:
  achow101:
    ACK 9a5b4d7
  jonatack:
    ACK 9a5b4d7

Tree-SHA512: 69aa71d362bb190c75d617766f6af945a43211ddb315ccfef5ebab2beefe020032d99f0d0d4b312aa349e605ba712a8bc91d7f0a22427681e08c95f8a6ea7e64
sidhujag pushed a commit to syscoin/syscoin that referenced this pull request Mar 16, 2022
9a5b4d7 doc: Delete old line of code that was commented out (Michael Folkson)

Pull request description:

  In bitcoin#23288 MarcoFalke [highlighted](https://github.com/bitcoin/bitcoin/pull/23288/files#r739817055) an old BOOST_CHECK that was commented out and replaced by a different BOOST_CHECK. I think this can be deleted and wasn't deliberately left in by achow101.

ACKs for top commit:
  achow101:
    ACK 9a5b4d7
  jonatack:
    ACK 9a5b4d7

Tree-SHA512: 69aa71d362bb190c75d617766f6af945a43211ddb315ccfef5ebab2beefe020032d99f0d0d4b312aa349e605ba712a8bc91d7f0a22427681e08c95f8a6ea7e64
@bitcoin bitcoin locked and limited conversation to collaborators Mar 16, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants