Skip to content

Conversation

@random-zebra
Copy link

Going on with the coins view cache update work (started with #1774 and #1777).
This backports the following upstream PRs (adapting them with the assumption that duplicate coinbase transactions are not possible):

Note: Will rebase #1773 on top of this one (to change ApplyTxInUndo return value in the unit test too).

random-zebra and others added 4 commits August 12, 2020 12:00
>>> backports bitcoin/bitcoin@07df40b

Add more comprehensive unit tests for CCoinsViewCache. Right now it is
hard to refactor caching code or fix bugs in the caching logic because
you have to try to mentally enumerate all the different states the cache
might be in to make sure a change doesn't cause unintended consequences.
The new tests explicitly enumerate relevant cache states, documenting
and verifying the behavior in each state, so it will be safer and easier
to make changes to the caching code in the future.
>>> backports bitcoin/bitcoin@b50cd7a

We were marking coins FRESH before being sure they were not overwriting
dirty undo data. This condition was never reached in existing code
because undo data was always flushed before UpdateCoins was called with
new transactions, but could have been exposed in an otherwise safe
refactor.
Clarify in the comments the assumptions made in ModifyNewCoins.
Add ability to undo transactions to UpdateCoins unit test.
Thanks to Russ Yanofsky for suggestion on how to make logic clearer and
fixing up the ccoins_modify_new test cases.
Copy link

@furszy furszy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pretty nice one, ACK cdbb1a6

Copy link
Collaborator

@Fuzzbawls Fuzzbawls left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK cdbb1a6

@random-zebra random-zebra merged commit af793b7 into PIVX-Project:master Aug 18, 2020
furszy added a commit that referenced this pull request Aug 19, 2020
83512d2 [Trivial] Mark CCoinsView child classes methods as override (random-zebra)
5643075 Store/allow tx metadata in all undo records (random-zebra)
8c4add4 Report on-disk size in gettxoutsetinfo (random-zebra)
6cb2989 Remove/ignore tx version in utxo and undo (random-zebra)
03b9724 Add specialization of SipHash for 256 + 32 bit data (Pieter Wuille)
e3cd496 Introduce CHashVerifier to hash read data (random-zebra)
10baa70 error() in disconnect for disk corruption, not inconsistency (random-zebra)
802f8c8 Add SizeEstimate to CDBBatch (Pieter Wuille)

Pull request description:

  This backports the commits listed as "Preparation" in bitcoin#10195.
  The rest will be included in a followup PR.

  Built on top of:
  - [x] #1772
  - [x] #1771
  - [x] #1793

  Here we:
  - add in-memory size estimation of a LevelDB batch
  - remove the usage of `error()` in ApplyTxInUndo/DisconnectBlock
  - Introduce the new CHashVerifier class to hash read data
  - Add a specialized version of SipHash for tuples of 256 bits and 32 bits data
  - Remove the transaction version from utxo and undo data structures
  - Add on-disk size of the utxo set to gettxoutsetinfo
  - store tx metadata in all undo records, instead of records corresponding only to the last output of a tx being spent.

ACKs for top commit:
  furszy:
    looking good, utACK 83512d2.
  Fuzzbawls:
    utACK 83512d2

Tree-SHA512: 7db730058ae208535aad5f5989abb9c55fd3722b79efef24fdd8c717687b67272bddaeae5e5a4a2d3e97626a7ec62a5d1ebdb5af0233f4a950b5a0c0b154484f
random-zebra added a commit that referenced this pull request Sep 5, 2020
535d8e4 scripted-diff: various renames for per-utxo consistency (random-zebra)
60c73ad Rename CCoinsCacheEntry::coins to coin (Pieter Wuille)
1166f1f Merge CCoinsViewCache's GetOutputFor and AccessCoin (Pieter Wuille)
f25d3c6 [MOVEONLY] Move old CCoins class to txdb.cpp (random-zebra)
aab17b3 Upgrade from per-tx database to per-txout (Pieter Wuille)
a55eb98 Reduce reserved memory space for flushing (random-zebra)
7d637ea Remove unused CCoins methods (random-zebra)
4b7b1a3 Extend coins_tests (random-zebra)
8cf2dc0 Switch CCoinsView and chainstate db from per-txid to per-txout (random-zebra)
b20a662 [Cleanup] Remove unused CCoinsViewCache::IsOutputAvailable (random-zebra)
ea82855 Refactor GetUTXOStats in preparation for per-COutPoint iteration (Pieter Wuille)
0f9f2b6 [Core] Fix not-pruned-check in miner for zerocoin mint outputs (random-zebra)
3698d47 Replace CCoins-based CTxMemPool::pruneSpent with isSpent (Pieter Wuille)
11c728a Remove ModifyCoins/ModifyNewCoins (Pieter Wuille)
3c65936 Switch tests from ModifyCoins to AddCoin/SpendCoin (random-zebra)
b05fe55 Switch CScriptCheck to use Coin instead of CCoins (random-zebra)
9e5c2ae Only pass things committed to by tx's witness hash to CScriptCheck (random-zebra)
22b0b4a [Cleanup] fix warning in comparisons with nCoinbaseMaturity (random-zebra)
666081d Switch from per-tx to per-txout CCoinsViewCache methods in some places (random-zebra)
53fc2be [Core] PIVX: remove potential_overwrite: BIP34 has always been enforced (random-zebra)
16924aa Introduce new per-txout CCoinsViewCache functions (random-zebra)
2d74bc1 Optimization: Coin&& to ApplyTxInUndo (random-zebra)
3fcb092 Replace CTxInUndo with Coin (random-zebra)
42a8996 [Core] PIVX: Add fCoinStake boolean field to Coin class (random-zebra)
a01a9f2 Introduce Coin, a single unspent output (random-zebra)

Pull request description:

  The chainstate database, and its cache, rely on a per-tx model (i.e. it is a map from txids to *lists* of unspent outputs).
  This PR chages it to a per-txout model (i.e. a map from outpoints to *single* unspent outputs).

  Pros:
  - Simpler code.
  - Avoiding the CPU overhead of deserializing and serializing the unused outputs.
  - More predictable memory usage.
  - More easily adaptable to various cache flushing strategies.

  Cons:
  - Slightly larger on-disk representation, and sometimes larger in-memory representation (when there are multiple outputs for the same txid in the cache, which becomes optional).

  Adapted from bitcoin#10195 (adding a `fCoinStake` boolean memeber to the Coin class and considering BIP34 always in effect as per #1775)

  Based on top of:
  - [x] #1799
  - [x] #1800
  - [x] #1797
  - [x] #1796
  - [x] #1795
  - [x] #1793
  - [x] #1773

  The actual PR starts with "Introduce Coin, a single unspent output" (171bf0b8a55de0e3f94305b33e97bb1cf4a01b70)

  ⚠️  **Note for testers**: Do a backup of the chainstate before testing. With this PR, at startup, the database is automatically upgraded to the new format (so a reindex is not needed).

ACKs for top commit:
  furszy:
    Code review ACK 535d8e4. Need some live testing now.
  furszy:
    ACK 535d8e4
  Fuzzbawls:
    ACK 535d8e4

Tree-SHA512: 8648d31c343ff901f8568c14d0848241c9b8aca0b9a8f89f3e5dbe02d384ba35faf572757e84f979fba1f198d721a2516d0e784b5e0dc8d02b6e308b4278b484
@random-zebra random-zebra modified the milestones: 5.0.0, 4.3.0 Sep 10, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants