Details

Reviewers

Group Reviewers

Restricted Project

Commits

rABCeacbda759109: Allow maintaining the blockfilterindex when using prune

Summary

PR description:

Maintaining the blockfilterindexes in prune mode is possible and may lead to efficient p2p based rescans of wallets (restore backups, import/sweep keys) beyond the prune height (rescans not part of that PR).

This PR allows running the blockfilterindex(es) in conjunction with pruning.

Bitcoind/Qt will shutdown during startup when missing block data has been detected ([re]enable -blockfilterindex when we already have pruned)

manual block pruning is disabled during blockfilterindex sync

auto-pruning is delayed during blockfilterindex sync

Allow blockfilter in conjunction with prune

https://github.com/bitcoin/bitcoin/pull/15946/commits/6abe9f5b11cd4a5ecb6caca8443fe2950a417842

index: Fix backwards search for bestblock

https://github.com/bitcoin/bitcoin/pull/23365/commits/698c524698c33595a4d555eaa9e21bc19b4d3e93

Avoid accessing nullpointer in BaseIndex::GetSummary()

https://github.com/bitcoin/bitcoin/pull/15946/commits/00d57ff76854938ead800767fb673a8af46eac8e

Avoid pruning below the blockfilterindex sync height

https://github.com/bitcoin/bitcoin/pull/15946/commits/5e112269c311a559bfded814d3c3c438349a1986

Add debug startup parameter -fastprune for more effective pruning tests

https://github.com/bitcoin/bitcoin/pull/15946/commits/00d57ff76854938ead800767fb673a8af46eac8e

Add functional test for blockfilterindex in prune-mode

https://github.com/bitcoin/bitcoin/pull/15946/commits/ab3a0a2fb915d8b8384c30a8b38b4b5cc35236fd

Fix several bugs in feature_blockfilterindex_prune.py

core#21230

test: Intermittent issue in feature_blockfilterindex_prune

core#21252

test: improve assertions in feature_blockfilterindex_prune.py
test: remove unneeded node from feature_blockfilterindex_prune.py

core#21297

test: Add edge case of pruning up to index height

https://github.com/bitcoin/bitcoin/pull/23365/commits/9600ea01450b0d39be90eb2971c1ac5c9b69a66e

This is a backport of core#15946, core#21230, core#21252, core#21297 and core#23365

Backport notes:

the test was buggy and ugly code after the first commit, so I had to squash all these commits to reach an acceptable quality.
I had to use different numbers of blocks that are generated and pruned, because Bitcoin ABC can fit more blocks into each blk?????.dat file than core because in this test the witness data of the coinbase transaction makes core blocks larger. Comments were added to explain this where needed.
I used the shortcut node = self.nodes[0] to make the code a bit more readable (shorter lines)

Test Plan

ninja all check-all

Run IBD with pruning enabled
src/bitcoind -prune=2000 -blockfilterindex=1

Diff Detail

Repository

rABC Bitcoin ABC

Lint

Lint Not Applicable

Unit

Tests Not Applicable

Event Timeline

PiRK requested review of this revision.Apr 4 2022, 07:51

PiRK created this revision.

teamcity edited the summary of this revision. (Show Details)Apr 4 2022, 07:51

PiRK edited the summary of this revision. (Show Details)Apr 4 2022, 07:53

Harbormaster completed remote builds in B18719: Diff 33056.Apr 4 2022, 07:54

Build Bitcoin ABC Diffs / Diff Testing (lint-circular-dependencies) failed.

Tail of the build log:

/work /work/abc-ci-builds/lint-circular-dependencies
A new circular dependency in the form of "index/blockfilterindex -> validation -> index/blockfilterindex" appears to have been introduced.

/work/abc-ci-builds/lint-circular-dependencies
Build lint-circular-dependencies failed with exit code 1

clean-up redundant comments. Use capitalized first words in sentence.

PiRK edited the summary of this revision. (Show Details)Apr 4 2022, 07:58

PiRK edited the summary of this revision. (Show Details)Apr 4 2022, 08:01

PiRK mentioned this in D11147: test: Add edge case of pruning up to index height.Apr 4 2022, 08:04

PiRK mentioned this in D11146: Add functional test for blockfilterindex in prune-mode.

PiRK mentioned this in D11145: Allow maintaining the blockfilterindex when using prune.

Harbormaster completed remote builds in B18720: Diff 33057.Apr 4 2022, 08:06

Build Bitcoin ABC Diffs / Diff Testing (lint-circular-dependencies) failed.

Tail of the build log:

/work /work/abc-ci-builds/lint-circular-dependencies
A new circular dependency in the form of "index/blockfilterindex -> validation -> index/blockfilterindex" appears to have been introduced.

/work/abc-ci-builds/lint-circular-dependencies
Build lint-circular-dependencies failed with exit code 1

need to figure out what the circular dependency is about

add expected circular dependency https://github.com/bitcoin/bitcoin/pull/15946/commits/84716b134e9afca2fba7731de4af1f22fa1b1518

Harbormaster completed remote builds in B18721: Diff 33058.Apr 4 2022, 08:27

Fabien edited the summary of this revision. (Show Details)Apr 5 2022, 08:41

PiRK added a child revision: D11303: doc: add explanations for assert in index and magic numbers in test.Apr 5 2022, 09:23

Fabien requested changes to this revision.Apr 5 2022, 09:48

Fabien added inline comments.