validation: do not wipe utxo cache for stats/scans/snapshots #33680

l0rinc · 2025-10-22T14:46:09Z

Revival of #30610 (comment) with the remaining comments applied on top

Since #28280, the cost of a non-wiping sync of the UTXO cache is only proportional to the number of dirty entries, rather than proportional to the size of the entire cache. Because of that, there is no reason to perform a wiping flush in case the contents of the cache is still useful.

Split the FlushStateMode::ALWAYS mode into a FORCE_SYNC (non-wiping) and a FORCE_FLUSH (wiping), and then use the former in scantxoutset, gettxoutsetinfo, snapshot creation.

(slightly updated after #30214)

DrahtBot · 2025-10-22T14:46:14Z

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage & Benchmarks

For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/33680.

Reviews

See the guideline for information on the review process.

Type	Reviewers
ACK	cedwies, optout21

If your review is incorrectly listed, please copy-paste  into the comment that the bot should ignore.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#34125 (validation: reuse should_empty check for chainstate flush by l0rinc)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

cedwies

Approach ACK

Do ForceFlushToDisk(...)/FlushToDisk(...) require cs_main to be held?
Most call sites hold it during shutdown flushes, but the function itself doesn’t assert or document that (no AssertLockHeld(cs_main)). If this is correct, I think a short comment can help that a Lock is not strictly needed.

cedwies · 2025-10-28T09:28:36Z

src/test/fuzz/utxo_snapshot.cpp

@@ -57,7 +57,7 @@ void sanity_check_snapshot()
    // Connect the chain to the tmp chainman and sanity check the chainparams snapshot values.
    LOCK(cs_main);
    auto& cs{node.chainman->ActiveChainstate()};
-    cs.ForceFlushStateToDisk();
+    cs.ForceFlushStateToDisk(/*wipe_cache=*/true);


Why are we wiping the cache here? We only compute stats from CoinsDB(), not the in-memory cache, so a non-wiping flush (wipe_cache=false) should be enough for consistency (?)

Good point, changed to false, added you as coauthor, thanks.

src/validation.cpp

optout21 · 2025-11-01T05:52:31Z

I suggest splitting into two commits:

Preparatory rename & new parameter(s) but without behavior change, to minimize diff in 2nd commit (rename ALWAYS to FORCE_FLUSH I suppose)
Change in behavior in the respective places (scantxoutset, gettxoutsetinfo, etc.)

l0rinc · 2025-11-01T21:26:09Z

Thanks @optout21 & @cedwies, split the change to 2 commits, one just renames the old value and adjusts the docs and scripts, the second introduces the new value.
I'm not sure about the locking, let's investigate that as a follow-up.

optout21 · 2025-11-03T10:50:32Z

contrib/tracing/README.md

-637657          ALWAYS     122320          17124.80 kB     False
-81349           ALWAYS     0               1383.49 kB      False
+Duration (µs)   Mode         Coins Count     Memory Usage    Flush for Prune
+2556340         IF_NEEDED    2899141         394844.34 kB    False


The 'Flush for Prune' has changed here to False, and now all lines have False value here. Can the False line be preserved, or is it OK in this sample output?

This is the actual output of running the script, looks like it wasn't updated in a while

I was wondering how tracing is affected by this change. The trace includes the mode, and it should display it fine, so the trace should be OK.

It would be nice to check how the trace looks for a FORCE_SYNC case as well; I guess that can be triggered by an RPC call to the running node.

I've never used tracing before, my understanding is that @0xB10C uses these for monitoring mostly.

This is the actual output of running the script, looks like it wasn't updated in a while

I think that the scripts only enables the trace and listens to it, but it does not trigger a particular use case.
If trace outputs are displayed with these changes, that should be sufficient check (i.e., tracing a FORCE_SYNC is not crucial).

I plan have a closer look.

I don't use the utxocache tracepoints at the moment.

This is the actual output of running the script, looks like it wasn't updated in a while

Since the script logs UTXO cache events, running it against a different node will produce different output. Not sure if worth changing the example output of the script here. The example output is just as valid just as it was before.

If you retouch, feel free to revert, but I don't think it matters much.

Thank you for checking!
The headers have changed and the column widths, so I had to regenerate the example to be accurate:

-Duration (µs) Mode Coins Count Memory Usage Prune +Duration (µs) Mode Coins Count Memory Usage Flush for Prune

optout21 · 2025-11-03T10:53:41Z

reACK c6ca2b8
Re-reviewed after two rebases.

(prev:
reACK 14abac10f2ebd93367c91dde7586a35456ef9d45
re utACK d4425982ca4aff9b856a26f1d83d1419252f10a2
utACK a59aacefd1a0cdb448c8f847665a5809f9897605)

l0rinc · 2025-11-04T17:49:41Z

rebased after #30595

cedwies · 2025-11-17T09:27:06Z

utACK d442598

0xB10C · 2025-11-19T16:12:49Z

Looked at the flush tracing changes a bit (d4425982ca4aff9b856a26f1d83d1419252f10a2). They seem fine to me. I can't really comment on the other changes, as I'm not too familiar with the utxo cache in general.

from #33680 (comment):

I was wondering how tracing is affected by this change. The trace includes the mode, and it should display it fine, so the trace should be OK.
It would be nice to check how the trace looks for a FORCE_SYNC case as well; I guess that can be triggered by an RPC call to the running node.

Tested this against a signet node. gettxoutsetinfo produces FORCE_SYNC (and the cache remains filled) and shutting down produces FORCE_FLUSH. As I would have expected.

$ python3 contrib/tracing/log_utxocache_flush.py 1234
Hooking into bitcoind with pid 1234
Logging utxocache flushes. Ctrl-C to end...
Duration (µs)   Mode         Coins Count     Memory Usage    Flush for Prune
3911            FORCE_SYNC   3               262.34 kB       False
12439           FORCE_SYNC   3               262.34 kB       False
4652            FORCE_SYNC   3               262.34 kB       False
1331            FORCE_FLUSH  16              262.46 kB       False
939             FORCE_FLUSH  0               262.24 kB       False

l0rinc · 2025-12-02T17:53:52Z

Rebased with trivial conflicts, ready for re-review.

cedwies

reACK 14abac1

l0rinc · 2025-12-16T18:22:55Z

Non-trivial rebase after #30214

Changes:

The new chainstate_write_tests.cpp also needed renaming in the first commit;
ForceFlushStateToDisk(bool wipe_cache = true) is default-true now to minimize the diff.

cedwies · 2025-12-25T09:14:45Z

checked the changes, ACK 11fe88a

This prepares the addition of `FORCE_SYNC`. `empty_cache` in `FlushStateToDisk` was moved up to be reusable and `FlushStateMode::FORCE_FLUSH` was used as a placeholder before we properly split the two new states. `log_utxocache_flush.py` was regenerated and the alignment adjusted for the wider `FlushStateMode` values. Co-authored-by: Pieter Wuille <[email protected]> Co-authored-by: optout <[email protected]>

Since bitcoin#28280, the cost of a non-wiping sync of the UTXO cache is only proportional to the number of dirty entries, rather than proportional to the size of the entire cache. Because of that, there is no reason to perform a wiping flush in case the contents of the cache is still useful. Split the FlushStateMode::ALWAYS mode into a FORCE_SYNC (non-wiping) and a FORCE_FLUSH (wiping), and then use the former in scantxoutset, gettxoutsetinfo, snapshot creation. Co-authored-by: l0rinc <[email protected]> Co-authored-by: cedwies <[email protected]>

l0rinc · 2026-01-03T11:50:50Z

Trivial rebase after #33866, you can check it with git range-diff -w -U0 master 11fe88a c6ca2b8.
Ready for review again!

cedwies · 2026-01-03T15:09:47Z

reACK c6ca2b8 (trivial)

DrahtBot added the Validation label Oct 22, 2025

l0rinc mentioned this pull request Oct 22, 2025

validation: do not wipe utxo cache for stats/scans/snapshots #30610

Closed

l0rinc force-pushed the l0rinc/force-sync branch from 0a769e3 to 0186bc4 Compare October 22, 2025 17:30

This was referenced Oct 22, 2025

Add libbitcoinkernel example files #33669

Closed

[IBD] coins: reduce lookups in dbcache layer propagation #33602

Merged

l0rinc force-pushed the l0rinc/force-sync branch 2 times, most recently from 395b567 to b92ee75 Compare October 22, 2025 21:21

cedwies reviewed Oct 28, 2025

View reviewed changes

l0rinc force-pushed the l0rinc/force-sync branch 2 times, most recently from adfb4eb to a59aace Compare November 1, 2025 21:24

optout21 reviewed Nov 3, 2025

View reviewed changes

DrahtBot requested a review from cedwies November 3, 2025 10:53

DrahtBot added the Needs rebase label Nov 4, 2025

l0rinc force-pushed the l0rinc/force-sync branch from a59aace to d442598 Compare November 4, 2025 17:49

DrahtBot removed the Needs rebase label Nov 4, 2025

DrahtBot mentioned this pull request Nov 14, 2025

refactor: Let CCoinsViewCache::BatchWrite return void #33866

Merged

DrahtBot mentioned this pull request Nov 27, 2025

log: Use more severe log level (warn/err) where appropriate #33960

Merged

DrahtBot added the Needs rebase label Dec 2, 2025

l0rinc force-pushed the l0rinc/force-sync branch from d442598 to 14abac1 Compare December 2, 2025 17:53

cedwies reviewed Dec 2, 2025

View reviewed changes

DrahtBot requested a review from optout21 December 2, 2025 18:23

DrahtBot removed the Needs rebase label Dec 2, 2025

DrahtBot added the Needs rebase label Dec 16, 2025

kevkevinpal mentioned this pull request Dec 16, 2025

refactor: Improve assumeutxo state representation kevkevinpal/bitcoin#231

Closed

l0rinc force-pushed the l0rinc/force-sync branch 2 times, most recently from bd0f8c0 to 6fc49a1 Compare December 16, 2025 18:14

DrahtBot added the CI failed label Dec 16, 2025

l0rinc force-pushed the l0rinc/force-sync branch from 6fc49a1 to 11fe88a Compare December 16, 2025 18:22

DrahtBot removed Needs rebase CI failed labels Dec 16, 2025

DrahtBot mentioned this pull request Dec 20, 2025

validation: reuse should_empty check for chainstate flush #34125

Open

DrahtBot added the Needs rebase label Jan 3, 2026

l0rinc and others added 2 commits January 3, 2026 12:43

l0rinc force-pushed the l0rinc/force-sync branch from 11fe88a to c6ca2b8 Compare January 3, 2026 11:44

DrahtBot removed the Needs rebase label Jan 3, 2026

validation: do not wipe utxo cache for stats/scans/snapshots #33680

Are you sure you want to change the base?

validation: do not wipe utxo cache for stats/scans/snapshots #33680

Uh oh!

Conversation

l0rinc commented Oct 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DrahtBot commented Oct 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Coverage & Benchmarks

Reviews

Conflicts

Uh oh!

cedwies left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

optout21 commented Nov 1, 2025

Uh oh!

l0rinc commented Nov 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

0xB10C Nov 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

optout21 commented Nov 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

l0rinc commented Nov 4, 2025

Uh oh!

cedwies commented Nov 17, 2025

Uh oh!

0xB10C commented Nov 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

l0rinc commented Dec 2, 2025

Uh oh!

cedwies left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

l0rinc commented Dec 16, 2025

Uh oh!

cedwies commented Dec 25, 2025

Uh oh!

l0rinc commented Jan 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cedwies commented Jan 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

l0rinc commented Oct 22, 2025 •

edited

Loading

DrahtBot commented Oct 22, 2025 •

edited

Loading

l0rinc commented Nov 1, 2025 •

edited

Loading

0xB10C Nov 19, 2025 •

edited

Loading

optout21 commented Nov 3, 2025 •

edited

Loading

0xB10C commented Nov 19, 2025 •

edited

Loading

cedwies left a comment •

edited

Loading

l0rinc commented Jan 3, 2026 •

edited

Loading