Bump `-dbcache` default to 300MiB #8273

laanwj · 2016-06-27T14:00:26Z

Bump -dbcache default value to 300MiB.

Also cap the allocations for the leveldb-specific caches to 32MiB, the current default for 100MiB. This avoids that the extra cache memory goes to the much less effective leveldb cache instead of our application-level cache.

Before:

`-dbcache`	`-txindex`	block index db (MiB)	chain state db (MiB)	in-memory UTXO set (MiB)
100 default	0	2.0	32.5	65.5
100	1	12.5	29.9	57.6
300	0	2.0	82.5	215.5
300	1	37.5	73.6	188.9

After:

`-dbcache`	`-txindex`	block index db (MiB)	chain state db (MiB)	in-memory UTXO set (MiB)
100	0	2.0	8.0	90.0
100	1	12.5	29.9	57.6
300 default	0	2.0	8.0	290.0
300	1	8.0	8.0	284.0

Number in bold is the size allocated to the CCoinsCache in the default case.

See #8245, as well as 2016-06-23 meeting log for discussion.

Note: after this, we also need something like #8138 - currently the dbcache determines how deep undo data is rolled back at start, so otherwise this will result in the start-up verification becoming even slower.

laanwj · 2016-06-27T14:00:59Z

Specific numbers are of course open for discussion.

maflcko · 2016-06-27T18:39:43Z

Would this require mention in the release notes?

Imagine someone updating on weak hardware, which does not support the new memory requirement.

jonasschnelli · 2016-06-27T19:21:19Z

This is a good idea and we should def. do this!

Long term I would wish a flexible solution. If you have to catch up a couple of days/week (or during IBD), bitcoind/qt should use a catchup-cache-setting (can be a different, second -dbcacheibd [or similar] settings).
Or we could think about adding "profiles" (set-changes of some CPU/MEM related settings).

jonasschnelli · 2016-06-27T19:22:29Z

utACK fdf085f
ACK for 0.13.
Would need prominent releasenotes part.

dcousens · 2016-06-28T04:37:44Z

src/txdb.h

nit: without knowing the shorthand by heart, it isn't exactly clear what << 20 represents here. It could have some implied context compared to the fact its just shorthand for MiB (assuming I've understood it correctly!)

yes, I don't want to mention numbers in the comments (as everyone forgets to update those as they're not checked by the compiler) but specifying that it's MiB is fine with me

dcousens · 2016-06-28T04:38:13Z

concept ACK

laanwj · 2016-06-28T07:51:38Z

Would this require mention in the release notes?

Yes.

Imagine someone updating on weak hardware, which does not support the new memory requirement.

Well at least now that the mempool is capped, that ought to free up some memory for other uses. But yes it could be the final straw, sure...

Long term I would wish a flexible solution. If you have to catch up a couple of days/week (or during IBD), bitcoind/qt should use a catchup-cache-setting (can be a different, second -dbcacheibd [or similar] settings).

Right - during initial sync (and catch-up) the mempool is virtually guaranteed to be empty. This fact could be used to temporarily bump dbcache. But it's too late for that for 0.13.

maflcko · 2016-06-28T08:39:27Z

Concept ACK for .13

maflcko · 2016-06-28T08:41:52Z

src/txdb.h

Nit: braces should be "(in MiB)" or remove "in"

gmaxwell · 2016-06-28T11:35:09Z

I am (slowly) trying various leveldb cache sizes to see how big it really needs to be.

gmaxwell · 2016-06-28T19:09:17Z

On a Xeon-D 1541 with 64GB ram, 7200 rpm disk, with no txindex, and all signature checks disabled, I found that:

Capping GetOptions ncachesize to 1MB (holding all else constant) resulted in a 1% performance loss and capping at 4MB to a 0.1% loss (31072.23 to reindex instead of 31021.76). (2MB was very close to the 1MB numbers). On this basis I recommend clamping at leveldb 4MB or 8MB (just in case future growth or other workloads makes it matter more), and handing the rest of the memory to the coins cache.

laanwj · 2016-06-29T07:23:48Z

Capping GetOptions ncachesize to 1MB (holding all else constant) resulted in a 1% performance loss and capping at 4MB to a 0.1% loss (31072.23 to reindex instead of 31021.76). (2MB was very close to the 1MB numbers). On this basis I recommend clamping at leveldb 4MB or 8MB (just in case future growth or other workloads makes it matter more), and handing the rest of the memory to the coins cache.

Thanks for testing! I'll change the maximums to 8 MiB.

gmaxwell · 2016-06-29T17:34:53Z

Benchmarked this pull req on the above system with reindex (and no signature checking at all), went from 31021.76 seconds to 21303.22.

It's still a lot slower than the 'infinite' dbcache size results; but a huge improvement over the old results.

ACK.

roques · 2016-06-29T18:22:48Z

@gmaxwell I recently enabled txindex and reindexed. Reindexing was IO-bound by LevelDB compacting. When setting -dbcache=16384 I observed that LevelDB's Level-0 files became much larger (~350MB) than with the default settings. This resulted in top LevelDB compactions of 4@0 + 0@1 files => 1458007320 bytes, resulting in several hundred new level-1 files of which the great majority can simply be moved into higher levels instead of requiring to be compacted into higher levels. This saved me quite a few compactions and thus sped-up the whole reindex.

Unfortunately, I'm unfamiliar with the exact memory management of LevelDB and what memory is used to build these large Level-0 files. However, if this patch clamps that memory of LevelDB to 8MB it will not be able to create such large Level-0 files and thus will have to do quite a few more lower-level compactions. — @maxwell, it would be nice if you could time an 'infinite dbcache' reindex with txindex=1 with this patch and see if the patch hurts its performance or if my guess is wrong.

gmaxwell · 2016-06-29T18:26:55Z

@roques I started that test before commenting above. Will update when it's done.

gmaxwell · 2016-06-30T00:52:00Z

With dbcache set to 8192, reindex with this patch and no signature checks took 8071.65 seconds, and without 8071.65 seconds. Test with txindex isn't done yet (it was later in my testqueue).

laanwj · 2016-06-30T08:03:18Z

Could set static const int64_t nMaxBlockDBAndTxIndexCache = 8; back to 32, that would be safer. It's possible that it makes more of a difference there then for the UTXO cache, as the access pattern is different, and I wasn't really planning to make any changes for -txindex here anyway.

gmaxwell · 2016-06-30T08:57:21Z

With txindex, no signature checks, and 8GB dbcache, before this PR 10131.43 and after 11682.22. So it makes a meaningful difference. I'm starting a test with the txindex case at 32 and seeing how that goes.

laanwj · 2016-06-30T09:03:42Z

Added mention in release notes.

petertodd · 2016-06-30T19:40:13Z

Concept ACK

gmaxwell · 2016-06-30T19:54:21Z

10721.93 with txindex, infinite dbcache, and the 32MB clamp.

laanwj · 2016-07-01T07:50:20Z

Ok, thanks for measuring, so that still suffers.
Removing the ceiling for txindex cache completely (well, bump it up to some unlikely number,1 GiB) in this PR. The exact performance compromise for txindex can be figured out later, this PR just aims to improve the common case.

jonasschnelli · 2016-07-01T18:27:09Z

Positing some slightly off topic results here:

Settings:

Current master da50997
--disable-debug
-dbcache=8000
SSD

Results:

~3h04min IBD from random peers debug.log
~2h45min -reindex debug.log
~2h10min -reindex-chainstate debug.log

jonasschnelli · 2016-07-02T19:10:52Z

More data:

~5h22min default DB cache -reindex with master (da50997) debug.log
~4h06min default DB cache -reindex this PR (f2bdbb7) debug.log

jonasschnelli · 2016-07-04T13:35:00Z

Tested ACK 1009626d308ff70e3c0457559b87aacd510d0c42.

Also cap the allocation for the leveldb-specific cache for the UTXO set to 8MiB. This avoids that the extra cache memory goes to the much less effective leveldb cache instead of our application-level cache.

laanwj · 2016-07-06T05:46:25Z

Squashed and updated the commit messages for the new values.

efd1d83 doc: Mention dbcache increase in release notes (Wladimir J. van der Laan) 32cab91 Bump `-dbcache` default to 300MiB (Wladimir J. van der Laan)

- reduces wallet memory usage - inline with Dash parameters see bitcoin/bitcoin#8273

- reduces wallet memory usage - inline with Dash parameters - reduce DEFAULT_MAX_MEMPOOL_SIZE see bitcoin/bitcoin#8273

73d7288 [Doc] Mention reindex-chainstate in the release notes (random-zebra) c45c0ec Skip MoneySupply update during reindex/reindex-chainstate (random-zebra) efe193f Scan for better chains in ThreadImport (random-zebra) 0aceea3 Make ProcessNewBlock dbp const and update comment (Pieter Wuille) 8255e82 [Cleanup] Remove fVerifyingBlocks global flag (random-zebra) 30c7aed Add -reindex-chainstate that does not rebuild block index (random-zebra) 79d5cb2 Verify DB with original default check-level 3 (random-zebra) ed96859 Reduce default number of blocks to check at startup (Pieter Wuille) d4516fa Log/report in 10% steps during VerifyDB (Jonas Schnelli) cce2c9e doc: Mention dbcache increase in release notes (random-zebra) 11dc022 [GUI] Update default db cache size in the options model (random-zebra) 9b7f764 Bump `-dbcache` default to 300MiB (Wladimir J. van der Laan) Pull request description: This was attempted already in #1864 and #1907. Third time's a charm. Now the speed is acceptable and (unlike what was happening in #1907) the process can be interrupted. Further, as it is performed in ThreadImport, the GUI is open and functional during the chainstate reindex, instead of being seemingly stuck at the splash screen. This PR also includes a few improvements coming from bitcoin#10919, bitcoin#8273, bitcoin#8136 and bitcoin#8611. Here's a quick comparison of the data I had testing here (empty wallet with GUI, empty pivx.conf / default values). The reindexes have been performed with 0 network connections. <table> <tr> <th>Mainnet (height=2847011)</th><th>Master</th><th>this PR</th> </tr><tr> <td>online sync</td><td>8 hr 13 min</td><td>6 hr 42 min</td> </tr><tr> <td>-reindex (offline)</td><td>7 hr 14 min</td><td>6 hr 21 min</td> </tr><tr> <td>-reindex-chainstate (offline)</td><td>N/A</td><td>6 hr 13 min</td> </tr><tr> <td>startup (synced node)</td><td>100 sec</td><td>83 sec</td> </tr><tr> <td>RAM (synced node)</td><td>1.65 GiB</td><td>1.5 GiB</td> </tr><tr> <td>utxo_cache (synced node)</td><td>28.9MiB</td><td>39.5MiB</td> </tr> </table> ACKs for top commit: furszy: re-ACK 73d7288, tested again. Tree-SHA512: fb828d89692ccb6105eae5411b70a3c7a129d98d3101121cf604171de1b818a8c68b3771cc69fcf0a04642fe1327ee7ce360a72ecd7ef019b1614bd83c081de5

laanwj added the UTXO Db and Indexes label Jun 27, 2016

laanwj added this to the 0.13.0 milestone Jun 27, 2016

dcousens reviewed Jun 28, 2016
View reviewed changes

maflcko reviewed Jun 28, 2016
View reviewed changes

src/txdb.h Outdated

Copy link

Member

maflcko Jun 28, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: braces should be "(in MiB)" or remove "in"

laanwj force-pushed the 2016_06_dbcache branch from fdf085f to f2bdbb7 Compare June 28, 2016 13:18

laanwj added 2 commits July 6, 2016 07:44

Bump -dbcache default to 300MiB

32cab91

Also cap the allocation for the leveldb-specific cache for the UTXO set to 8MiB. This avoids that the extra cache memory goes to the much less effective leveldb cache instead of our application-level cache.

doc: Mention dbcache increase in release notes

efd1d83

laanwj force-pushed the 2016_06_dbcache branch from 1009626 to efd1d83 Compare July 6, 2016 05:45

laanwj merged commit efd1d83 into bitcoin:master Jul 6, 2016

laanwj added a commit that referenced this pull request Jul 6, 2016

Merge #8273: Bump -dbcache default to 300MiB

396f9d6

efd1d83 doc: Mention dbcache increase in release notes (Wladimir J. van der Laan) 32cab91 Bump `-dbcache` default to 300MiB (Wladimir J. van der Laan)

laanwj mentioned this pull request Jul 11, 2016

Reindex seems really slow (at least with default dbcache) #8245

Closed

jonasschnelli mentioned this pull request Jul 26, 2016

[Qt] Add dbcache migration path #8407

Merged

dagurval mentioned this pull request Nov 10, 2016

Bump default dbcache bitcoinxt/bitcoinxt#168

Merged

AmirAbrams added a commit to duality-solutions/Dynamic that referenced this pull request Aug 3, 2018

Update default txdb cache parameters

d21c9f1

- reduces wallet memory usage - inline with Dash parameters see bitcoin/bitcoin#8273

AmirAbrams added a commit to duality-solutions/Dynamic that referenced this pull request Aug 4, 2018

Update default txdb cache parameters

38c087b

- reduces wallet memory usage - inline with Dash parameters - reduce DEFAULT_MAX_MEMPOOL_SIZE see bitcoin/bitcoin#8273

random-zebra mentioned this pull request May 16, 2021

[Core] Add reindex-chainstate startup option PIVX-Project/PIVX#2391

Merged

bitcoin locked as resolved and limited conversation to collaborators Sep 8, 2021

Bump -dbcache default to 300MiB #8273

Bump -dbcache default to 300MiB #8273

Uh oh!

Conversation

laanwj commented Jun 27, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

laanwj commented Jun 27, 2016

Uh oh!

maflcko commented Jun 27, 2016

Uh oh!

jonasschnelli commented Jun 27, 2016

Uh oh!

jonasschnelli commented Jun 27, 2016

Uh oh!

dcousens Jun 28, 2016

Choose a reason for hiding this comment

Uh oh!

laanwj Jun 28, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dcousens commented Jun 28, 2016

Uh oh!

laanwj commented Jun 28, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

maflcko commented Jun 28, 2016

Uh oh!

maflcko Jun 28, 2016

Choose a reason for hiding this comment

Uh oh!

gmaxwell commented Jun 28, 2016

Uh oh!

gmaxwell commented Jun 28, 2016

Uh oh!

laanwj commented Jun 29, 2016

Uh oh!

gmaxwell commented Jun 29, 2016

Uh oh!

roques commented Jun 29, 2016

Uh oh!

gmaxwell commented Jun 29, 2016

Uh oh!

gmaxwell commented Jun 30, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

laanwj commented Jun 30, 2016

Uh oh!

gmaxwell commented Jun 30, 2016

Uh oh!

laanwj commented Jun 30, 2016

Uh oh!

petertodd commented Jun 30, 2016

Uh oh!

gmaxwell commented Jun 30, 2016

Uh oh!

laanwj commented Jul 1, 2016

Uh oh!

jonasschnelli commented Jul 1, 2016

Uh oh!

jonasschnelli commented Jul 2, 2016

Uh oh!

jonasschnelli commented Jul 4, 2016

Uh oh!

laanwj commented Jul 6, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Bump `-dbcache` default to 300MiB #8273

Bump `-dbcache` default to 300MiB #8273

laanwj commented Jun 27, 2016 •

edited

Loading

laanwj Jun 28, 2016 •

edited

Loading

laanwj commented Jun 28, 2016 •

edited

Loading

gmaxwell commented Jun 30, 2016 •

edited

Loading