Compaction uses more CPU in 10.6 when compression is enabled #13996
Description
The CPU overhead for compaction increases by ~1.5X for fillseq and ~1.2X for overwrite in 10.6.0 compared to 10.5.5. Given that compaction runs in the background it doesn't always hurt throughput but for some of my benchmarks it also reduces throughput from either competition for the CPU or more write stalls.
Results from a small server with an IO-bound db_bench workload and 1 thread are here. The CompMergeCPU column shows that CPU time is larger in 10.6.0 vs 10.5.5 -- 735.55s vs 479.29s for fillseq and 4089.44s vs 3387.73s for overwrite.
I can also produce this with simpler command lines, but these are what my helper scripts create when I do a proper run of db_bench.
The fillseq command line is:
./db_bench --benchmarks=fillseq,stats --allow_concurrent_memtable_write=false --level0_file_num_compaction_trigger=4 --level0_slowdown_writes_trigger=20 --level0_stop_writes_trigger=30 --max_background_jobs=4 --max_write_buffer_number=8 --undefok=use_blob_cache,blob_file_starting_level,use_shared_block_and_blob_cache,blob_cache_size,blob_cache_numshardbits,prepopulate_blob_cache,multiread_batched,cache_low_pri_pool_ratio,cache_type,prepopulate_block_cache --db=/data/m/rx --wal_dir=/data/m/rx --num=400000000 --key_size=20 --value_size=400 --block_size=8192 --cache_size=21474836480 --cache_numshardbits=6 --cache_type=lru_cache --compression_max_dict_bytes=0 --compression_ratio=0.5 --compression_type=lz4 --bytes_per_sync=0 --cache_index_and_filter_blocks=1 --cache_high_pri_pool_ratio=0.5 --cache_low_pri_pool_ratio=0 --partition_index_and_filters=0 --pin_top_level_index_and_filter=0 --metadata_block_size=16384 --use_direct_reads --use_direct_io_for_flush_and_compaction --prepopulate_block_cache=1 --benchmark_write_rate_limit=0 --write_buffer_size=16777216 --target_file_size_base=16777216 --max_bytes_for_level_base=67108864 --verify_checksum=1 --delete_obsolete_files_period_micros=62914560 --max_bytes_for_level_multiplier=8 --statistics=0 --stats_per_interval=1 --stats_interval_seconds=20 --report_interval_seconds=1 --histogram=1 --memtablerep=skip_list --bloom_bits=10 --open_files=-1 --subcompactions=1 --compaction_style=0 --num_levels=8 --min_level_to_compress=3 --level_compaction_dynamic_level_bytes=true --pin_l0_filter_and_index_blocks_in_cache=1 --min_level_to_compress=0 --use_existing_db=0 --sync=0 --threads=1 --memtablerep=vector --allow_concurrent_memtable_write=false --disable_wal=1 --seed=1758413825
The overwrite command line is:
./db_bench --benchmarks=overwrite,flush,levelstats,waitforcompaction,stats --use_existing_db=1 --sync=0 --level0_file_num_compaction_trigger=4 --level0_slowdown_writes_trigger=20 --level0_stop_writes_trigger=30 --max_background_jobs=4 --max_write_buffer_number=8 --undefok=use_blob_cache,blob_file_starting_level,use_shared_block_and_blob_cache,blob_cache_size,blob_cache_numshardbits,prepopulate_blob_cache,multiread_batched,cache_low_pri_pool_ratio,cache_type,prepopulate_block_cache --db=/data/m/rx --wal_dir=/data/m/rx --num=400000000 --key_size=20 --value_size=400 --block_size=8192 --cache_size=21474836480 --cache_numshardbits=6 --cache_type=lru_cache --compression_max_dict_bytes=0 --compression_ratio=0.5 --compression_type=lz4 --bytes_per_sync=0 --cache_index_and_filter_blocks=1 --cache_high_pri_pool_ratio=0.5 --cache_low_pri_pool_ratio=0 --partition_index_and_filters=0 --pin_top_level_index_and_filter=0 --metadata_block_size=16384 --use_direct_reads --use_direct_io_for_flush_and_compaction --prepopulate_block_cache=1 --benchmark_write_rate_limit=0 --write_buffer_size=16777216 --target_file_size_base=16777216 --max_bytes_for_level_base=67108864 --verify_checksum=1 --delete_obsolete_files_period_micros=62914560 --max_bytes_for_level_multiplier=8 --statistics=0 --stats_per_interval=1 --stats_interval_seconds=20 --report_interval_seconds=1 --histogram=1 --memtablerep=skip_list --bloom_bits=10 --open_files=-1 --subcompactions=1 --compaction_style=0 --num_levels=8 --min_level_to_compress=3 --level_compaction_dynamic_level_bytes=true --pin_l0_filter_and_index_blocks_in_cache=1 --duration=1800 --threads=1 --merge_operator="put" --seed=1758420262
--- fillseq
- 10.5.5
Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) WPreComp(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB)
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
L0 2/0 60.32 MB 0.5 0.0 0.0 0.0 87.8 159.7 87.8 0.0 1.0 0.0 172.6 521.00 479.29 2983 0.175 399M 0 0.0 0.0
L3 2/0 60.29 MB 0.9 0.0 0.0 0.0 0.0 0.0 0.0 51.2 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0
L4 5/1 155.07 MB 0.8 0.0 0.0 0.0 0.0 0.0 0.0 83.2 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0
L5 40/0 1.18 GB 1.0 0.0 0.0 0.0 0.0 0.0 0.0 86.9 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0
L6 325/0 9.57 GB 1.0 0.0 0.0 0.0 0.0 0.0 0.0 86.3 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0
L7 2609/0 76.79 GB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 76.8 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0
Sum 2983/1 87.80 GB 0.0 0.0 0.0 0.0 87.8 159.7 87.8 384.4 1.0 0.0 172.6 521.00 479.29 2983 0.175 399M 0 0.0 0.0
- 10.6.2
Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) WPreComp(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB)
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
L0 2/0 60.29 MB 0.5 0.0 0.0 0.0 87.8 159.7 87.8 0.0 1.0 0.0 115.9 775.77 735.55 2983 0.260 399M 0 0.0 0.0
L3 2/0 60.29 MB 0.9 0.0 0.0 0.0 0.0 0.0 0.0 51.2 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0
L4 6/1 180.90 MB 7.3 0.0 0.0 0.0 0.0 0.0 0.0 83.2 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0
L5 40/0 1.18 GB 1.0 0.0 0.0 0.0 0.0 0.0 0.0 87.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0
L6 325/0 9.57 GB 1.0 0.0 0.0 0.0 0.0 0.0 0.0 86.3 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0
L7 2608/0 76.77 GB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 76.8 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0
Sum 2983/1 87.81 GB 0.0 0.0 0.0 0.0 87.8 159.7 87.8 384.4 1.0 0.0 115.9 775.77 735.55 2983 0.260 399M 0 0.0 0.0
--- overwrite
- 10.5.5
Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) WPreComp(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB)
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
L0 1/0 4.89 MB 0.2 10.4 0.0 10.4 98.9 98.9 88.6 0.0 1.1 30.3 288.6 351.13 167.37 6158 0.057 247M 5095 0.0 0.0
L3 3/0 50.54 MB 0.8 246.6 88.6 158.0 246.4 246.4 88.4 0.0 2.8 323.2 322.9 781.32 419.56 893 0.875 616M 332K 0.0 0.0
L4 14/0 158.70 MB 0.9 176.4 83.6 92.9 176.1 176.1 83.2 4.8 2.1 267.4 266.9 675.74 313.18 5432 0.124 441M 693K 0.0 0.0
L5 103/0 1.35 GB 1.0 268.7 88.1 180.7 228.0 411.3 47.3 0.0 2.6 181.4 153.9 1517.35 1070.09 6630 0.229 1035M 6578K 0.0 0.0
L6 728/0 10.81 GB 1.0 190.1 47.1 143.0 181.3 327.0 38.4 0.2 3.8 160.1 152.7 1215.98 858.71 3804 0.320 858M 39M 0.0 0.0
L7 6389/0 86.58 GB 0.0 149.3 38.7 110.5 109.4 201.9 -1.1 0.0 2.8 194.5 142.5 786.00 558.82 2673 0.294 680M 174M 0.0 0.0
Sum 7238/0 98.95 GB 0.0 1041.5 346.1 695.5 1040.2 1461.7 344.7 5.1 11.7 200.2 199.9 5327.52 3387.73 25590 0.208 3878M 221M 0.0 0.0
- 10.6.2
Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) WPreComp(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB)
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
L0 0/0 0.00 KB 0.0 8.4 0.0 8.4 81.4 81.4 73.0 0.0 1.1 31.1 301.0 276.95 139.27 5065 0.055 203M 4310 0.0 0.0
L3 3/0 54.04 MB 0.8 202.5 73.0 129.5 202.4 202.4 72.8 0.0 2.8 346.9 346.6 597.89 339.03 723 0.827 506M 286K 0.0 0.0
L4 13/0 167.17 MB 1.0 145.4 68.8 76.6 145.2 145.2 68.5 4.0 2.1 276.2 275.7 539.12 257.21 4461 0.121 363M 599K 0.0 0.0
L5 103/0 1.35 GB 1.0 224.7 72.6 152.1 191.1 344.6 38.9 0.0 2.6 128.9 109.6 1784.36 1476.26 5390 0.331 868M 5705K 0.0 0.0
L6 727/0 10.81 GB 1.0 152.1 38.7 113.4 145.0 261.6 31.6 0.2 3.7 112.0 106.8 1390.52 1132.25 3098 0.449 686M 31M 0.0 0.0
L7 6350/0 86.64 GB 0.0 125.4 32.0 93.3 92.2 170.2 -1.1 0.0 2.9 140.7 103.6 912.16 745.43 2203 0.414 570M 144M 0.0 0.0
Sum 7196/0 99.01 GB 0.0 858.5 285.1 573.4 857.3 1205.3 283.8 4.3 11.7 159.8 159.6 5501.00 4089.44 20940 0.263 3198M 183M 0.0 0.0
In some benchmarks (fillseq, overwrite) this hurts performance because there are more write stalls.
ops_sec mb_sec lsm_sz blob_sz c_wgb w_amp c_mbps c_wsecs c_csecs b_rgb b_wgb usec_op p50 p99 p99.9 p99.99 pmax uptime stall% Nstall u_cpu s_cpu rss test date version job_id githash
735799 294.7 88GB 0GB 87.8 1.1 165.4 173 521 0 0 1.4 0.5 1 6 1235 9855 544 50.0 0.7 0.0 NA fillseq.wal_disabled.v400 2025-09-21T00:17:05 10.5.5 f4bf18a60b
501181 200.7 88GB 0GB 87.8 1.1 112.7 116 776 0 0 2.0 0.5 1 947 1266 9622 798 64.8 1.0 0.0 NA fillseq.wal_disabled.v400 2025-09-21T02:56:22 10.6.2 fae8a27fbd
ops_sec mb_sec lsm_sz blob_sz c_wgb w_amp c_mbps c_wsecs c_csecs b_rgb b_wgb usec_op p50 p99 p99.9 p99.99 pmax uptime stall% Nstall u_cpu s_cpu rss test date version job_id githash
80110 306.4 88GB 0GB 0.1 NA 0.2 170 0 0 0 12.5 12.5 16 21 22 7273 300 0.0 0.3 0.0 NA revrange.t1 2025-09-21T00:26:13 10.5.5 f4bf18a60b
78465 300.1 88GB 0GB 0.1 NA 0.2 112 1 0 0 12.7 12.5 16 21 22 5040 300 0.0 0.3 0.0 NA revrange.t1 2025-09-21T03:09:44 10.6.2 fae8a27fbd
ops_sec mb_sec lsm_sz blob_sz c_wgb w_amp c_mbps c_wsecs c_csecs b_rgb b_wgb usec_op p50 p99 p99.9 p99.99 pmax uptime stall% Nstall u_cpu s_cpu rss test date version job_id githash
261468 104.7 97GB 0GB 117.7 8.3 786.4 283 426 1128 0 3.8 3.0 9 12 72 4509 153 0.0 0.4 0.1 NA overwritesome.t1.s0 2025-09-21T00:31:14 10.5.5 f4bf18a60b
243125 97.4 97GB 0GB 117.9 8.3 732.9 253 477 1150 0 4.1 3.1 9 13 1126 4757 165 6.2 0.4 0.1 NA overwritesome.t1.s0 2025-09-21T03:14:45 10.6.2 fae8a27fbd
ops_sec mb_sec lsm_sz blob_sz c_wgb w_amp c_mbps c_wsecs c_csecs b_rgb b_wgb usec_op p50 p99 p99.9 p99.99 pmax uptime stall% Nstall u_cpu s_cpu rss test date version job_id githash
3660 14.7 98GB 0GB 36.7 11.2 20.9 262 144 874 0 273.2 275.1 564 845 2960 16913 1800 0.0 0.6 0.2 NA revrangewhilewriting.t1 2025-09-21T00:34:12 10.5.5 f4bf18a60b
3707 14.9 98GB 0GB 36.8 11.2 20.9 181 208 817 0 269.7 271.6 563 837 2889 15518 1800 0.0 0.6 0.2 NA revrangewhilewriting.t1 2025-09-21T03:18:08 10.6.2 fae8a27fbd
ops_sec mb_sec lsm_sz blob_sz c_wgb w_amp c_mbps c_wsecs c_csecs b_rgb b_wgb usec_op p50 p99 p99.9 p99.99 pmax uptime stall% Nstall u_cpu s_cpu rss test date version job_id githash
3878 15.5 100GB 0GB 39.1 11.9 22.3 256 157 1248 0 257.9 262.6 554 842 2878 13518 1800 0.0 0.5 0.2 NA fwdrangewhilewriting.t1 2025-09-21T01:04:16 10.5.5 f4bf18a60b
3940 15.8 100GB 0GB 39.2 11.9 22.3 178 225 1297 0 253.8 257.6 548 833 2888 16340 1800 0.0 0.6 0.2 NA fwdrangewhilewriting.t1 2025-09-21T03:48:12 10.6.2 fae8a27fbd
ops_sec mb_sec lsm_sz blob_sz c_wgb w_amp c_mbps c_wsecs c_csecs b_rgb b_wgb usec_op p50 p99 p99.9 p99.99 pmax uptime stall% Nstall u_cpu s_cpu rss test date version job_id githash
11158 4.5 100GB 0GB 48.7 14.5 27.7 238 209 6958 0 89.6 89.3 195 282 1843 11608 1800 0.0 0.5 0.2 NA readwhilewriting.t1 2025-09-21T01:34:19 10.5.5 f4bf18a60b
11419 4.6 100GB 0GB 48.7 14.5 27.7 162 308 6959 0 87.6 88.5 184 259 1834 12825 1800 0.0 0.6 0.2 NA readwhilewriting.t1 2025-09-21T04:18:15 10.6.2 fae8a27fbd
ops_sec mb_sec lsm_sz blob_sz c_wgb w_amp c_mbps c_wsecs c_csecs b_rgb b_wgb usec_op p50 p99 p99.9 p99.99 pmax uptime stall% Nstall u_cpu s_cpu rss test date version job_id githash
122888 49.2 99GB 0GB 1040.2 12.6 559.3 200 5328 221 0 8.1 3.1 10 1191 1295 5078 1905 46.7 3.7 0.8 NA overwriteandwait.t1.s0 2025-09-21T02:04:23 10.5.5 f4bf18a60b
101267 40.6 99GB 0GB 857.3 12.6 451.5 160 5501 183 0 9.9 3.2 12 1223 1296 4832 1944 52.9 4.4 0.6 NA overwriteandwait.t1.s0 2025-09-21T04:48:19 10.6.2 fae8a27fbd