Skip to content

Commit c360054

Browse files
martin.foo838claude
andcommitted
[improvement](be) State which SNII .prx windows the zstd budget is a share of
The budget is a share of the smallest frames of the windows that get a zstd frame, the only ones whose codec a share decides; windows written without one (a singleton, or too short to compress) are not counted. Comments only. Co-Authored-By: Claude Opus 5.5 <[email protected]>
1 parent 170822d commit c360054

2 files changed

Lines changed: 12 additions & 8 deletions

File tree

‎be/src/common/config.cpp‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1430,9 +1430,9 @@ DEFINE_mInt32(snii_prx_zstd_level, "3");
14301430
// least this percentage; 0 picks the smaller frame. Every read of a zstd window pays a
14311431
// decompression that PFOR does not, but how much zstd saves depends on the data, so no one
14321432
// percentage fits all data. The default (negative) lets each index choose: it stores in their
1433-
// faster frame the windows zstd shrinks least, spending at most 8% of their smallest size (see
1434-
// format::PrxZstdBudget). Readers handle every codec, so changing this never affects existing
1435-
// segments.
1433+
// faster frame the windows zstd shrinks least, spending at most 8% of the smallest size of the
1434+
// windows that get a zstd frame (see format::PrxZstdBudget). Readers handle every codec, so
1435+
// changing this never affects existing segments.
14361436
DEFINE_mInt32(snii_prx_zstd_min_saving_percent, "-1");
14371437
// Zstd level for direct-load .prx regions. Compaction rewrites them using snii_prx_zstd_level.
14381438
DEFINE_mInt32(snii_prx_zstd_level_direct_load, "3");

‎be/src/storage/index/inverted/snii/format/prx_pod.h‎

Lines changed: 9 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -80,11 +80,15 @@ AutoPrxCodecChoice select_auto_prx_codec(size_t pfor_payload_size, size_t plain_
8080
// zstd saves depends on the data: a fixed share (zstd_min_saving_percent) of 30 spends about 8% more
8181
// bytes on the windows of agent logs, about 5% on those of service logs, and much more on data whose
8282
// windows zstd shrinks by just under the share. The budget instead spends at most
83-
// kPrxZstdBudgetPercent of the smallest frames, on the windows zstd shrinks least: a window is
84-
// traded when its saving is below the share whose trades, over the windows seen so far, fit the
85-
// budget, and when what has actually been spent still fits it, so no order of windows can make it
86-
// spend more. Until kPrxZstdBudgetWarmupWindows windows are seen the share is
87-
// kPrxZstdBudgetPriorPercent. Writer-only and deterministic for a given input.
83+
// kPrxZstdBudgetPercent of the smallest frames of the windows that get a zstd frame (a plain
84+
// payload of at least kPrxAutoZstdMinBytes), the only ones whose codec a share decides, on the
85+
// windows zstd shrinks least: a window is traded when its saving is below the share whose trades,
86+
// over the windows seen so far, fit the budget, and when what has actually been spent still fits
87+
// it, so no order of windows can make it spend more. Until kPrxZstdBudgetWarmupWindows windows are
88+
// seen the share is kPrxZstdBudgetPriorPercent. Windows written without a zstd frame (a singleton,
89+
// or too short to compress) are not counted: no share changes them, and counting their bytes would
90+
// let an index of many short posting lists grow its large windows by more than the budget.
91+
// Writer-only and deterministic for a given input.
8892
inline constexpr uint32_t kPrxZstdBudgetPercent = 8;
8993
inline constexpr uint32_t kPrxZstdBudgetPriorPercent = 30;
9094
inline constexpr uint32_t kPrxZstdBudgetWarmupWindows = 64;

0 commit comments

Comments
 (0)