Skip to content

Commit 0d020fb

Browse files
airborne12martin.foo838claude
authored
branch-4.2: [improvement](be) Let each SNII index spend a size budget on faster .prx windows #69769 (#69796)
### What problem does this PR solve? Issue Number: None Related PR: #69769, #69649 (backport #69795), #69770 (backport #69788) Problem Summary: Backport of #69769 to branch-4.2, stacked on #69795 (the backport of #69649). #69649 stores a .prx window as PFOR instead of zstd when zstd saves less than `snii_prx_zstd_min_saving_percent` (30) of the PFOR frame, because every read of a zstd window pays a decompression. What that costs and buys depends on how much zstd saves on the data. Replaying the 1.6M windows of 10M AgentLogs rows, 30 spends 8% more bytes on the windows zstd compresses and cuts their decode time by 42%; on the windows of TextBench service logs it spends 5% and cuts 6%; on data whose windows zstd shrinks by just under 30% it stores every window as PFOR. No single percentage fits all data. Each index of a segment now spends a size budget instead: `PrxZstdBudget`, owned by its `LogicalIndexWriter` and reached through `PrxWindowLimits`, so the resident and the spilled encoder, and the ordinary and the compaction build, all use it. A window whose smallest frame is zstd is stored in its faster frame (PFOR, or raw when smaller) when - its saving is below the share whose trades, over the windows the index has seen, fit 8% of their smallest frames, which puts the budget on the windows zstd shrinks least, and - the bytes the index has actually traded, this window included, still fit 8% of the smallest frames seen. This makes 8% a bound for any order of windows: fed the TextBench windows from the largest saving to the smallest, the share alone spends 36%. Until it has seen 64 windows the share is 30, the previous default. The windows counted are those that get a zstd frame (a plain payload of at least 512 bytes), the only ones whose codec a share decides; singleton and short windows are written as before and counted neither in the allowance nor in the warmup. The choice uses the frames the writer builds anyway: nothing is compressed twice or scanned ahead, and the same input gives the same bytes. `snii_prx_zstd_min_saving_percent` now defaults to -1 (the budget); 0-100 keeps a fixed share, 0 the smallest frame. The budget is 8% rather than less: with it the AgentLogs phrase queries below read nearly the same frames as with 30, while in the replay a 5% budget cuts the decode time by 31% instead of 39%. Same binary, one stream load per table, size against the smallest frames (0): | data | index, 30 | index, budget | table, 30 | table, budget | |---|---|---|---|---| | AgentLogs 99M rows | +4.13% | +3.65% | +1.92% | +1.69% | | AgentLogs 10M rows | +4.59% | +3.37% | +2.13% | +1.57% | | TextBench 12M rows | +0.89% | +0.79% | +0.51% | +0.45% | - The phrase queries read the same frames with 30 and with the budget: at 99M rows all PFOR for DS1, DS6, DS7, A6 and A9 (for example 107,307 frames for DS7), and about 11,170 zstd frames of about 259,000 for A10 with both; on TextBench most windows stay zstd either way (582 against 583 zstd frames of 599 for 'connection reset'). - Hot queries in one process, budget data against 30 data with the same read path (paired cycles: 6 for the 25 AgentLogs queries at 10M rows, 10 for the 15 TextBench queries): 0.96-1.06 and 0.97-1.05; every query returns the same result. - Against the data written before #69649 (smallest frames), the 11 AgentLogs phrase queries at 99M rows take 0.83x in geometric mean with the budget data (DS7 0.63x, A6 0.67x, A9 0.70x, DS4 0.74x, A10 0.77x), as with 30 (0.83x). This run also had the read-path change of #69770, which does not touch the PFOR windows these queries read. - Data bytes are identical, and the index written with 0 and with 30 has exactly the size the previous binary writes. - Write cost: the choice adds about 48 ns per window, 79 ms over the 1.6M AgentLogs windows. ### Branch adaptations None. `git cherry-pick 0571eb6` applied cleanly on top of the backport of #69649, which it needs: it changes the setting #69649 adds, and the automatic pick to branch-4.2 failed because branch-4.2 does not have #69649. This PR is stacked on that backport, #69795, so it shows both commits until #69795 merges; merge #69795 first. ### Validation - Built and run with both backports (this one and the stacked #69769 backport) applied to branch-4.2 at 711db43, the commit before #69625: its lance-c update is not in the local third-party build, and the 16 commits after it touch no SNII file. `PrxPod*`, `Snii*`, `DorisSniiFileReader*`, `*InvertedIndex*`, `*IndexFileWriter*` and `*IndexFileReader*`: 1710 tests passed, including the new budget, codec choice, writer and configuration tests. - On master both changes passed CI. - Not built locally on the branch-4.2 tip; this PR's BE UT run covers that. ### Release note SNII now chooses per index how many position windows to store in the faster but larger PFOR form, spending at most 8% of the size of the windows zstd compresses; `snii_prx_zstd_min_saving_percent` defaults to -1 for this, and 0-100 still sets a fixed percentage. ### Check List (For Author) - Test: Unit Test - SniiPrxZstdBudget: the prior share, the windows zstd shrinks least are traded first, the bound under a descending order, windows with nothing to trade. - SniiPrxPodTest: the faster frame (PFOR or raw) and when the budget is asked. - SniiStreamedWriterSessionTest: windows that save 1-29% through the ordinary and the streamed (compaction) build; 30 trades all of them, the budget at most 8% of the smallest frames. - SniiWriterPrxZstdSaving: the default and the config mapping. - Manual test: index size of 10M and 99M AgentLogs and 12M TextBench loads with 0, 30 and the budget; frames read per codec from query profiles; interleaved hot A/B in one process. - Behavior changed: Yes (the default .prx codec choice of newly written segments; existing segments are read as before) - Does this need documentation: No 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: martin.foo838 <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]>
1 parent 91f3bc7 commit 0d020fb

11 files changed

Lines changed: 521 additions & 17 deletions

File tree

‎be/src/common/config.cpp‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1441,6 +1441,14 @@ DEFINE_mInt32(inverted_index_max_buffered_docs, "-1");
14411441
// Zstd levels for SNII DICT blocks and .prx windows. These affect writes; each segment records its compression mode.
14421442
DEFINE_mInt32(snii_dict_block_zstd_level, "3");
14431443
DEFINE_mInt32(snii_prx_zstd_level, "3");
1444+
// A .prx window is stored zstd-compressed only if that frame is smaller than the PFOR one by at
1445+
// least this percentage; 0 picks the smaller frame. Every read of a zstd window pays a
1446+
// decompression that PFOR does not, but how much zstd saves depends on the data, so no one
1447+
// percentage fits all data. The default (negative) lets each index choose: it stores in their
1448+
// faster frame the windows zstd shrinks least, spending at most 8% of the smallest size of the
1449+
// windows that get a zstd frame (see format::PrxZstdBudget). Readers handle every codec, so
1450+
// changing this never affects existing segments.
1451+
DEFINE_mInt32(snii_prx_zstd_min_saving_percent, "-1");
14441452
// Zstd level for direct-load .prx regions. Compaction rewrites them using snii_prx_zstd_level.
14451453
DEFINE_mInt32(snii_prx_zstd_level_direct_load, "3");
14461454
// Handing a few small page-cache reads to the shared prefetch pool costs more than the reads: a

‎be/src/common/config.h‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1438,6 +1438,10 @@ DECLARE_mInt32(inverted_index_max_buffered_docs);
14381438
// import CPU; see the DEFINEs in config.cpp).
14391439
DECLARE_mInt32(snii_dict_block_zstd_level);
14401440
DECLARE_mInt32(snii_prx_zstd_level);
1441+
// A .prx window is stored zstd-compressed only if that frame is smaller than the PFOR one by at
1442+
// least this percentage; 0 picks the smaller frame. Negative (the default) lets each index choose
1443+
// within a size budget. Affects segments written from now on.
1444+
DECLARE_mInt32(snii_prx_zstd_min_saving_percent);
14411445
// Patch C: prx zstd level for DIRECT-LOAD segments only (default 3, cheaper
14421446
// import); compaction rewrites at snii_prx_zstd_level so settled segments are
14431447
// unaffected. Full contract at the DEFINE in config.cpp.

‎be/src/storage/index/index_file_writer.cpp‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -120,6 +120,10 @@ void snii_resolve_index_write_params(bool is_direct_load, bool has_norms,
120120
input->prx_zstd_level = std::clamp(
121121
is_direct_load ? config::snii_prx_zstd_level_direct_load : config::snii_prx_zstd_level,
122122
3, 19);
123+
// Negative: the index chooses the percentage within its own size budget.
124+
input->prx_zstd_budget = config::snii_prx_zstd_min_saving_percent < 0;
125+
input->prx_window_limits.zstd_min_saving_percent =
126+
static_cast<uint32_t>(std::clamp(config::snii_prx_zstd_min_saving_percent, 0, 100));
123127
// G16-d: dict block size experiment knob; <= 0 keeps the format default.
124128
if (config::snii_target_dict_block_bytes > 0) {
125129
input->target_dict_block_bytes =

‎be/src/storage/index/inverted/snii/format/prx_pod.cpp‎

Lines changed: 103 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -60,6 +60,10 @@ Status validate_prx_window_limits(const PrxWindowLimits& limits) {
6060
return Status::Error<ErrorCode::INVALID_ARGUMENT, false>(
6161
"prx: writer window limits exceed reader limits");
6262
}
63+
if (limits.zstd_min_saving_percent > 100) {
64+
return Status::Error<ErrorCode::INVALID_ARGUMENT, false>(
65+
"prx: zstd minimum saving is a percentage");
66+
}
6367
return Status::OK();
6468
}
6569

@@ -451,7 +455,8 @@ size_t zstd_frame_size(size_t plain_size, size_t compressed_size) {
451455
// before PFOR materialization in build_prx_window_auto_from_flat.
452456
AutoPrxCodecChoice select_auto_prx_codec(size_t pfor_payload_size, size_t plain_payload_size,
453457
size_t compressed_payload_size, bool has_compressed,
454-
uint32_t max_uncomp_bytes) {
458+
uint32_t max_uncomp_bytes,
459+
uint32_t zstd_min_saving_percent) {
455460
const bool pfor_readable = pfor_payload_size <= max_uncomp_bytes;
456461
const bool plain_readable = plain_payload_size <= max_uncomp_bytes;
457462
AutoPrxCodecChoice choice;
@@ -462,7 +467,14 @@ AutoPrxCodecChoice select_auto_prx_codec(size_t pfor_payload_size, size_t plain_
462467
}
463468
if (has_compressed && plain_readable) {
464469
const size_t frame_size = zstd_frame_size(plain_payload_size, compressed_payload_size);
465-
if (!choice.readable || frame_size < selected_frame_size) {
470+
// Against a PFOR frame ZSTD must save the configured share; without one it only has to win.
471+
const bool saves_enough =
472+
!choice.readable ||
473+
(frame_size < selected_frame_size &&
474+
static_cast<uint64_t>(frame_size) * 100 <=
475+
static_cast<uint64_t>(selected_frame_size) *
476+
(100 - std::min<uint32_t>(zstd_min_saving_percent, 100)));
477+
if (saves_enough) {
466478
choice = {.codec = PrxCodec::kZstd, .readable = true};
467479
selected_frame_size = frame_size;
468480
}
@@ -476,6 +488,88 @@ AutoPrxCodecChoice select_auto_prx_codec(size_t pfor_payload_size, size_t plain_
476488
return choice;
477489
}
478490

491+
void PrxZstdBudget::add(uint64_t smallest_frame_bytes) {
492+
_smallest_bytes += smallest_frame_bytes;
493+
count_window();
494+
}
495+
496+
bool PrxZstdBudget::trade(uint64_t zstd_frame_bytes, uint64_t faster_extra_bytes,
497+
uint32_t saving_percent) {
498+
_smallest_bytes += zstd_frame_bytes;
499+
const bool faster =
500+
saving_percent < _min_saving_percent &&
501+
(_spent_bytes + faster_extra_bytes) * 100 <= _smallest_bytes * kPrxZstdBudgetPercent;
502+
if (faster) {
503+
_spent_bytes += faster_extra_bytes;
504+
}
505+
_extra_by_saving[std::min<uint32_t>(saving_percent, 100)] += faster_extra_bytes;
506+
count_window();
507+
return faster;
508+
}
509+
510+
void PrxZstdBudget::count_window() {
511+
if (++_windows < kPrxZstdBudgetWarmupWindows) {
512+
return;
513+
}
514+
// The largest share whose trades, over the windows seen so far, fit the budget. A window whose
515+
// saving is p percent is traded under every share above p, so the trades of share s are the
516+
// buckets below s.
517+
const uint64_t allowed = _smallest_bytes * kPrxZstdBudgetPercent / 100;
518+
uint64_t extra = 0;
519+
uint32_t share = 0;
520+
while (share < 100 && extra + _extra_by_saving[share] <= allowed) {
521+
extra += _extra_by_saving[share];
522+
++share;
523+
}
524+
_min_saving_percent = share;
525+
}
526+
527+
namespace {
528+
529+
uint64_t chosen_frame_size(PrxCodec codec, size_t pfor_payload_size, size_t plain_payload_size,
530+
size_t compressed_payload_size) {
531+
switch (codec) {
532+
case PrxCodec::kPfor:
533+
return pfor_frame_size(pfor_payload_size);
534+
case PrxCodec::kZstd:
535+
return zstd_frame_size(plain_payload_size, compressed_payload_size);
536+
default:
537+
return raw_frame_size(plain_payload_size);
538+
}
539+
}
540+
541+
} // namespace
542+
543+
AutoPrxCodecChoice choose_auto_prx_codec(const PrxWindowLimits& limits, size_t pfor_payload_size,
544+
size_t plain_payload_size, size_t compressed_payload_size,
545+
bool has_compressed) {
546+
const auto select = [&](uint32_t zstd_min_saving_percent) {
547+
return select_auto_prx_codec(pfor_payload_size, plain_payload_size, compressed_payload_size,
548+
has_compressed, limits.max_uncomp_bytes,
549+
zstd_min_saving_percent);
550+
};
551+
PrxZstdBudget* const budget = limits.zstd_budget;
552+
if (budget == nullptr) {
553+
return select(limits.zstd_min_saving_percent);
554+
}
555+
const auto bytes = [&](AutoPrxCodecChoice choice) {
556+
return chosen_frame_size(choice.codec, pfor_payload_size, plain_payload_size,
557+
compressed_payload_size);
558+
};
559+
const AutoPrxCodecChoice smallest = select(0);
560+
if (smallest.codec != PrxCodec::kZstd || pfor_payload_size > limits.max_uncomp_bytes) {
561+
budget->add(bytes(smallest));
562+
return smallest;
563+
}
564+
// Never zstd: PFOR, or raw when that is smaller still.
565+
const AutoPrxCodecChoice faster = select(100);
566+
const uint64_t zstd_bytes = bytes(smallest);
567+
const uint64_t pfor_bytes = pfor_frame_size(pfor_payload_size);
568+
const auto saving_percent = static_cast<uint32_t>((pfor_bytes - zstd_bytes) * 100 / pfor_bytes);
569+
return budget->trade(zstd_bytes, bytes(faster) - zstd_bytes, saving_percent) ? faster
570+
: smallest;
571+
}
572+
479573
namespace {
480574

481575
void write_zstd_compressed(Slice plain, Slice compressed, ByteSink* sink) {
@@ -543,9 +637,8 @@ Status build_prx_window_auto_from_flat(std::span<const uint32_t> positions_flat,
543637
testing::note_prx_raw_build();
544638
RETURN_IF_ERROR(zstd_compress(plain.view(), zstd_level, &compressed));
545639
}
546-
const AutoPrxCodecChoice choice =
547-
select_auto_prx_codec(payload.size(), plain.size(), compressed.size(),
548-
has_compressed, limits.max_uncomp_bytes);
640+
const AutoPrxCodecChoice choice = choose_auto_prx_codec(
641+
limits, payload.size(), plain.size(), compressed.size(), has_compressed);
549642
DCHECK(choice.readable);
550643
if (choice.codec == PrxCodec::kZstd) {
551644
write_zstd_compressed(plain.view(), Slice(compressed), sink);
@@ -1451,10 +1544,11 @@ void note_prx_delta_materialization() {
14511544
}
14521545

14531546
uint8_t select_auto_prx_codec_for_test(size_t pfor_payload_size, size_t plain_payload_size,
1454-
size_t compressed_payload_size, uint32_t max_uncomp_bytes) {
1455-
const AutoPrxCodecChoice choice =
1456-
select_auto_prx_codec(pfor_payload_size, plain_payload_size, compressed_payload_size,
1457-
plain_payload_size >= kPrxAutoZstdMinBytes, max_uncomp_bytes);
1547+
size_t compressed_payload_size, uint32_t max_uncomp_bytes,
1548+
uint32_t zstd_min_saving_percent) {
1549+
const AutoPrxCodecChoice choice = select_auto_prx_codec(
1550+
pfor_payload_size, plain_payload_size, compressed_payload_size,
1551+
plain_payload_size >= kPrxAutoZstdMinBytes, max_uncomp_bytes, zstd_min_saving_percent);
14581552
DCHECK(choice.readable);
14591553
return static_cast<uint8_t>(choice.codec);
14601554
}

‎be/src/storage/index/inverted/snii/format/prx_pod.h‎

Lines changed: 60 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@
1717

1818
#pragma once
1919

20+
#include <array>
2021
#include <cstddef>
2122
#include <cstdint>
2223
#include <span>
@@ -65,14 +66,64 @@ struct AutoPrxCodecChoice {
6566

6667
// Common policy for resident and replayable encoders: compare complete frames,
6768
// preserve PFOR/ZSTD/RAW priority on ties, and honor the reader byte limit.
69+
// A readable PFOR frame is replaced by the ZSTD one only if ZSTD is smaller by at least
70+
// `zstd_min_saving_percent` of the PFOR frame: every read of a ZSTD window pays a decompression,
71+
// which PFOR does not. 0 picks the smallest frame.
6872
AutoPrxCodecChoice select_auto_prx_codec(size_t pfor_payload_size, size_t plain_payload_size,
6973
size_t compressed_payload_size, bool has_compressed,
70-
uint32_t max_uncomp_bytes);
74+
uint32_t max_uncomp_bytes,
75+
uint32_t zstd_min_saving_percent = 0);
76+
77+
// Decides which .prx windows of one index of one segment are stored in a larger frame that reads
78+
// faster. Every read of a zstd window pays a decompression that PFOR does not, so storing a window
79+
// whose smallest frame is zstd in its PFOR (or smaller raw) frame trades bytes for reads. How much
80+
// zstd saves depends on the data: a fixed share (zstd_min_saving_percent) of 30 spends about 8% more
81+
// bytes on the windows of agent logs, about 5% on those of service logs, and much more on data whose
82+
// windows zstd shrinks by just under the share. The budget instead spends at most
83+
// kPrxZstdBudgetPercent of the smallest frames of the windows that get a zstd frame (a plain
84+
// payload of at least kPrxAutoZstdMinBytes), the only ones whose codec a share decides, on the
85+
// windows zstd shrinks least: a window is traded when its saving is below the share whose trades,
86+
// over the windows seen so far, fit the budget, and when what has actually been spent still fits
87+
// it, so no order of windows can make it spend more. Until kPrxZstdBudgetWarmupWindows windows are
88+
// seen the share is kPrxZstdBudgetPriorPercent. Windows written without a zstd frame (a singleton,
89+
// or too short to compress) are not counted: no share changes them, and counting their bytes would
90+
// let an index of many short posting lists grow its large windows by more than the budget.
91+
// Writer-only and deterministic for a given input.
92+
inline constexpr uint32_t kPrxZstdBudgetPercent = 8;
93+
inline constexpr uint32_t kPrxZstdBudgetPriorPercent = 30;
94+
inline constexpr uint32_t kPrxZstdBudgetWarmupWindows = 64;
95+
96+
class PrxZstdBudget {
97+
public:
98+
// Records a window that has nothing to trade: zstd is not its smallest frame, or PFOR is not
99+
// readable.
100+
void add(uint64_t smallest_frame_bytes);
101+
// Records a window whose smallest frame is zstd. `faster_extra_bytes` is what its faster frame
102+
// costs on top of the zstd one, `saving_percent` how much smaller the zstd frame is than the PFOR
103+
// one, in whole percent of the PFOR frame. Returns whether to store the faster frame.
104+
bool trade(uint64_t zstd_frame_bytes, uint64_t faster_extra_bytes, uint32_t saving_percent);
105+
// A window saving less than this share is traded if the budget allows.
106+
uint32_t min_saving_percent() const { return _min_saving_percent; }
107+
108+
private:
109+
void count_window();
110+
111+
// The extra bytes of trading each window seen, by its saving_percent.
112+
std::array<uint64_t, 101> _extra_by_saving {};
113+
uint64_t _smallest_bytes = 0;
114+
uint64_t _spent_bytes = 0;
115+
uint32_t _windows = 0;
116+
uint32_t _min_saving_percent = kPrxZstdBudgetPriorPercent;
117+
};
71118

72119
struct PrxWindowLimits {
73120
uint32_t max_docs;
74121
uint32_t max_positions;
75122
uint32_t max_uncomp_bytes;
123+
// Writer-only choice, ignored by readers: see select_auto_prx_codec.
124+
uint32_t zstd_min_saving_percent = 0;
125+
// Writer-only: when set, the segment's budget chooses the share instead.
126+
PrxZstdBudget* zstd_budget = nullptr;
76127
};
77128

78129
enum class PrxWindowBuildOutcome : uint8_t {
@@ -86,6 +137,12 @@ inline constexpr PrxWindowLimits kReaderPrxWindowLimits {
86137
.max_uncomp_bytes = 256U * 1024 * 1024,
87138
};
88139

140+
// The codec of a window under the writer's limits: select_auto_prx_codec with the fixed share, or,
141+
// when the limits carry a budget, the smallest frame unless the budget trades it for a faster one.
142+
AutoPrxCodecChoice choose_auto_prx_codec(const PrxWindowLimits& limits, size_t pfor_payload_size,
143+
size_t plain_payload_size, size_t compressed_payload_size,
144+
bool has_compressed);
145+
89146
// Writer policies may tighten, but never exceed, the reader's allocation
90147
// guards. Keeping this validation in the format layer makes it impossible for
91148
// a caller to configure a writer that emits frames the current reader rejects.
@@ -197,7 +254,8 @@ uint64_t prx_delta_materialization_count();
197254
void reset_prx_delta_materialization_count();
198255
void note_prx_delta_materialization();
199256
uint8_t select_auto_prx_codec_for_test(size_t pfor_payload_size, size_t plain_payload_size,
200-
size_t compressed_payload_size, uint32_t max_uncomp_bytes);
257+
size_t compressed_payload_size, uint32_t max_uncomp_bytes,
258+
uint32_t zstd_min_saving_percent = 0);
201259
#endif
202260

203261
} // namespace doris::snii::format::testing

‎be/src/storage/index/inverted/snii/writer/logical_index_writer.cpp‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -559,7 +559,9 @@ LogicalIndexWriter::LogicalIndexWriter(const SniiIndexInput& in, TrackedNullDoci
559559
: in.mem_reporter->make_reservation()),
560560
bsbf_bytes_reservation_(in.mem_reporter == nullptr
561561
? MemoryReporter::Reservation()
562-
: in.mem_reporter->make_reservation()) {}
562+
: in.mem_reporter->make_reservation()) {
563+
prx_window_limits_.zstd_budget = in.prx_zstd_budget ? &prx_zstd_budget_ : nullptr;
564+
}
563565

564566
Status LogicalIndexWriter::reserve_term_hash_for_append() {
565567
if (memory_reporter_ == nullptr || term_hashes_.size() < term_hashes_.capacity()) {

‎be/src/storage/index/inverted/snii/writer/logical_index_writer.h‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -76,6 +76,9 @@ struct SniiIndexInput {
7676
// tests may only tighten them to exercise extreme-window behavior without
7777
// allocating hundreds of MiB.
7878
format::PrxWindowLimits prx_window_limits = format::kReaderPrxWindowLimits;
79+
// When set, the .prx windows of the index share one format::PrxZstdBudget, which replaces
80+
// prx_window_limits.zstd_min_saving_percent.
81+
bool prx_zstd_budget = false;
7982
// Lexicographically sorted terms with ascending-docid postings. Used when
8083
// `term_source` is null (callers that already hold a materialized vector,
8184
// e.g. unit tests). The writer reads but does not retain these.
@@ -351,6 +354,8 @@ class LogicalIndexWriter {
351354
int dict_block_zstd_level_ = 3;
352355
int prx_zstd_level_ = 3;
353356
format::PrxWindowLimits prx_window_limits_ = format::kReaderPrxWindowLimits;
357+
// The windows of the index share it when SniiIndexInput::prx_zstd_budget is set.
358+
format::PrxZstdBudget prx_zstd_budget_;
354359
// The DICT region (zstd-compressed blocks) is staged here as blocks flush. It must
355360
// land contiguously AFTER the posting region (which streams concurrently to the
356361
// output), so it cannot stream directly; the orchestrator streams it into the

‎be/src/storage/index/inverted/snii/writer/posting_prx_encoder.cpp‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -176,7 +176,7 @@ Status encode_position_payloads(const PostingPositionView& positions,
176176

177177
Status choose_position_codec(PostingByteBuffer* plain, PostingByteBuffer* pfor,
178178
PostingByteBuffer* compressed, int level, bool singleton,
179-
uint32_t byte_limit, format::PrxCodec* codec) {
179+
const format::PrxWindowLimits& limits, format::PrxCodec* codec) {
180180
if (level >= 0) {
181181
*codec = format::PrxCodec::kRaw;
182182
if (level > 0) {
@@ -186,15 +186,16 @@ Status choose_position_codec(PostingByteBuffer* plain, PostingByteBuffer* pfor,
186186
return Status::OK();
187187
}
188188
*codec = singleton ? format::PrxCodec::kRaw : format::PrxCodec::kPfor;
189+
const uint32_t byte_limit = limits.max_uncomp_bytes;
189190
const bool use_zstd = plain->size() >= format::kPrxAutoZstdMinBytes;
190191
if (singleton || plain->size() > byte_limit || (!use_zstd && pfor->size() <= byte_limit)) {
191192
return Status::OK();
192193
}
193194
if (use_zstd) {
194195
RETURN_IF_ERROR(compress_posting_bytes(plain, -level, compressed));
195196
}
196-
*codec = format::select_auto_prx_codec(pfor->size(), plain->size(), compressed->size(),
197-
use_zstd, byte_limit)
197+
*codec = format::choose_auto_prx_codec(limits, pfor->size(), plain->size(), compressed->size(),
198+
use_zstd)
198199
.codec;
199200
return Status::OK();
200201
}
@@ -451,8 +452,7 @@ Status PostingPrxEncoder::build_replayable(const PostingPositionView& positions,
451452
}
452453
format::PrxCodec codec = format::PrxCodec::kRaw;
453454
RETURN_IF_ERROR(choose_position_codec(&plain, &pfor, &compressed, level,
454-
freqs.size() == 1 && freqs.front() == 1,
455-
limits.max_uncomp_bytes, &codec));
455+
freqs.size() == 1 && freqs.front() == 1, limits, &codec));
456456
ByteSink header;
457457
header.put_u8(static_cast<uint8_t>(codec));
458458
header.put_varint32(

0 commit comments

Comments
 (0)