Skip to content

Commit 5a970b5

Browse files
martin.foo838claude
andcommitted
[improvement](be) Let SNII skip zstd for .prx windows where it saves little over PFOR
### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: The auto .prx codec stores each position window in the smallest of PFOR, zstd and raw. Every read of a zstd window pays a decompression that PFOR does not, and the choice ignores that cost. On 99M AgentLogs rows 74% of the position windows read by an unfiltered phrase query were zstd, and decoding them took 58% of the index execution time of the query. PrxWindowLimits gets zstd_min_saving_percent, set from the new mutable config snii_prx_zstd_min_saving_percent. A readable PFOR frame is replaced by the zstd one only if zstd is smaller by at least that percentage of the PFOR frame. The default is 30; 0 picks the smaller frame as before and writes a byte-identical index (99M rows: 61,886,854,661 bytes with and without this change). Readers handle every codec, so the setting affects only segments written afterwards, and compaction rewrites with whatever is set then. The compression is still computed, so the write cost does not change. Effect of the setting, 10M rows (index size, DS7 = unfiltered phrase on the output column, position decode time from the query profile): 10 -> +0.02% size, no change for DS7; 20 -> +0.5%, no change; 30 -> +2.1%, decode 252 ms -> 85 ms; 50 -> +6.3%, no further gain; 100 (never zstd) -> +8.5%. 30 is the smallest value that removes the zstd windows of frequent terms, which zstd beats PFOR by 20-30% on and which phrase queries read most. On 99M rows with 30: index +1.9% (63.07 GB against 61.89 GB, still below the 65.70 GB of V3), load time within 2% of a control load, hot queries in the same process 0.63x (unfiltered phrase), 0.67x (phrase in input or output), 0.75x (phrase prefix), 0.77x, 0.87x, the others within 1.02x (one query measured 1.06x once and 1.00-1.05x in three repeats); cold runs 10-18% faster for the phrase queries. The best value depends on the data, since it is how much zstd beats PFOR on the window contents; it was calibrated on one corpus. ### Release note SNII position windows are stored with zstd only where it saves at least 30% over PFOR (snii_prx_zstd_min_saving_percent): phrase queries are faster and the index is about 2% larger. Set it to 0 for the previous behavior. ### Check List (For Author) - Test: Unit Test (SniiPrxPodTest: the saving boundary, PFOR unreadable, limit validation; SniiWriterGoldenBytes: the config reaches the writer; the default is 30). Manual test: index size and load time at 10M and 99M rows, interleaved hot A/B and repeated cold runs. - Behavior changed: Yes (newly written segments are about 2% larger; set snii_prx_zstd_min_saving_percent to 0 to restore) - Does this need documentation: No Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
1 parent da4f490 commit 5a970b5

8 files changed

Lines changed: 137 additions & 16 deletions

File tree

‎be/src/common/config.cpp‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1426,6 +1426,12 @@ DEFINE_mInt32(inverted_index_max_buffered_docs, "-1");
14261426
// Zstd levels for SNII DICT blocks and .prx windows. These affect writes; each segment records its compression mode.
14271427
DEFINE_mInt32(snii_dict_block_zstd_level, "3");
14281428
DEFINE_mInt32(snii_prx_zstd_level, "3");
1429+
// A .prx window is stored zstd-compressed only if that frame is smaller than the PFOR one by at
1430+
// least this percentage; 0 picks the smaller frame. Every read of a zstd window pays a
1431+
// decompression that PFOR does not, and on text positions zstd beats PFOR by 20-30% only on the
1432+
// large windows of frequent terms, which are the ones phrase queries read most: 30 costs about 2%
1433+
// of the index size. Readers handle every codec, so changing this never affects existing segments.
1434+
DEFINE_mInt32(snii_prx_zstd_min_saving_percent, "30");
14291435
// Zstd level for direct-load .prx regions. Compaction rewrites them using snii_prx_zstd_level.
14301436
DEFINE_mInt32(snii_prx_zstd_level_direct_load, "3");
14311437
// G16-d: target SNII dict block size in bytes; 0 uses the format default

‎be/src/common/config.h‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1466,6 +1466,9 @@ DECLARE_mInt32(inverted_index_max_buffered_docs);
14661466
// import CPU; see the DEFINEs in config.cpp).
14671467
DECLARE_mInt32(snii_dict_block_zstd_level);
14681468
DECLARE_mInt32(snii_prx_zstd_level);
1469+
// A .prx window is stored zstd-compressed only if that frame is smaller than the PFOR one by at
1470+
// least this percentage; 0 picks the smaller frame. Affects segments written from now on.
1471+
DECLARE_mInt32(snii_prx_zstd_min_saving_percent);
14691472
// Patch C: prx zstd level for DIRECT-LOAD segments only (default 3, cheaper
14701473
// import); compaction rewrites at snii_prx_zstd_level so settled segments are
14711474
// unaffected. Full contract at the DEFINE in config.cpp.

‎be/src/storage/index/index_file_writer.cpp‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -146,6 +146,8 @@ void snii_resolve_index_write_params(bool is_direct_load, bool has_norms,
146146
input->prx_zstd_level = std::clamp(
147147
is_direct_load ? config::snii_prx_zstd_level_direct_load : config::snii_prx_zstd_level,
148148
3, 19);
149+
input->prx_window_limits.zstd_min_saving_percent =
150+
static_cast<uint32_t>(std::clamp(config::snii_prx_zstd_min_saving_percent, 0, 100));
149151
// G16-d: dict block size experiment knob; <= 0 keeps the format default.
150152
if (config::snii_target_dict_block_bytes > 0) {
151153
input->target_dict_block_bytes =

‎be/src/storage/index/inverted/snii/format/prx_pod.cpp‎

Lines changed: 22 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -60,6 +60,10 @@ Status validate_prx_window_limits(const PrxWindowLimits& limits) {
6060
return Status::Error<ErrorCode::INVALID_ARGUMENT, false>(
6161
"prx: writer window limits exceed reader limits");
6262
}
63+
if (limits.zstd_min_saving_percent > 100) {
64+
return Status::Error<ErrorCode::INVALID_ARGUMENT, false>(
65+
"prx: zstd minimum saving is a percentage");
66+
}
6367
return Status::OK();
6468
}
6569

@@ -451,7 +455,8 @@ size_t zstd_frame_size(size_t plain_size, size_t compressed_size) {
451455
// before PFOR materialization in build_prx_window_auto_from_flat.
452456
AutoPrxCodecChoice select_auto_prx_codec(size_t pfor_payload_size, size_t plain_payload_size,
453457
size_t compressed_payload_size, bool has_compressed,
454-
uint32_t max_uncomp_bytes) {
458+
uint32_t max_uncomp_bytes,
459+
uint32_t zstd_min_saving_percent) {
455460
const bool pfor_readable = pfor_payload_size <= max_uncomp_bytes;
456461
const bool plain_readable = plain_payload_size <= max_uncomp_bytes;
457462
AutoPrxCodecChoice choice;
@@ -462,7 +467,14 @@ AutoPrxCodecChoice select_auto_prx_codec(size_t pfor_payload_size, size_t plain_
462467
}
463468
if (has_compressed && plain_readable) {
464469
const size_t frame_size = zstd_frame_size(plain_payload_size, compressed_payload_size);
465-
if (!choice.readable || frame_size < selected_frame_size) {
470+
// Against a PFOR frame ZSTD must save the configured share; without one it only has to win.
471+
const bool saves_enough =
472+
!choice.readable ||
473+
(frame_size < selected_frame_size &&
474+
static_cast<uint64_t>(frame_size) * 100 <=
475+
static_cast<uint64_t>(selected_frame_size) *
476+
(100 - std::min<uint32_t>(zstd_min_saving_percent, 100)));
477+
if (saves_enough) {
466478
choice = {.codec = PrxCodec::kZstd, .readable = true};
467479
selected_frame_size = frame_size;
468480
}
@@ -543,9 +555,9 @@ Status build_prx_window_auto_from_flat(std::span<const uint32_t> positions_flat,
543555
testing::note_prx_raw_build();
544556
RETURN_IF_ERROR(zstd_compress(plain.view(), zstd_level, &compressed));
545557
}
546-
const AutoPrxCodecChoice choice =
547-
select_auto_prx_codec(payload.size(), plain.size(), compressed.size(),
548-
has_compressed, limits.max_uncomp_bytes);
558+
const AutoPrxCodecChoice choice = select_auto_prx_codec(
559+
payload.size(), plain.size(), compressed.size(), has_compressed,
560+
limits.max_uncomp_bytes, limits.zstd_min_saving_percent);
549561
DCHECK(choice.readable);
550562
if (choice.codec == PrxCodec::kZstd) {
551563
write_zstd_compressed(plain.view(), Slice(compressed), sink);
@@ -1405,10 +1417,11 @@ void note_prx_delta_materialization() {
14051417
}
14061418

14071419
uint8_t select_auto_prx_codec_for_test(size_t pfor_payload_size, size_t plain_payload_size,
1408-
size_t compressed_payload_size, uint32_t max_uncomp_bytes) {
1409-
const AutoPrxCodecChoice choice =
1410-
select_auto_prx_codec(pfor_payload_size, plain_payload_size, compressed_payload_size,
1411-
plain_payload_size >= kPrxAutoZstdMinBytes, max_uncomp_bytes);
1420+
size_t compressed_payload_size, uint32_t max_uncomp_bytes,
1421+
uint32_t zstd_min_saving_percent) {
1422+
const AutoPrxCodecChoice choice = select_auto_prx_codec(
1423+
pfor_payload_size, plain_payload_size, compressed_payload_size,
1424+
plain_payload_size >= kPrxAutoZstdMinBytes, max_uncomp_bytes, zstd_min_saving_percent);
14121425
DCHECK(choice.readable);
14131426
return static_cast<uint8_t>(choice.codec);
14141427
}

‎be/src/storage/index/inverted/snii/format/prx_pod.h‎

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -65,14 +65,20 @@ struct AutoPrxCodecChoice {
6565

6666
// Common policy for resident and replayable encoders: compare complete frames,
6767
// preserve PFOR/ZSTD/RAW priority on ties, and honor the reader byte limit.
68+
// A readable PFOR frame is replaced by the ZSTD one only if ZSTD is smaller by at least
69+
// `zstd_min_saving_percent` of the PFOR frame: every read of a ZSTD window pays a decompression,
70+
// which PFOR does not. 0 picks the smallest frame.
6871
AutoPrxCodecChoice select_auto_prx_codec(size_t pfor_payload_size, size_t plain_payload_size,
6972
size_t compressed_payload_size, bool has_compressed,
70-
uint32_t max_uncomp_bytes);
73+
uint32_t max_uncomp_bytes,
74+
uint32_t zstd_min_saving_percent = 0);
7175

7276
struct PrxWindowLimits {
7377
uint32_t max_docs;
7478
uint32_t max_positions;
7579
uint32_t max_uncomp_bytes;
80+
// Writer-only choice, ignored by readers: see select_auto_prx_codec.
81+
uint32_t zstd_min_saving_percent = 0;
7682
};
7783

7884
enum class PrxWindowBuildOutcome : uint8_t {
@@ -197,7 +203,8 @@ uint64_t prx_delta_materialization_count();
197203
void reset_prx_delta_materialization_count();
198204
void note_prx_delta_materialization();
199205
uint8_t select_auto_prx_codec_for_test(size_t pfor_payload_size, size_t plain_payload_size,
200-
size_t compressed_payload_size, uint32_t max_uncomp_bytes);
206+
size_t compressed_payload_size, uint32_t max_uncomp_bytes,
207+
uint32_t zstd_min_saving_percent = 0);
201208
#endif
202209

203210
} // namespace doris::snii::format::testing

‎be/src/storage/index/inverted/snii/writer/posting_prx_encoder.cpp‎

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -176,7 +176,8 @@ Status encode_position_payloads(const PostingPositionView& positions,
176176

177177
Status choose_position_codec(PostingByteBuffer* plain, PostingByteBuffer* pfor,
178178
PostingByteBuffer* compressed, int level, bool singleton,
179-
uint32_t byte_limit, format::PrxCodec* codec) {
179+
uint32_t byte_limit, uint32_t zstd_min_saving_percent,
180+
format::PrxCodec* codec) {
180181
if (level >= 0) {
181182
*codec = format::PrxCodec::kRaw;
182183
if (level > 0) {
@@ -194,7 +195,7 @@ Status choose_position_codec(PostingByteBuffer* plain, PostingByteBuffer* pfor,
194195
RETURN_IF_ERROR(compress_posting_bytes(plain, -level, compressed));
195196
}
196197
*codec = format::select_auto_prx_codec(pfor->size(), plain->size(), compressed->size(),
197-
use_zstd, byte_limit)
198+
use_zstd, byte_limit, zstd_min_saving_percent)
198199
.codec;
199200
return Status::OK();
200201
}
@@ -450,9 +451,9 @@ Status PostingPrxEncoder::build_replayable(const PostingPositionView& positions,
450451
return Status::OK();
451452
}
452453
format::PrxCodec codec = format::PrxCodec::kRaw;
453-
RETURN_IF_ERROR(choose_position_codec(&plain, &pfor, &compressed, level,
454-
freqs.size() == 1 && freqs.front() == 1,
455-
limits.max_uncomp_bytes, &codec));
454+
RETURN_IF_ERROR(choose_position_codec(
455+
&plain, &pfor, &compressed, level, freqs.size() == 1 && freqs.front() == 1,
456+
limits.max_uncomp_bytes, limits.zstd_min_saving_percent, &codec));
456457
ByteSink header;
457458
header.put_u8(static_cast<uint8_t>(codec));
458459
header.put_varint32(

‎be/test/storage/index/inverted/snii/format/prx_pod_test.cpp‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1453,6 +1453,41 @@ TEST(SniiPrxPodTest, MaterializedRawCandidateMustStrictlyReduceFrame) {
14531453
static_cast<uint8_t>(PrxCodec::kPfor));
14541454
}
14551455

1456+
// ZSTD replaces a readable PFOR frame only if it saves the configured share of the PFOR frame.
1457+
// PFOR frame 1007 bytes, ZSTD frame 859 bytes: it saves 14.7%.
1458+
TEST(SniiPrxPodTest, ZstdMustSaveTheConfiguredShareOverPfor) {
1459+
constexpr uint32_t kLimit = 4096;
1460+
auto select = [&](uint32_t min_saving_percent) {
1461+
return doris::snii::format::testing::select_auto_prx_codec_for_test(
1462+
/*pfor_payload_size=*/1000, /*plain_payload_size=*/2000,
1463+
/*compressed_payload_size=*/850, kLimit, min_saving_percent);
1464+
};
1465+
EXPECT_EQ(select(0), static_cast<uint8_t>(PrxCodec::kZstd));
1466+
EXPECT_EQ(select(10), static_cast<uint8_t>(PrxCodec::kZstd));
1467+
EXPECT_EQ(select(14), static_cast<uint8_t>(PrxCodec::kZstd));
1468+
EXPECT_EQ(select(15), static_cast<uint8_t>(PrxCodec::kPfor));
1469+
EXPECT_EQ(select(20), static_cast<uint8_t>(PrxCodec::kPfor));
1470+
EXPECT_EQ(select(100), static_cast<uint8_t>(PrxCodec::kPfor));
1471+
}
1472+
1473+
// The share only protects PFOR: when the PFOR frame exceeds the reader byte limit, ZSTD is the
1474+
// only readable compressed form and is chosen whatever the share.
1475+
TEST(SniiPrxPodTest, ZstdShareDoesNotApplyWhenPforIsUnreadable) {
1476+
constexpr uint32_t kLimit = 1500;
1477+
EXPECT_EQ(doris::snii::format::testing::select_auto_prx_codec_for_test(
1478+
/*pfor_payload_size=*/1600, /*plain_payload_size=*/1400,
1479+
/*compressed_payload_size=*/500, kLimit, /*zstd_min_saving_percent=*/100),
1480+
static_cast<uint8_t>(PrxCodec::kZstd));
1481+
}
1482+
1483+
TEST(SniiPrxPodTest, ZstdMinSavingAboveOneHundredPercentIsRejected) {
1484+
doris::snii::format::PrxWindowLimits limits = doris::snii::format::kReaderPrxWindowLimits;
1485+
limits.zstd_min_saving_percent = 100;
1486+
EXPECT_TRUE(doris::snii::format::validate_prx_window_limits(limits).ok());
1487+
limits.zstd_min_saving_percent = 101;
1488+
EXPECT_FALSE(doris::snii::format::validate_prx_window_limits(limits).ok());
1489+
}
1490+
14561491
// [functional] F-RT-small: a small flat window round-trips through the CSR reader
14571492
// (codec=pfor path) back to the original per-doc positions.
14581493
TEST(SniiPrxPodTest, SmallFlatWindowRoundTripsViaCsr) {

‎be/test/storage/index/inverted/snii/writer/snii_writer_golden_bytes_test.cpp‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -253,6 +253,25 @@ uint64_t golden_digest(const std::string& name, const TabletIndex& meta) {
253253
return fnv1a64(bytes);
254254
}
255255

256+
// Writes `rows` through the production stack and returns the size of the segment file.
257+
size_t written_size(const std::string& name, const TabletIndex& meta,
258+
const std::vector<std::string>& rows) {
259+
const std::string path = std::string(kTestDir) + "/" + name + ".idx";
260+
io::FileWriterPtr file_writer;
261+
EXPECT_TRUE(io::global_local_filesystem()->create_file(path, &file_writer).ok());
262+
IndexFileWriter index_file_writer(io::global_local_filesystem(), path, "size_rowset",
263+
/*seg_id=*/0, InvertedIndexStorageFormatPB::SNII,
264+
std::move(file_writer), /*can_use_ram_dir=*/true,
265+
/*tablet_id=*/901);
266+
SniiIndexColumnWriter writer(&index_file_writer, &meta, FieldType::OLAP_FIELD_TYPE_VARCHAR);
267+
EXPECT_TRUE(writer.init().ok());
268+
add_batch(&writer, rows);
269+
EXPECT_TRUE(writer.finish().ok());
270+
EXPECT_TRUE(index_file_writer.begin_close().ok());
271+
EXPECT_TRUE(index_file_writer.finish_close().ok());
272+
return read_file_bytes(path).size();
273+
}
274+
256275
class SniiWriterGoldenBytes : public testing::Test {
257276
protected:
258277
void SetUp() override {
@@ -261,24 +280,28 @@ class SniiWriterGoldenBytes : public testing::Test {
261280
_saved_dict_lvl = config::snii_dict_block_zstd_level;
262281
_saved_prx_lvl = config::snii_prx_zstd_level;
263282
_saved_prx_load_lvl = config::snii_prx_zstd_level_direct_load;
283+
_saved_prx_saving = config::snii_prx_zstd_min_saving_percent;
264284
assert_ok(io::global_local_filesystem()->delete_directory(kTestDir));
265285
assert_ok(io::global_local_filesystem()->create_directory(kTestDir));
266286
config::snii_dict_block_zstd_level = 3;
267287
config::snii_prx_zstd_level = 3;
268288
config::snii_prx_zstd_level_direct_load = 3;
289+
config::snii_prx_zstd_min_saving_percent = 0;
269290
}
270291

271292
void TearDown() override {
272293
config::snii_dict_block_zstd_level = _saved_dict_lvl;
273294
config::snii_prx_zstd_level = _saved_prx_lvl;
274295
config::snii_prx_zstd_level_direct_load = _saved_prx_load_lvl;
296+
config::snii_prx_zstd_min_saving_percent = _saved_prx_saving;
275297
EXPECT_TRUE(io::global_local_filesystem()->delete_directory(kTestDir).ok());
276298
}
277299

278300
private:
279301
int32_t _saved_dict_lvl = 3;
280302
int32_t _saved_prx_lvl = 3;
281303
int32_t _saved_prx_load_lvl = 3;
304+
int32_t _saved_prx_saving = 0;
282305
};
283306

284307
// Whole-image digests re-harvested after core metadata stopped writing an empty
@@ -295,6 +318,37 @@ constexpr uint64_t kGoldenEnglishPhrase = 0x8157f9dd81c5397cULL;
295318
constexpr uint64_t kGoldenUnicodePhrase = 0x9e2a4ba4abedaef9ULL;
296319
constexpr uint64_t kGoldenKeywordDocsOnly = 0x27cec04c70573201ULL;
297320

321+
// The default of the knob is part of the written format's behavior: the fixture above pins it to 0
322+
// for the digests, so this reads the value a fresh process starts with.
323+
TEST(SniiWriterPrxZstdSaving, DefaultIsThirtyPercent) {
324+
EXPECT_EQ(config::snii_prx_zstd_min_saving_percent, 30);
325+
}
326+
327+
// Every document repeats the same positions, so a window is far smaller as zstd than as PFOR.
328+
// 0 and 30 keep zstd; 100 never picks it, so the segment is larger.
329+
TEST_F(SniiWriterGoldenBytes, PrxZstdMinSavingConfigReachesTheWriter) {
330+
std::vector<std::string> rows;
331+
for (int doc = 0; doc < 2000; ++doc) {
332+
std::string row;
333+
for (int token = 0; token < 31; ++token) {
334+
row += (token == 1 || token == 5 || token == 6 || token == 17 || token == 30)
335+
? "x "
336+
: "t" + std::to_string(token) + " ";
337+
}
338+
rows.push_back(row);
339+
}
340+
const TabletIndex meta =
341+
make_meta({{"parser", "unicode"}, {"lower_case", "true"}, {"support_phrase", "true"}});
342+
config::snii_prx_zstd_min_saving_percent = 0;
343+
const size_t smallest = written_size("saving_0", meta, rows);
344+
config::snii_prx_zstd_min_saving_percent = 30;
345+
const size_t thirty = written_size("saving_30", meta, rows);
346+
config::snii_prx_zstd_min_saving_percent = 100;
347+
const size_t never_zstd = written_size("saving_100", meta, rows);
348+
EXPECT_EQ(thirty, smallest);
349+
EXPECT_GT(never_zstd, smallest + 1000) << "smallest=" << smallest;
350+
}
351+
298352
TEST_F(SniiWriterGoldenBytes, EnglishPhrase) {
299353
const TabletIndex meta =
300354
make_meta({{"parser", "english"}, {"lower_case", "true"}, {"support_phrase", "true"}});

0 commit comments

Comments
 (0)