You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 5a970b5
Browse filesBrowse the repository at this point in the historyBrowse files
[improvement](be) Let SNII skip zstd for .prx windows where it saves little over PFOR
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: The auto .prx codec stores each position window in the smallest of PFOR, zstd and
raw. Every read of a zstd window pays a decompression that PFOR does not, and the choice ignores
that cost. On 99M AgentLogs rows 74% of the position windows read by an unfiltered phrase query
were zstd, and decoding them took 58% of the index execution time of the query.
PrxWindowLimits gets zstd_min_saving_percent, set from the new mutable config
snii_prx_zstd_min_saving_percent. A readable PFOR frame is replaced by the zstd one only if zstd is
smaller by at least that percentage of the PFOR frame. The default is 30; 0 picks the smaller frame
as before and writes a byte-identical index (99M rows: 61,886,854,661 bytes with and without this
change). Readers handle every codec, so the setting affects only segments written afterwards, and
compaction rewrites with whatever is set then. The compression is still computed, so the write cost
does not change.
Effect of the setting, 10M rows (index size, DS7 = unfiltered phrase on the output column, position
decode time from the query profile): 10 -> +0.02% size, no change for DS7; 20 -> +0.5%, no change;
30 -> +2.1%, decode 252 ms -> 85 ms; 50 -> +6.3%, no further gain; 100 (never zstd) -> +8.5%.
30 is the smallest value that removes the zstd windows of frequent terms, which zstd beats PFOR by
20-30% on and which phrase queries read most. On 99M rows with 30: index +1.9% (63.07 GB against
61.89 GB, still below the 65.70 GB of V3), load time within 2% of a control load, hot queries in the
same process 0.63x (unfiltered phrase), 0.67x (phrase in input or output), 0.75x (phrase prefix),
0.77x, 0.87x, the others within 1.02x (one query measured 1.06x once and 1.00-1.05x in three
repeats); cold runs 10-18% faster for the phrase queries. The best value depends on the data, since
it is how much zstd beats PFOR on the window contents; it was calibrated on one corpus.
### Release note
SNII position windows are stored with zstd only where it saves at least 30% over PFOR
(snii_prx_zstd_min_saving_percent): phrase queries are faster and the index is about 2% larger.
Set it to 0 for the previous behavior.
### Check List (For Author)
- Test: Unit Test (SniiPrxPodTest: the saving boundary, PFOR unreadable, limit validation; SniiWriterGoldenBytes: the config reaches the writer; the default is 30). Manual test: index size and load time at 10M and 99M rows, interleaved hot A/B and repeated cold runs.
- Behavior changed: Yes (newly written segments are about 2% larger; set snii_prx_zstd_min_saving_percent to 0 to restore)
- Does this need documentation: No
Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
0 commit comments