You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 428c241
Browse filesBrowse the repository at this point in the historyBrowse files
[improvement](be) Let SNII skip zstd for .prx windows where it saves little over PFOR (apache#69649)
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: The auto .prx codec stores each position window in the
smallest of PFOR, zstd and
raw. Every read of a zstd window pays a decompression that PFOR does
not, and the choice ignores
that cost. On 99M AgentLogs rows 74% of the position windows read by an
unfiltered phrase query
were zstd, and decoding them took 58% of the index execution time of the
query.
PrxWindowLimits gets zstd_min_saving_percent, set from the new mutable
config
snii_prx_zstd_min_saving_percent. A readable PFOR frame is replaced by
the zstd one only if zstd is
smaller by at least that percentage of the PFOR frame. The default is
30; 0 picks the smaller frame
as before and writes a byte-identical index (99M rows: 61,886,854,661
bytes with and without this
change). Readers handle every codec, so the setting affects only
segments written afterwards, and
compaction rewrites with whatever is set then. The compression is still
computed, so the write cost
does not change.
Effect of the setting, 10M rows (index size, DS7 = unfiltered phrase on
the output column, position
decode time from the query profile): 10 -> +0.02% size, no change for
DS7; 20 -> +0.5%, no change;
30 -> +2.1%, decode 252 ms -> 85 ms; 50 -> +6.3%, no further gain; 100
(never zstd) -> +8.5%.
30 is the smallest value that removes the zstd windows of frequent
terms, which zstd beats PFOR by
20-30% on and which phrase queries read most. On 99M rows with 30: index
+1.9% (63.07 GB against
61.89 GB, still below the 65.70 GB of V3), load time within 2% of a
control load, hot queries in the
same process 0.63x (unfiltered phrase), 0.67x (phrase in input or
output), 0.75x (phrase prefix),
0.77x, 0.87x, the others within 1.02x (one query measured 1.06x once and
1.00-1.05x in three
repeats); cold runs 10-18% faster for the phrase queries. The best value
depends on the data, since
it is how much zstd beats PFOR on the window contents; it was calibrated
on one corpus.
0 commit comments