You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 0571eb6
Browse filesBrowse the repository at this point in the historyBrowse files
[improvement](be) Let each SNII index spend a size budget on faster .prx windows (#69769)
### What problem does this PR solve?
Issue Number: None
Related PR: #69649, #69770
Problem Summary: #69649 stores a .prx window as PFOR instead of zstd
when zstd saves less than `snii_prx_zstd_min_saving_percent` (30) of the
PFOR frame, because every read of a zstd window pays a decompression.
What that costs and buys depends on how much zstd saves on the data.
Replaying the 1.6M windows of 10M AgentLogs rows, 30 spends 8% more
bytes on the windows zstd compresses and cuts their decode time by 42%;
on the windows of TextBench service logs it spends 5% and cuts 6%; on
data whose windows zstd shrinks by just under 30% it stores every window
as PFOR. No single percentage fits all data.
Each index of a segment now spends a size budget instead:
`PrxZstdBudget`, owned by its `LogicalIndexWriter` and reached through
`PrxWindowLimits`, so the resident and the spilled encoder, and the
ordinary and the compaction build, all use it. A window whose smallest
frame is zstd is stored in its faster frame (PFOR, or raw when smaller)
when
- its saving is below the share whose trades, over the windows the index
has seen, fit 8% of their smallest frames, which puts the budget on the
windows zstd shrinks least, and
- the bytes the index has actually traded, this window included, still
fit 8% of the smallest frames seen. This makes 8% a bound for any order
of windows: fed the TextBench windows from the largest saving to the
smallest, the share alone spends 36%.
Until it has seen 64 windows the share is 30, the previous default. The
windows counted are those that get a zstd frame (a plain payload of at
least 512 bytes), the only ones whose codec a share decides; singleton
and short windows are written as before and counted neither in the
allowance nor in the warmup. The choice uses the frames the writer
builds anyway: nothing is compressed twice or scanned ahead, and the
same input gives the same bytes. `snii_prx_zstd_min_saving_percent` now
defaults to -1 (the budget); 0-100 keeps a fixed share, 0 the smallest
frame.
The budget is 8% rather than less: with it the AgentLogs phrase queries
below read nearly the same frames as with 30, while in the replay a 5%
budget cuts the decode time by 31% instead of 39%.
Same binary, one stream load per table, size against the smallest frames
(0):
| data | index, 30 | index, budget | table, 30 | table, budget |
|---|---|---|---|---|
| AgentLogs 99M rows | +4.13% | +3.65% | +1.92% | +1.69% |
| AgentLogs 10M rows | +4.59% | +3.37% | +2.13% | +1.57% |
| TextBench 12M rows | +0.89% | +0.79% | +0.51% | +0.45% |
- The phrase queries read the same frames with 30 and with the budget:
at 99M rows all PFOR for DS1, DS6, DS7, A6 and A9 (for example 107,307
frames for DS7), and about 11,170 zstd frames of about 259,000 for A10
with both; on TextBench most windows stay zstd either way (582 against
583 zstd frames of 599 for 'connection reset').
- Hot queries in one process, budget data against 30 data with the same
read path (paired cycles: 6 for the 25 AgentLogs queries at 10M rows, 10
for the 15 TextBench queries): 0.96-1.06 and 0.97-1.05; every query
returns the same result.
- Against the data written before #69649 (smallest frames), the 11
AgentLogs phrase queries at 99M rows take 0.83x in geometric mean with
the budget data (DS7 0.63x, A6 0.67x, A9 0.70x, DS4 0.74x, A10 0.77x),
as with 30 (0.83x). This run also had the read-path change of #69770,
which does not touch the PFOR windows these queries read.
- Data bytes are identical, and the index written with 0 and with 30 has
exactly the size the previous binary writes.
- Write cost: the choice adds about 48 ns per window, 79 ms over the
1.6M AgentLogs windows.
### Release note
SNII now chooses per index how many position windows to store in the
faster but larger PFOR form, spending at most 8% of the size of the
windows zstd compresses; `snii_prx_zstd_min_saving_percent` defaults to
-1 for this, and 0-100 still sets a fixed percentage.
0 commit comments