You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 170822d
Browse filesBrowse the repository at this point in the historyBrowse files
[improvement](be) Let each SNII index spend a size budget on faster .prx windows
### What problem does this PR solve?
Issue Number: None
Related PR: #69649
Problem Summary: #69649 stores a .prx window as PFOR instead of zstd when zstd saves less than
snii_prx_zstd_min_saving_percent (30) of the PFOR frame, because every read of a zstd window pays a
decompression. What that costs and buys depends on how much zstd saves on the data. Replaying the
1.6M windows of 10M AgentLogs rows, 30 spends 8% more bytes on the windows zstd compresses and
cuts their decode time by 42%; on the windows of TextBench service logs it spends 5% and cuts 6%;
on data whose windows zstd shrinks by just under 30% it stores every window as PFOR. No single
percentage fits all data.
Each index of a segment now spends a size budget instead: PrxZstdBudget, owned by its
LogicalIndexWriter and reached through PrxWindowLimits, so the resident and the spilled encoder,
and the ordinary and the compaction build, all use it. A window whose smallest frame is zstd is
stored in its faster frame (PFOR, or raw when smaller) when
- its saving is below the share whose trades, over the windows the index has seen, fit 8% of their
smallest frames, which puts the budget on the windows zstd shrinks least, and
- the bytes the index has actually traded, this window included, still fit 8% of the smallest
frames seen. This makes 8% a bound for any order of windows: fed the TextBench windows from the
largest saving to the smallest, the share alone spends 36%.
Until it has seen 64 windows the share is 30, the previous default. The choice uses the frames the
writer builds anyway: nothing is compressed twice or scanned ahead, and the same input gives the
same bytes. snii_prx_zstd_min_saving_percent now defaults to -1 (the budget); 0-100 keeps a fixed
share, 0 the smallest frame.
The budget is 8% rather than less: with it the AgentLogs phrase queries below read nearly the same
frames as with 30, while in the replay a 5% budget cuts the decode time by 31% instead of 39%.
10M rows each, same binary, one stream load per table, size against the smallest frames (0):
- AgentLogs: index +4.59% with 30, +3.37% with the budget (table +2.13%, +1.57%). The phrase
queries read the same frames: all PFOR for DS1, DS2, DS6, DS7, A6 and A9 with both, and 326
against 324 zstd frames of 5011 for DS4, 1049 against 1041 of 25930 for A10.
- TextBench: index +0.89% with 30, +0.79% with the budget (table +0.51%, +0.45%). Most windows
stay zstd either way (P1 'connection reset': 582 against 583 zstd frames of 599).
- Data bytes are identical; the index written with 0 and with 30 is byte-for-byte the size the
previous binary writes.
- Hot queries, budget data against 30 data in one process (6-10 paired cycles, ratio of medians per
cycle): all 25 AgentLogs and 15 TextBench queries 0.96-1.04, the same results.
- Write cost: the choice adds about 48 ns per window, 79 ms over the 1.6M AgentLogs windows.
### Release note
SNII now chooses per index how many position windows to store in the faster but larger PFOR form,
spending at most 8% of their size; snii_prx_zstd_min_saving_percent defaults to -1 for this, and
0-100 still sets a fixed percentage.
### Check List (For Author)
- Test: Unit Test (SniiPrxZstdBudget: the prior share, the windows zstd shrinks least traded first,
the bound under a descending order, windows with nothing to trade; SniiPrxPodTest: the faster
frame and when the budget is asked; SniiStreamedWriterSessionTest: windows that save 1-29% through
the ordinary and the streamed build, where 30 trades all of them and the budget at most 8%;
SniiWriterPrxZstdSaving: the default and the config mapping). Manual test: index size of 10M-row
AgentLogs and TextBench loads with 0, 30 and the budget, frames read per codec from query profiles.
- Behavior changed: Yes (the default .prx codec choice of newly written segments; existing segments
are read as before)
- Does this need documentation: No
Co-Authored-By: Claude Opus 5.5 <[email protected]>
0 commit comments