…prx windows (#69769)

### What problem does this PR solve?

Issue Number: None

Related PR: #69649, #69770

Problem Summary: #69649 stores a .prx window as PFOR instead of zstd
when zstd saves less than `snii_prx_zstd_min_saving_percent` (30) of the
PFOR frame, because every read of a zstd window pays a decompression.
What that costs and buys depends on how much zstd saves on the data.
Replaying the 1.6M windows of 10M AgentLogs rows, 30 spends 8% more
bytes on the windows zstd compresses and cuts their decode time by 42%;
on the windows of TextBench service logs it spends 5% and cuts 6%; on
data whose windows zstd shrinks by just under 30% it stores every window
as PFOR. No single percentage fits all data.

Each index of a segment now spends a size budget instead:
`PrxZstdBudget`, owned by its `LogicalIndexWriter` and reached through
`PrxWindowLimits`, so the resident and the spilled encoder, and the
ordinary and the compaction build, all use it. A window whose smallest
frame is zstd is stored in its faster frame (PFOR, or raw when smaller)
when
- its saving is below the share whose trades, over the windows the index
has seen, fit 8% of their smallest frames, which puts the budget on the
windows zstd shrinks least, and
- the bytes the index has actually traded, this window included, still
fit 8% of the smallest frames seen. This makes 8% a bound for any order
of windows: fed the TextBench windows from the largest saving to the
smallest, the share alone spends 36%.

Until it has seen 64 windows the share is 30, the previous default. The
windows counted are those that get a zstd frame (a plain payload of at
least 512 bytes), the only ones whose codec a share decides; singleton
and short windows are written as before and counted neither in the
allowance nor in the warmup. The choice uses the frames the writer
builds anyway: nothing is compressed twice or scanned ahead, and the
same input gives the same bytes. `snii_prx_zstd_min_saving_percent` now
defaults to -1 (the budget); 0-100 keeps a fixed share, 0 the smallest
frame.

The budget is 8% rather than less: with it the AgentLogs phrase queries
below read nearly the same frames as with 30, while in the replay a 5%
budget cuts the decode time by 31% instead of 39%.

Same binary, one stream load per table, size against the smallest frames
(0):

| data | index, 30 | index, budget | table, 30 | table, budget |
|---|---|---|---|---|
| AgentLogs 99M rows | +4.13% | +3.65% | +1.92% | +1.69% |
| AgentLogs 10M rows | +4.59% | +3.37% | +2.13% | +1.57% |
| TextBench 12M rows | +0.89% | +0.79% | +0.51% | +0.45% |

- The phrase queries read the same frames with 30 and with the budget:
at 99M rows all PFOR for DS1, DS6, DS7, A6 and A9 (for example 107,307
frames for DS7), and about 11,170 zstd frames of about 259,000 for A10
with both; on TextBench most windows stay zstd either way (582 against
583 zstd frames of 599 for 'connection reset').
- Hot queries in one process, budget data against 30 data with the same
read path (paired cycles: 6 for the 25 AgentLogs queries at 10M rows, 10
for the 15 TextBench queries): 0.96-1.06 and 0.97-1.05; every query
returns the same result.
- Against the data written before #69649 (smallest frames), the 11
AgentLogs phrase queries at 99M rows take 0.83x in geometric mean with
the budget data (DS7 0.63x, A6 0.67x, A9 0.70x, DS4 0.74x, A10 0.77x),
as with 30 (0.83x). This run also had the read-path change of #69770,
which does not touch the PFOR windows these queries read.
- Data bytes are identical, and the index written with 0 and with 30 has
exactly the size the previous binary writes.
- Write cost: the choice adds about 48 ns per window, 79 ms over the
1.6M AgentLogs windows.

### Release note

SNII now chooses per index how many position windows to store in the
faster but larger PFOR form, spending at most 8% of the size of the
windows zstd compresses; `snii_prx_zstd_min_saving_percent` defaults to
-1 for this, and 0-100 still sets a fixed percentage.

### What problem does this PR solve?

Issue Number: None

Related PR: #69649, #69769

Problem Summary: A phrase query decodes the positions of the selected
documents of every .prx window it reads. Timed per window on a 10M-row
AgentLogs load, a zstd window cost 18.6 us against 1.4 us for a PFOR
window: 11.7 us to decode the plain payload, 4.8 us to decompress, and
about 2 us more because every window created and freed its own zstd
context (about 96 KB) and a zero-filled output vector. Two commits:

1. **Decode the plain payload without a Status per document.**
`decode_payload_csr_selective` made at least one out-of-line call
returning a Status for every document of the window, selected or not
(`get_varint32_fast`, `skip_varints`, `decode_delta_run`). The loop now
keeps the cursor in a register and handles the usual cases inline: a
single-byte count, walking past the deltas of a document that is not
selected, and a run of single-byte deltas. Anything else -- a multi-byte
value, a run that does not fit, a prefix sum that would overflow -- goes
to the checked decoders from the same cursor, so they decide and report
as before. The output vector grows as before; nothing is reserved up
front.
2. **Reuse one zstd context and buffer per index source.**
`ZstdDecompressor` keeps one context, created at the first zstd frame,
and one output buffer that only grows and is not zero-filled
(decompression writes every byte before it is read). The buffer comes
from the Doris allocator (`DorisUniqueBufferPtr`), so the decoding
task's memory tracker accounts for it while it is held.
`SniiIndexSource` owns one, and the cursors it opens pass it to the .prx
decoder through `PrxDecodeContext`. The cursors of a source decode on
its thread one window at a time, and the decoded payload is consumed
before the next window is read. A source lives for one query on one
segment, so a query holds at most one context and one buffer per segment
it reads, and only if it decodes a zstd window; before, each window
created and released them. Compaction decodes through an allocation
gate, which keeps the old path, as does a cursor given no decompressor.

Hot queries, old against new read path in one process (run-time switch
in an experiment build, 6 paired cycles), on 99M AgentLogs rows written
with the smallest frames, as every segment written before #69649 is
(mostly zstd windows): DS7 (unfiltered phrase) 0.87x, A6 0.89x, A10
0.93x, DS4 (phrase prefix) 0.94x, A9 0.95x, DS1 0.96x; repeating the old
path against itself gave 0.93-1.05. The gain follows the zstd windows a
query reads: the phrase queries of AgentLogs written with #69649 read
PFOR windows and do not change, and on TextBench, whose large windows
stay zstd, the time per window is the decompression itself and the
change is within noise. Every query returns the same result.

### Release note

None

### What problem does this PR solve?

Issue Number: None

Related PR: #69649, #69769

Problem Summary: A phrase query decodes the positions of the selected
documents of every .prx window it reads. Timed per window on a 10M-row
AgentLogs load, a zstd window cost 18.6 us against 1.4 us for a PFOR
window: 11.7 us to decode the plain payload, 4.8 us to decompress, and
about 2 us more because every window created and freed its own zstd
context (about 96 KB) and a zero-filled output vector. Two commits:

1. **Decode the plain payload without a Status per document.**
`decode_payload_csr_selective` made at least one out-of-line call
returning a Status for every document of the window, selected or not
(`get_varint32_fast`, `skip_varints`, `decode_delta_run`). The loop now
keeps the cursor in a register and handles the usual cases inline: a
single-byte count, walking past the deltas of a document that is not
selected, and a run of single-byte deltas. Anything else -- a multi-byte
value, a run that does not fit, a prefix sum that would overflow -- goes
to the checked decoders from the same cursor, so they decide and report
as before. The output vector grows as before; nothing is reserved up
front.
2. **Reuse one zstd context and buffer per index source.**
`ZstdDecompressor` keeps one context, created at the first zstd frame,
and one output buffer that only grows and is not zero-filled
(decompression writes every byte before it is read). The buffer comes
from the Doris allocator (`DorisUniqueBufferPtr`), so the decoding
task's memory tracker accounts for it while it is held.
`SniiIndexSource` owns one, and the cursors it opens pass it to the .prx
decoder through `PrxDecodeContext`. The cursors of a source decode on
its thread one window at a time, and the decoded payload is consumed
before the next window is read. A source lives for one query on one
segment, so a query holds at most one context and one buffer per segment
it reads, and only if it decodes a zstd window; before, each window
created and released them. Compaction decodes through an allocation
gate, which keeps the old path, as does a cursor given no decompressor.

Hot queries, old against new read path in one process (run-time switch
in an experiment build, 6 paired cycles), on 99M AgentLogs rows written
with the smallest frames, as every segment written before #69649 is
(mostly zstd windows): DS7 (unfiltered phrase) 0.87x, A6 0.89x, A10
0.93x, DS4 (phrase prefix) 0.94x, A9 0.95x, DS1 0.96x; repeating the old
path against itself gave 0.93-1.05. The gain follows the zstd windows a
query reads: the phrase queries of AgentLogs written with #69649 read
PFOR windows and do not change, and on TextBench, whose large windows
stay zstd, the time per window is the decompression itself and the
change is within noise. Every query returns the same result.

### Release note

None

… on faster .prx windows #69769 (#69796)

### What problem does this PR solve?

Issue Number: None

Related PR: #69769, #69649 (backport #69795), #69770 (backport #69788)

Problem Summary: Backport of #69769 to branch-4.2, stacked on #69795
(the backport of #69649). #69649 stores a .prx window as PFOR instead of
zstd when zstd saves less than `snii_prx_zstd_min_saving_percent` (30)
of the PFOR frame, because every read of a zstd window pays a
decompression. What that costs and buys depends on how much zstd saves
on the data. Replaying the 1.6M windows of 10M AgentLogs rows, 30 spends
8% more bytes on the windows zstd compresses and cuts their decode time
by 42%; on the windows of TextBench service logs it spends 5% and cuts
6%; on data whose windows zstd shrinks by just under 30% it stores every
window as PFOR. No single percentage fits all data.

Each index of a segment now spends a size budget instead:
`PrxZstdBudget`, owned by its `LogicalIndexWriter` and reached through
`PrxWindowLimits`, so the resident and the spilled encoder, and the
ordinary and the compaction build, all use it. A window whose smallest
frame is zstd is stored in its faster frame (PFOR, or raw when smaller)
when
- its saving is below the share whose trades, over the windows the index
has seen, fit 8% of their smallest frames, which puts the budget on the
windows zstd shrinks least, and
- the bytes the index has actually traded, this window included, still
fit 8% of the smallest frames seen. This makes 8% a bound for any order
of windows: fed the TextBench windows from the largest saving to the
smallest, the share alone spends 36%.

Until it has seen 64 windows the share is 30, the previous default. The
windows counted are those that get a zstd frame (a plain payload of at
least 512 bytes), the only ones whose codec a share decides; singleton
and short windows are written as before and counted neither in the
allowance nor in the warmup. The choice uses the frames the writer
builds anyway: nothing is compressed twice or scanned ahead, and the
same input gives the same bytes. `snii_prx_zstd_min_saving_percent` now
defaults to -1 (the budget); 0-100 keeps a fixed share, 0 the smallest
frame.

The budget is 8% rather than less: with it the AgentLogs phrase queries
below read nearly the same frames as with 30, while in the replay a 5%
budget cuts the decode time by 31% instead of 39%.

Same binary, one stream load per table, size against the smallest frames
(0):

| data | index, 30 | index, budget | table, 30 | table, budget |
|---|---|---|---|---|
| AgentLogs 99M rows | +4.13% | +3.65% | +1.92% | +1.69% |
| AgentLogs 10M rows | +4.59% | +3.37% | +2.13% | +1.57% |
| TextBench 12M rows | +0.89% | +0.79% | +0.51% | +0.45% |

- The phrase queries read the same frames with 30 and with the budget:
at 99M rows all PFOR for DS1, DS6, DS7, A6 and A9 (for example 107,307
frames for DS7), and about 11,170 zstd frames of about 259,000 for A10
with both; on TextBench most windows stay zstd either way (582 against
583 zstd frames of 599 for 'connection reset').
- Hot queries in one process, budget data against 30 data with the same
read path (paired cycles: 6 for the 25 AgentLogs queries at 10M rows, 10
for the 15 TextBench queries): 0.96-1.06 and 0.97-1.05; every query
returns the same result.
- Against the data written before #69649 (smallest frames), the 11
AgentLogs phrase queries at 99M rows take 0.83x in geometric mean with
the budget data (DS7 0.63x, A6 0.67x, A9 0.70x, DS4 0.74x, A10 0.77x),
as with 30 (0.83x). This run also had the read-path change of #69770,
which does not touch the PFOR windows these queries read.
- Data bytes are identical, and the index written with 0 and with 30 has
exactly the size the previous binary writes.
- Write cost: the choice adds about 48 ns per window, 79 ms over the
1.6M AgentLogs windows.

### Branch adaptations

None. `git cherry-pick 0571eb6` applied cleanly on top of the
backport of #69649, which it needs: it changes the setting #69649 adds,
and the automatic pick to branch-4.2 failed because branch-4.2 does not
have #69649. This PR is stacked on that backport, #69795, so it shows
both commits until #69795 merges; merge #69795 first.

### Validation

- Built and run with both backports (this one and the stacked #69769
backport) applied to branch-4.2 at 711db43, the commit before #69625:
its lance-c update is not in the local third-party build, and the 16
commits after it touch no SNII file. `PrxPod*`, `Snii*`,
`DorisSniiFileReader*`, `*InvertedIndex*`, `*IndexFileWriter*` and
`*IndexFileReader*`: 1710 tests passed, including the new budget, codec
choice, writer and configuration tests.
- On master both changes passed CI.
- Not built locally on the branch-4.2 tip; this PR's BE UT run covers
that.

### Release note

SNII now chooses per index how many position windows to store in the
faster but larger PFOR form, spending at most 8% of the size of the
windows zstd compresses; `snii_prx_zstd_min_saving_percent` defaults to
-1 for this, and 0-100 still sets a fixed percentage.

### Check List (For Author)

- Test: Unit Test
- SniiPrxZstdBudget: the prior share, the windows zstd shrinks least are
traded first, the bound under a descending order, windows with nothing
to trade.
- SniiPrxPodTest: the faster frame (PFOR or raw) and when the budget is
asked.
- SniiStreamedWriterSessionTest: windows that save 1-29% through the
ordinary and the streamed (compaction) build; 30 trades all of them, the
budget at most 8% of the smallest frames.
    - SniiWriterPrxZstdSaving: the default and the config mapping.
- Manual test: index size of 10M and 99M AgentLogs and 12M TextBench
loads with 0, 30 and the budget; frames read per codec from query
profiles; interleaved hot A/B in one process.
- Behavior changed: Yes (the default .prx codec choice of newly written
segments; existing segments are read as before)
- Does this need documentation: No

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: martin.foo838 <[email protected]>
Co-authored-by: Claude Opus 5.5 <[email protected]>

…little over PFOR

### What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary: The auto .prx codec stores each position window in the smallest of PFOR, zstd and
raw. Every read of a zstd window pays a decompression that PFOR does not, and the choice ignores
that cost. On 99M AgentLogs rows 74% of the position windows read by an unfiltered phrase query
were zstd, and decoding them took 58% of the index execution time of the query.

PrxWindowLimits gets zstd_min_saving_percent, set from the new mutable config
snii_prx_zstd_min_saving_percent. A readable PFOR frame is replaced by the zstd one only if zstd is
smaller by at least that percentage of the PFOR frame. The default is 30; 0 picks the smaller frame
as before and writes a byte-identical index (99M rows: 61,886,854,661 bytes with and without this
change). Readers handle every codec, so the setting affects only segments written afterwards, and
compaction rewrites with whatever is set then. The compression is still computed, so the write cost
does not change.

Effect of the setting, 10M rows (index size, DS7 = unfiltered phrase on the output column, position
decode time from the query profile): 10 -> +0.02% size, no change for DS7; 20 -> +0.5%, no change;
30 -> +2.1%, decode 252 ms -> 85 ms; 50 -> +6.3%, no further gain; 100 (never zstd) -> +8.5%.
30 is the smallest value that removes the zstd windows of frequent terms, which zstd beats PFOR by
20-30% on and which phrase queries read most. On 99M rows with 30: index +1.9% (63.07 GB against
61.89 GB, still below the 65.70 GB of V3), load time within 2% of a control load, hot queries in the
same process 0.63x (unfiltered phrase), 0.67x (phrase in input or output), 0.75x (phrase prefix),
0.77x, 0.87x, the others within 1.02x (one query measured 1.06x once and 1.00-1.05x in three
repeats); cold runs 10-18% faster for the phrase queries. The best value depends on the data, since
it is how much zstd beats PFOR on the window contents; it was calibrated on one corpus.

### Release note

SNII position windows are stored with zstd only where it saves at least 30% over PFOR
(snii_prx_zstd_min_saving_percent): phrase queries are faster and the index is about 2% larger.
Set it to 0 for the previous behavior.

### Check List (For Author)

- Test: Unit Test on master, 1723 tests pass with the inline-read change stacked (SniiPrxPodTest: the saving boundary, PFOR unreadable, limit validation; SniiWriterGoldenBytes: the config reaches the writer; the default is 30). Manual test: index size and load time at 10M and 99M rows, interleaved hot A/B and repeated cold runs.
- Behavior changed: Yes (newly written segments are about 2% larger; set snii_prx_zstd_min_saving_percent to 0 to restore)
- Does this need documentation: No

Source-PR: apache#69649
Source-Head: 428c241

Co-Authored-By: Claude Opus 5.5 <[email protected]>