Repository navigation
Commit 9ca7f96
[improvement](be) Reuse one zstd context and buffer per SNII index source
### What problem does this PR solve?
Issue Number: None
Related PR: #69649
Problem Summary: Every zstd .prx window a query reads was decompressed with ZSTD_decompress, which
creates and frees a decompression context (about 96 KB) per call, into a zero-filled vector that is
also allocated per call. Timed per window on a 10M-row AgentLogs load, the decompression took
6.8 us with a context per call and 4.8 us with a reused one.
ZstdDecompressor keeps one context, created at the first frame, and one output buffer that only
grows and is not zero-filled (decompression writes every byte before it is read). SniiIndexSource
owns one, and the cursors it opens pass it to the .prx decoder through PrxDecodeContext; the
cursors of a source decode on its thread one window at a time, and the decoded plain payload is
consumed before the next window is read. A source lives for one query on one segment, so a query
holds at most one context and one buffer per segment it reads, created only if it decodes a zstd
window; before, each window created and released them. Compaction decodes through an allocation
gate, which keeps the old path, as does a cursor given no decompressor.
Hot phrase queries on data whose position windows are mostly zstd (10M rows, interleaved A/B in one
process, on top of the plain-decode change): 0.95-0.96x for the two most position-heavy queries,
the others within noise.
### Release note
None
### Check List (For Author)
- Test: Unit Test (SniiZstdDecompressor: frames of different sizes through one decompressor,
corruption reported and the next frame decoded; PrxPodTest: random windows decoded through one
shared decompressor)
- Behavior changed: No
- Does this need documentation: No
Co-Authored-By: Claude Opus 5.5 <[email protected]>1 parent 299715b commit 9ca7f96
10 files changed
Lines changed: 146 additions & 8 deletions
File tree
- be
- src/storage/index/inverted/snii
- encoding
- format
- reader
- test/storage/index/inverted/snii
- encoding
- format
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
49 | 49 | | |
50 | 50 | | |
51 | 51 | | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
52 | 84 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
19 | 19 | | |
20 | 20 | | |
21 | 21 | | |
| 22 | + | |
22 | 23 | | |
23 | 24 | | |
24 | 25 | | |
| |||
30 | 31 | | |
31 | 32 | | |
32 | 33 | | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
33 | 53 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
23 | 23 | | |
24 | 24 | | |
25 | 25 | | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
26 | 30 | | |
27 | 31 | | |
28 | 32 | | |
| |||
73 | 77 | | |
74 | 78 | | |
75 | 79 | | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
76 | 83 | | |
77 | 84 | | |
78 | 85 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1110 | 1110 | | |
1111 | 1111 | | |
1112 | 1112 | | |
1113 | | - | |
| 1113 | + | |
1114 | 1114 | | |
1115 | 1115 | | |
1116 | 1116 | | |
| |||
1125 | 1125 | | |
1126 | 1126 | | |
1127 | 1127 | | |
1128 | | - | |
| 1128 | + | |
| 1129 | + | |
| 1130 | + | |
1129 | 1131 | | |
1130 | 1132 | | |
1131 | 1133 | | |
| |||
1200 | 1202 | | |
1201 | 1203 | | |
1202 | 1204 | | |
1203 | | - | |
| 1205 | + | |
| 1206 | + | |
1204 | 1207 | | |
1205 | 1208 | | |
1206 | 1209 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
124 | 124 | | |
125 | 125 | | |
126 | 126 | | |
127 | | - | |
| 127 | + | |
128 | 128 | | |
129 | 129 | | |
130 | 130 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
28 | 28 | | |
29 | 29 | | |
30 | 30 | | |
| 31 | + | |
31 | 32 | | |
32 | 33 | | |
33 | 34 | | |
| |||
89 | 90 | | |
90 | 91 | | |
91 | 92 | | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
92 | 96 | | |
93 | 97 | | |
94 | 98 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
98 | 98 | | |
99 | 99 | | |
100 | 100 | | |
101 | | - | |
| 101 | + | |
| 102 | + | |
102 | 103 | | |
103 | 104 | | |
104 | 105 | | |
| |||
108 | 109 | | |
109 | 110 | | |
110 | 111 | | |
| 112 | + | |
111 | 113 | | |
112 | 114 | | |
113 | 115 | | |
| |||
533 | 535 | | |
534 | 536 | | |
535 | 537 | | |
536 | | - | |
| 538 | + | |
537 | 539 | | |
538 | 540 | | |
539 | 541 | | |
| |||
795 | 797 | | |
796 | 798 | | |
797 | 799 | | |
798 | | - | |
| 800 | + | |
799 | 801 | | |
800 | 802 | | |
801 | 803 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
85 | 85 | | |
86 | 86 | | |
87 | 87 | | |
88 | | - | |
| 88 | + | |
| 89 | + | |
89 | 90 | | |
90 | 91 | | |
91 | 92 | | |
| |||
187 | 188 | | |
188 | 189 | | |
189 | 190 | | |
| 191 | + | |
| 192 | + | |
190 | 193 | | |
191 | 194 | | |
192 | 195 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
19 | 19 | | |
20 | 20 | | |
21 | 21 | | |
| 22 | + | |
22 | 23 | | |
23 | 24 | | |
24 | 25 | | |
| |||
49 | 50 | | |
50 | 51 | | |
51 | 52 | | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
551 | 551 | | |
552 | 552 | | |
553 | 553 | | |
| 554 | + | |
| 555 | + | |
554 | 556 | | |
555 | 557 | | |
556 | 558 | | |
| |||
597 | 599 | | |
598 | 600 | | |
599 | 601 | | |
| 602 | + | |
| 603 | + | |
| 604 | + | |
| 605 | + | |
| 606 | + | |
| 607 | + | |
| 608 | + | |
| 609 | + | |
| 610 | + | |
| 611 | + | |
| 612 | + | |
| 613 | + | |
| 614 | + | |
| 615 | + | |
| 616 | + | |
| 617 | + | |
600 | 618 | | |
601 | 619 | | |
602 | 620 | | |
| |||
0 commit comments