Repository navigation
Commit 7c13a60
authored
perf: zstd implement Pooling + DecodeAll with no new reader. (#1331)
### Rationale for this change
Performance improvement on the zstd path.
### What changes are included in this PR?
- zstd: decoders come from a `sync.Pool` instead of being created for
every record batch, and each compressed buffer is decoded with one
`DecodeAll` call straight into the destination buffer (decoder built
with `WithDecoderConcurrency(1)` and `WithDecodeAllCapLimit(true))`
instead of `Reset(bytes.NewReader)` + `io.ReadFull` on a streaming
decoder.
- lz4 : no functional change. It keeps its original per-batch
`lz4.Reader`; the library already pools its large block buffers
internally. (Only a Decompress method wrapper was added.)
### Are these changes tested?
Added tests + benchmark for zstd (the changed path) - every case
improves
```
SMALL buffers (4 KiB per column)
time : -70.5% to -78.3% in all 8 cases
1 col, 1 batch (stream) 9.89 us -> 2.92 us
1 col, 64 batches (stream) 592.9 us -> 128.7 us
16 cols, 64 batches (file) 3.67 ms -> 894 us
B/op : -60% (1 col, 1 batch), -64% (1 col, 64 batches), -22% (16 cols)
allocs/op : -45% (1 col, 1 batch) to -70% (16 cols, 64 batches)
LARGE buffers
per column time B/op allocs/op
16 KiB 77.6 us -> 24.0 us (-69%) 306.9 KiB -> 157.1 KiB (-49%) 184 -> 78 (-58%)
256 KiB 796.7 us -> 191.7 us (-76%) 20.6 MiB -> 2.05 MiB (-90%) 185 -> 81 (-56%)
4 MiB 4.63 ms -> 3.05 ms (-34%) 51.1 MiB -> 32.1 MiB (-37%) 189 -> 79 (-58%)
```
### Are there any user-facing changes?
Strictness change for zstd. A buffer whose decompressed size differs
from the uncompressed-length prefix now returns an error. This matches
other arrow libraries (e.g.:
[arrow-cpp](https://github.com/apache/arrow/blob/f3b46fb7d4b8f5fb3fc459860038f0ed8b02bba5/cpp/src/arrow/ipc/reader.cc#L571-L575))1 parent b02cc01 commit 7c13a60
4 files changed
Lines changed: 515 additions & 14 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
17 | 17 | | |
18 | 18 | | |
19 | 19 | | |
| 20 | + | |
| 21 | + | |
20 | 22 | | |
| 23 | + | |
21 | 24 | | |
22 | 25 | | |
23 | 26 | | |
| |||
81 | 84 | | |
82 | 85 | | |
83 | 86 | | |
84 | | - | |
85 | | - | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
86 | 93 | | |
87 | 94 | | |
88 | 95 | | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
89 | 113 | | |
90 | 114 | | |
91 | 115 | | |
92 | 116 | | |
93 | | - | |
94 | | - | |
95 | | - | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
96 | 120 | | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
97 | 133 | | |
98 | 134 | | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
99 | 139 | | |
100 | | - | |
| 140 | + | |
101 | 141 | | |
102 | 142 | | |
103 | 143 | | |
104 | 144 | | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
105 | 168 | | |
106 | 169 | | |
107 | 170 | | |
| |||
111 | 174 | | |
112 | 175 | | |
113 | 176 | | |
114 | | - | |
| 177 | + | |
115 | 178 | | |
116 | | - | |
117 | | - | |
118 | | - | |
119 | | - | |
120 | | - | |
| 179 | + | |
121 | 180 | | |
122 | 181 | | |
123 | 182 | | |
| |||
0 commit comments