Repository navigation
Commit a400b5c
committed
[fix](variant) Stream the VARIANT root column instead of buffering a segment (CIR-22037)
ColumnStr<uint32_t> rejects more than 4 GiB of uncompressed chars (E-3113).
The VARIANT V2 writer kept the whole output segment's root JSONB in one
ColumnString before finalize, so a vertical compaction whose segment carried
more than 4 GiB of uncompressed variant data failed deterministically. The
compressed-byte segment budget cannot prevent this for high-compression-ratio
data.
Stream the root column: create the root column writer at init and append every
input batch to it, so the writer never holds a segment-sized ColumnString.
Keep a data-driven segment budget as a guard for the remaining variant buffers:
sum the raw (pre-encoding) data bytes of the input segments' VARIANT column
metas and clamp the row budget with
variant_compaction_max_segment_uncompressed_bytes (2 GiB by default). Segments
written before variant writers reported data bytes keep the compressed-size
budget.1 parent 211a79a commit a400b5c
2 files changed
Lines changed: 36 additions & 14 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
109 | 109 | | |
110 | 110 | | |
111 | 111 | | |
112 | | - | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
113 | 118 | | |
114 | 119 | | |
115 | 120 | | |
| |||
129 | 134 | | |
130 | 135 | | |
131 | 136 | | |
132 | | - | |
| 137 | + | |
133 | 138 | | |
134 | 139 | | |
135 | 140 | | |
136 | 141 | | |
137 | 142 | | |
138 | 143 | | |
139 | | - | |
140 | | - | |
| 144 | + | |
141 | 145 | | |
142 | 146 | | |
143 | 147 | | |
| |||
152 | 156 | | |
153 | 157 | | |
154 | 158 | | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
155 | 178 | | |
156 | 179 | | |
157 | 180 | | |
| |||
231 | 254 | | |
232 | 255 | | |
233 | 256 | | |
234 | | - | |
235 | | - | |
236 | | - | |
237 | | - | |
| 257 | + | |
| 258 | + | |
238 | 259 | | |
239 | 260 | | |
240 | 261 | | |
| |||
343 | 364 | | |
344 | 365 | | |
345 | 366 | | |
346 | | - | |
347 | | - | |
| 367 | + | |
| 368 | + | |
348 | 369 | | |
349 | 370 | | |
350 | 371 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
38 | 38 | | |
39 | 39 | | |
40 | 40 | | |
41 | | - | |
42 | | - | |
43 | | - | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
44 | 44 | | |
45 | 45 | | |
46 | 46 | | |
| |||
65 | 65 | | |
66 | 66 | | |
67 | 67 | | |
| 68 | + | |
| 69 | + | |
68 | 70 | | |
69 | 71 | | |
70 | 72 | | |
| |||
74 | 76 | | |
75 | 77 | | |
76 | 78 | | |
77 | | - | |
78 | 79 | | |
79 | 80 | | |
80 | 81 | | |
| |||
0 commit comments