Repository navigation
Commit 76b988c
feat(array): builders no longer canonicalize their children
Nested builders used to push every appended child array through
`append_to_builder`, which decoded it into the child's canonical builder.
That work is wasted: `Canonical` only promises a canonical *top level*, so
struct fields, list elements and extension storage are free to stay
compressed.
Introduce `ChildBuilder`, which accumulates a child as a `Vec<ArrayRef>` of
chunks plus a scalar builder for the values that cannot come from an array,
and stitches them into a `ChunkedArray` on `finish` when more than one chunk
accumulated. `StructBuilder`, `ListBuilder`, `ListViewBuilder`,
`FixedSizeListBuilder` and `ExtensionBuilder` now hold their children this
way.
However short the appended array, it becomes a chunk. Deciding on the caller's
behalf that its values are cheaper copied than referenced would be guessing at
a boundary only the caller can see, and a caller that wants them copied has
`append_scalar`. A child is therefore chunked on exactly the boundaries it was
appended on.
Signed-off-by: Claude <[email protected]>
Signed-off-by: Robert Kruszewski <[email protected]>
Signed-off-by: Robert Kruszewski <[email protected]>
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>1 parent e570927 commit 76b988c
10 files changed
Lines changed: 994 additions & 49 deletions
File tree
- encodings/sparse/src
- vortex-array/src
- aggregate_fn/fns/uncompressed_size_in_bytes
- builders
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
586 | 586 | | |
587 | 587 | | |
588 | 588 | | |
| 589 | + | |
589 | 590 | | |
590 | 591 | | |
591 | 592 | | |
| |||
594 | 595 | | |
595 | 596 | | |
596 | 597 | | |
| 598 | + | |
597 | 599 | | |
598 | 600 | | |
599 | 601 | | |
| |||
1339 | 1341 | | |
1340 | 1342 | | |
1341 | 1343 | | |
| 1344 | + | |
| 1345 | + | |
| 1346 | + | |
| 1347 | + | |
| 1348 | + | |
| 1349 | + | |
| 1350 | + | |
| 1351 | + | |
| 1352 | + | |
| 1353 | + | |
| 1354 | + | |
| 1355 | + | |
| 1356 | + | |
| 1357 | + | |
| 1358 | + | |
| 1359 | + | |
| 1360 | + | |
| 1361 | + | |
| 1362 | + | |
| 1363 | + | |
| 1364 | + | |
| 1365 | + | |
| 1366 | + | |
| 1367 | + | |
| 1368 | + | |
| 1369 | + | |
| 1370 | + | |
| 1371 | + | |
| 1372 | + | |
| 1373 | + | |
| 1374 | + | |
| 1375 | + | |
| 1376 | + | |
| 1377 | + | |
| 1378 | + | |
| 1379 | + | |
| 1380 | + | |
| 1381 | + | |
| 1382 | + | |
| 1383 | + | |
| 1384 | + | |
| 1385 | + | |
| 1386 | + | |
| 1387 | + | |
1342 | 1388 | | |
1343 | 1389 | | |
1344 | 1390 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
343 | 343 | | |
344 | 344 | | |
345 | 345 | | |
| 346 | + | |
346 | 347 | | |
347 | 348 | | |
348 | 349 | | |
| |||
382 | 383 | | |
383 | 384 | | |
384 | 385 | | |
| 386 | + | |
385 | 387 | | |
386 | 388 | | |
387 | | - | |
388 | | - | |
389 | | - | |
390 | | - | |
| 389 | + | |
391 | 390 | | |
392 | | - | |
| 391 | + | |
| 392 | + | |
| 393 | + | |
| 394 | + | |
| 395 | + | |
| 396 | + | |
| 397 | + | |
| 398 | + | |
| 399 | + | |
| 400 | + | |
393 | 401 | | |
394 | 402 | | |
395 | 403 | | |
| |||
0 commit comments