Repository navigation
Commit b2d08cc
authored
perf(scan): merge narrow splits at boundaries only some fields of a struct have (#11128)
## Summary
- Tracking Issue: #11123
**Problem:** A scan splits a file wherever any column's chunk ends, and
a split decodes the whole chunk of every column it overlaps. When a
struct's fields chunk at unrelated rows, as list elements and
dictionary-encoded columns of a wide nested file do, the union of their
boundaries leaves spans of a few dozen rows. Successive splits then each
decode the same chunks of the other fields. On a 180 MB list-layout
file, a full scan held 3.6 GiB of native memory.
**Fix:** The struct reader marks boundaries that only some of its fields
have as *soft*. `SplitBy::LayoutSubSplitting` merges spans that end at a
soft boundary until they reach 8,192 rows. Boundaries that every field
has are never merged, and neither are boundaries from layouts above the
struct, so files whose columns chunk alike split as before.
**Result:** full scan of one file, with Arrow conversion as JNI does it.
Live native bytes come from a counting allocator:
| | Splits | Peak live | Time | Bytes requested |
|---|---|---|---|---|
| List layout, 180 MB, before | 139 (smallest 70 rows) | 3.57 GiB | 1.7
s | 6.2 GB |
| List layout, 180 MB, after | 60 (smallest 8,192 rows) | **1.10 GiB** |
**0.6 s** | 0.99 GB |
| Default layout, 111 MB, before | 59 | 224 MiB | 0.2 s | |
| Default layout, 111 MB, after | 48 | 224 MiB | 0.2 s | |
## Changes
- **`RowSplits` gains soft marks.** Each struct field registers into its
own `RowSplits`, and the struct takes their union. Any boundary that not
every field has is marked soft.
- The shared-boundary intersection is computed once. On the existing
256-column split-collection benchmark, the warm median falls from about
338 ms to 3 ms.
- The struct's validity child is classified like any field, so a
nullable struct merges the same way at any depth.
- Soft marks inherited from nested structs are kept.
- A single-field struct without validity registers exactly as before.
- `SplitBy::Layout` collects positions only
(`BoundaryMode::PositionsOnly`), so it skips the classification.
- **`SplitBy::LayoutSubSplitting`** merges spans ending at a soft
boundary until they reach `DEFAULT_MIN_SPLIT_ROWS` (8,192, the writer's
row block), or `max_rows` if that is smaller. It never merges a span
past `max_rows`. The merge runs before the existing sub-division of wide
spans.
---------
Signed-off-by: Christopher Wirt <[email protected]>1 parent d455777 commit b2d08cc
4 files changed
Lines changed: 668 additions & 18 deletions
File tree
- vortex-file/benches
- vortex-layout/src
- layouts/struct_
- scan
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | 3 | | |
4 | | - | |
| 4 | + | |
| 5 | + | |
5 | 6 | | |
6 | 7 | | |
7 | 8 | | |
| |||
55 | 56 | | |
56 | 57 | | |
57 | 58 | | |
58 | | - | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
59 | 68 | | |
60 | 69 | | |
61 | 70 | | |
| |||
194 | 203 | | |
195 | 204 | | |
196 | 205 | | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
197 | 210 | | |
198 | | - | |
| 211 | + | |
199 | 212 | | |
200 | 213 | | |
201 | 214 | | |
| |||
248 | 261 | | |
249 | 262 | | |
250 | 263 | | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
0 commit comments