Repository navigation
Commit 00130bd
perf(scan): merge narrow splits at boundaries only some fields of a struct have (#67)
Fixes #61.
## Problem
- A scan splits a file wherever any column's chunk ends, and a split
decodes the whole chunk of every column it overlaps.
- When a struct's fields chunk at unrelated rows, as list elements and
dictionary-encoded columns of a wide nested file do, the union of their
boundaries holds spans of a few dozen rows.
- Successive splits then each decode the same chunks of the other
fields. On a 180 MB list-layout file, a full scan held 3.6 GiB of native
memory, and one 176-row output array referenced 42 MB.
## Change
- **The struct reader marks soft boundaries.** These are boundaries that
only some of its fields have. Each field registers into its own
`RowSplits`, and the struct takes their union, marking any boundary not
every field has as soft. A single-field struct registers exactly as
before.
- **`SplitBy::LayoutSubSplitting` merges spans ending at a soft
boundary** until they reach `DEFAULT_MIN_SPLIT_ROWS` (8,192, the
writer's row block), or `max_rows` if that is smaller. It runs before
the existing sub-division of wide spans.
- **Hard boundaries are never merged.** These are boundaries every field
has, and those registered by layouts above the struct, such as top-level
chunk ends. Files whose columns chunk alike split as before.
## Effect
Full scan of one file, with Arrow conversion as JNI does it. Live native
bytes are measured with a counting allocator:
| | Splits | Peak live | Time | Bytes requested |
|---|---|---|---|---|
| List layout, 180 MB, before | 139 (smallest 70 rows) | 3.57 GiB | 1.7
s | 6.2 GB |
| List layout, 180 MB, after | 60 (smallest 8,192 rows) | **1.10 GiB** |
**0.6 s** | 0.99 GB |
| Default layout, 111 MB, before | 59 | 224 MiB | 0.2 s | |
| Default layout, 111 MB, after | 48 | 224 MiB | 0.2 s | |
## Tests
- **Struct reader** (real layouts written with `RepartitionStrategy`):
- fields chunked every 1,000 and 3,000 rows mark only the unshared
boundaries soft;
- the same struct under a top-level chunked layout keeps the chunk end
hard;
- `LayoutSubSplitting` merges soft boundaries and preserves shared and
validity boundaries, for nullable and non-nullable structs.
- **`merge_small_spans`:** unit cases plus 1,000 generated cases
checking endpoints, hard boundaries, exact coverage, minimum spans at
retained soft boundaries, and subsequent subdivision.
- The nullable integration regression failed before the
validity-boundary fix.
- **Checks:** 836 `vortex-layout` library tests; all-target, all-feature
Clippy for `vortex-layout` and `vortex-file` with `-D warnings`; pinned
Rust formatting; Java Spotless; full `./gradlew build --no-parallel
--max-workers=1` including JNI and Spark checks; patch whitespace. All
passed. The full Rust workspace suite was not rerun locally; Linux musl
CI passed the workspace tests.
## Review follow-up
- Compute the shared-boundary intersection once instead of searching
every field for every boundary. The existing 256-column split-collection
debug benchmark's warm median fell from about 338 ms to 3 ms.
- Preserve inherited soft marks while keeping this struct's validity
boundaries hard, including where they overlap a soft data-field
boundary.
- Move soft marks when appending split sets instead of cloning their
buffer.
## Not done here
- The minimum is a row count. Rows with many list elements are larger
than rows with few, so a byte-based minimum would be more precise.
- An earlier variant measured less memory at 16,384 rows; the minimum
isn't tuned beyond 8,192.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: Christopher Wirt <[email protected]>
Co-authored-by: Claude Opus 5.5 <[email protected]>1 parent 1aa2ad7 commit 00130bd
4 files changed
Lines changed: 368 additions & 12 deletions
File tree
- vortex-file/benches
- vortex-layout/src
- layouts/struct_
- scan
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
55 | 55 | | |
56 | 56 | | |
57 | 57 | | |
58 | | - | |
| 58 | + | |
59 | 59 | | |
60 | 60 | | |
61 | 61 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
523 | 523 | | |
524 | 524 | | |
525 | 525 | | |
526 | | - | |
527 | | - | |
528 | | - | |
529 | | - | |
| 526 | + | |
| 527 | + | |
| 528 | + | |
| 529 | + | |
| 530 | + | |
| 531 | + | |
| 532 | + | |
| 533 | + | |
| 534 | + | |
530 | 535 | | |
| 536 | + | |
| 537 | + | |
531 | 538 | | |
| 539 | + | |
532 | 540 | | |
533 | | - | |
| 541 | + | |
| 542 | + | |
| 543 | + | |
534 | 544 | | |
| 545 | + | |
| 546 | + | |
| 547 | + | |
| 548 | + | |
| 549 | + | |
| 550 | + | |
| 551 | + | |
| 552 | + | |
| 553 | + | |
| 554 | + | |
| 555 | + | |
| 556 | + | |
| 557 | + | |
| 558 | + | |
| 559 | + | |
| 560 | + | |
| 561 | + | |
| 562 | + | |
| 563 | + | |
| 564 | + | |
| 565 | + | |
| 566 | + | |
| 567 | + | |
| 568 | + | |
| 569 | + | |
| 570 | + | |
| 571 | + | |
| 572 | + | |
| 573 | + | |
535 | 574 | | |
536 | 575 | | |
537 | 576 | | |
| |||
679 | 718 | | |
680 | 719 | | |
681 | 720 | | |
| 721 | + | |
682 | 722 | | |
| 723 | + | |
683 | 724 | | |
684 | 725 | | |
685 | 726 | | |
| |||
712 | 753 | | |
713 | 754 | | |
714 | 755 | | |
| 756 | + | |
| 757 | + | |
715 | 758 | | |
716 | 759 | | |
| 760 | + | |
| 761 | + | |
717 | 762 | | |
718 | 763 | | |
719 | 764 | | |
| 765 | + | |
720 | 766 | | |
721 | 767 | | |
722 | 768 | | |
| |||
803 | 849 | | |
804 | 850 | | |
805 | 851 | | |
| 852 | + | |
| 853 | + | |
| 854 | + | |
| 855 | + | |
| 856 | + | |
| 857 | + | |
| 858 | + | |
| 859 | + | |
| 860 | + | |
| 861 | + | |
| 862 | + | |
| 863 | + | |
| 864 | + | |
| 865 | + | |
| 866 | + | |
| 867 | + | |
| 868 | + | |
| 869 | + | |
| 870 | + | |
| 871 | + | |
| 872 | + | |
| 873 | + | |
| 874 | + | |
| 875 | + | |
| 876 | + | |
| 877 | + | |
| 878 | + | |
| 879 | + | |
| 880 | + | |
| 881 | + | |
| 882 | + | |
| 883 | + | |
| 884 | + | |
| 885 | + | |
| 886 | + | |
| 887 | + | |
| 888 | + | |
| 889 | + | |
| 890 | + | |
| 891 | + | |
| 892 | + | |
| 893 | + | |
| 894 | + | |
| 895 | + | |
| 896 | + | |
| 897 | + | |
| 898 | + | |
| 899 | + | |
| 900 | + | |
| 901 | + | |
| 902 | + | |
| 903 | + | |
| 904 | + | |
| 905 | + | |
| 906 | + | |
| 907 | + | |
| 908 | + | |
| 909 | + | |
| 910 | + | |
| 911 | + | |
| 912 | + | |
| 913 | + | |
| 914 | + | |
| 915 | + | |
| 916 | + | |
| 917 | + | |
| 918 | + | |
| 919 | + | |
| 920 | + | |
| 921 | + | |
| 922 | + | |
| 923 | + | |
| 924 | + | |
| 925 | + | |
| 926 | + | |
| 927 | + | |
| 928 | + | |
| 929 | + | |
| 930 | + | |
| 931 | + | |
| 932 | + | |
| 933 | + | |
| 934 | + | |
| 935 | + | |
| 936 | + | |
| 937 | + | |
| 938 | + | |
| 939 | + | |
| 940 | + | |
| 941 | + | |
| 942 | + | |
| 943 | + | |
| 944 | + | |
| 945 | + | |
| 946 | + | |
| 947 | + | |
| 948 | + | |
| 949 | + | |
| 950 | + | |
| 951 | + | |
| 952 | + | |
| 953 | + | |
| 954 | + | |
| 955 | + | |
| 956 | + | |
| 957 | + | |
| 958 | + | |
| 959 | + | |
| 960 | + | |
| 961 | + | |
| 962 | + | |
| 963 | + | |
| 964 | + | |
| 965 | + | |
| 966 | + | |
| 967 | + | |
| 968 | + | |
| 969 | + | |
| 970 | + | |
| 971 | + | |
| 972 | + | |
| 973 | + | |
| 974 | + | |
| 975 | + | |
| 976 | + | |
| 977 | + | |
| 978 | + | |
| 979 | + | |
| 980 | + | |
| 981 | + | |
| 982 | + | |
| 983 | + | |
| 984 | + | |
| 985 | + | |
| 986 | + | |
| 987 | + | |
| 988 | + | |
| 989 | + | |
| 990 | + | |
| 991 | + | |
| 992 | + | |
| 993 | + | |
806 | 994 | | |
807 | 995 | | |
808 | 996 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
100 | 100 | | |
101 | 101 | | |
102 | 102 | | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
103 | 107 | | |
104 | 108 | | |
105 | 109 | | |
106 | 110 | | |
107 | 111 | | |
108 | 112 | | |
| 113 | + | |
| 114 | + | |
109 | 115 | | |
110 | 116 | | |
111 | 117 | | |
| |||
162 | 168 | | |
163 | 169 | | |
164 | 170 | | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
165 | 192 | | |
166 | 193 | | |
167 | 194 | | |
168 | 195 | | |
169 | 196 | | |
170 | 197 | | |
| 198 | + | |
171 | 199 | | |
172 | 200 | | |
173 | 201 | | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
174 | 210 | | |
175 | 211 | | |
176 | 212 | | |
| |||
0 commit comments