Repository navigation
Commit 769938b
authored
fix(jni): cache each open file's segments and merge reads across small gaps only (#66)
## Summary
Closes #65. Spark reads of list-layout files read several times the
bytes they need, because the JNI data
source opens files without a segment cache and a list column's element
segments are requested once per row range
of a scan. On a 47 MB list-layout file of nested event data, a 29-column
read now reads 13.7 MB instead of 39.5 MB,
and 15.7 MB instead of 68.8 MB when Spark splits the file in four.
Before this change, the same read of the file
written without the list layout read 30.0 MB, so the list layout now
pays off instead of costing more.
## Changes
- Each file the JNI data source opens gets its own segment cache, capped
at 64 MiB and dropped with the open file.
Both `NativeDataSource.open` and `openFiles` use it.
- `JavaReadable` merges reads across gaps of at most 64 KiB instead of 1
MiB. With a cache, a merged read that
spans a segment the cache already holds reads it again; at 1 MiB that
left list-layout reads at 1.3x to 1.8x
the bytes they need, at 64 KiB 1.0x to 1.2x.
## Testing
- `writer::tests::list_layout_scan_reads_each_segment_once`: writes a
list-of-struct column with the list layout,
scans it as four row ranges through the data source, and checks that no
segment is read twice with the JNI open
options. It also checks that without them some segments are, so the test
keeps reproducing the problem.
- `cargo test -p vortex-jni`, `cargo clippy -p vortex-jni
--all-targets`.
- Spark, local mode, the connector on develop with this native library,
same file, bytes the JVM read for the
29-column read: list layout 21.5 to 25.0 MB before, 14.5 MB after; split
in four, 77 to 92 MB before, 48.4 MB
after. Without the list layout: 30.6 MB before, 29.5 MB after. Row
counts match.
## Not fixed here
Spark reads each split of a file in its own task, so splits can't share
a cache, and an elements segment that
spans the whole file is read once per split. Reading a list-layout file
in one split avoids that
(`vortex.splitFiles=false`); writing list elements in chunks that line
up with the outer row chunks would fix it
for split reads too.
Signed-off-by: gitRasheed <[email protected]>1 parent abb4830 commit 769938b
3 files changed
Lines changed: 155 additions & 5 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
24 | 24 | | |
25 | 25 | | |
26 | 26 | | |
| 27 | + | |
27 | 28 | | |
28 | 29 | | |
29 | 30 | | |
30 | 31 | | |
31 | 32 | | |
| 33 | + | |
| 34 | + | |
32 | 35 | | |
33 | 36 | | |
34 | 37 | | |
| |||
39 | 42 | | |
40 | 43 | | |
41 | 44 | | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
42 | 64 | | |
43 | 65 | | |
44 | 66 | | |
| |||
93 | 115 | | |
94 | 116 | | |
95 | 117 | | |
96 | | - | |
| 118 | + | |
97 | 119 | | |
98 | 120 | | |
99 | 121 | | |
| |||
165 | 187 | | |
166 | 188 | | |
167 | 189 | | |
168 | | - | |
| 190 | + | |
169 | 191 | | |
170 | 192 | | |
171 | 193 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
91 | 91 | | |
92 | 92 | | |
93 | 93 | | |
94 | | - | |
95 | | - | |
96 | | - | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
97 | 99 | | |
98 | 100 | | |
99 | 101 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
703 | 703 | | |
704 | 704 | | |
705 | 705 | | |
| 706 | + | |
| 707 | + | |
| 708 | + | |
| 709 | + | |
| 710 | + | |
| 711 | + | |
| 712 | + | |
| 713 | + | |
| 714 | + | |
| 715 | + | |
| 716 | + | |
| 717 | + | |
| 718 | + | |
706 | 719 | | |
707 | 720 | | |
708 | 721 | | |
| 722 | + | |
709 | 723 | | |
710 | 724 | | |
711 | 725 | | |
| |||
824 | 838 | | |
825 | 839 | | |
826 | 840 | | |
| 841 | + | |
| 842 | + | |
| 843 | + | |
| 844 | + | |
| 845 | + | |
| 846 | + | |
| 847 | + | |
| 848 | + | |
| 849 | + | |
| 850 | + | |
| 851 | + | |
| 852 | + | |
| 853 | + | |
| 854 | + | |
| 855 | + | |
| 856 | + | |
| 857 | + | |
| 858 | + | |
| 859 | + | |
| 860 | + | |
| 861 | + | |
| 862 | + | |
| 863 | + | |
| 864 | + | |
| 865 | + | |
| 866 | + | |
| 867 | + | |
| 868 | + | |
| 869 | + | |
| 870 | + | |
| 871 | + | |
| 872 | + | |
| 873 | + | |
| 874 | + | |
| 875 | + | |
| 876 | + | |
| 877 | + | |
| 878 | + | |
| 879 | + | |
| 880 | + | |
| 881 | + | |
| 882 | + | |
| 883 | + | |
| 884 | + | |
| 885 | + | |
| 886 | + | |
| 887 | + | |
| 888 | + | |
| 889 | + | |
| 890 | + | |
| 891 | + | |
| 892 | + | |
| 893 | + | |
| 894 | + | |
| 895 | + | |
| 896 | + | |
| 897 | + | |
| 898 | + | |
| 899 | + | |
| 900 | + | |
| 901 | + | |
| 902 | + | |
| 903 | + | |
| 904 | + | |
| 905 | + | |
| 906 | + | |
| 907 | + | |
| 908 | + | |
| 909 | + | |
| 910 | + | |
| 911 | + | |
| 912 | + | |
| 913 | + | |
| 914 | + | |
| 915 | + | |
| 916 | + | |
| 917 | + | |
| 918 | + | |
| 919 | + | |
| 920 | + | |
| 921 | + | |
| 922 | + | |
| 923 | + | |
| 924 | + | |
| 925 | + | |
| 926 | + | |
| 927 | + | |
| 928 | + | |
| 929 | + | |
| 930 | + | |
| 931 | + | |
| 932 | + | |
| 933 | + | |
| 934 | + | |
| 935 | + | |
| 936 | + | |
| 937 | + | |
| 938 | + | |
| 939 | + | |
| 940 | + | |
| 941 | + | |
| 942 | + | |
| 943 | + | |
| 944 | + | |
| 945 | + | |
| 946 | + | |
| 947 | + | |
| 948 | + | |
| 949 | + | |
| 950 | + | |
| 951 | + | |
| 952 | + | |
827 | 953 | | |
0 commit comments