Repository navigation
Commit 7c1972b
authored
fix(spark): release native read resources deterministically (#11367)
## Summary
Fixes #11364.
Closing a Spark reader currently leaves its Java-owned native read
handles and file caches alive until Cleaner processing. Make that
release explicit and idempotent, while preserving the resources still
needed by derived scans, returned partitions and Arrow batches. Ports
the lifecycle fix from [QC
#103](quantcast#103).
## Changes
- Make `DataSource` and `Scan` implement `AutoCloseable`, retaining
Cleaner fallback. Keep native calls synchronized with explicit close;
fence scan options so their expressions remain alive across JNI.
- Release partitions prefetched by `hasNext()` but never returned.
- Let Spark's owning wrappers close at EOF. Reader and footer cleanup
attempts every close in ownership order and preserves the original
failure.
- Release native partition ownership when Arrow stream allocation or
import fails.
- Reject reads on closed handles with `IllegalStateException`. An open,
exhausted scan still returns `false` from `hasNext()` and throws
`NoSuchElementException` from `next()`.
---------
Signed-off-by: Christopher Wirt <[email protected]>1 parent 5dac460 commit 7c1972b
8 files changed
Lines changed: 694 additions & 113 deletions
File tree
- java
- vortex-jni/src
- main/java/dev/vortex
- api
- io
- test/java/dev/vortex/api
- vortex-spark/common/src
- main/java/dev/vortex/spark/read
- test/java/dev/vortex/spark/read
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
7 | 7 | | |
8 | 8 | | |
9 | 9 | | |
| 10 | + | |
| 11 | + | |
10 | 12 | | |
11 | 13 | | |
12 | 14 | | |
| |||
22 | 24 | | |
23 | 25 | | |
24 | 26 | | |
25 | | - | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
26 | 33 | | |
27 | | - | |
| 34 | + | |
28 | 35 | | |
29 | 36 | | |
| 37 | + | |
| 38 | + | |
30 | 39 | | |
31 | 40 | | |
32 | 41 | | |
33 | 42 | | |
34 | 43 | | |
35 | | - | |
| 44 | + | |
36 | 45 | | |
37 | 46 | | |
38 | 47 | | |
| |||
88 | 97 | | |
89 | 98 | | |
90 | 99 | | |
91 | | - | |
92 | | - | |
| 100 | + | |
| 101 | + | |
93 | 102 | | |
94 | 103 | | |
95 | 104 | | |
| |||
118 | 127 | | |
119 | 128 | | |
120 | 129 | | |
121 | | - | |
| 130 | + | |
| 131 | + | |
122 | 132 | | |
123 | 133 | | |
124 | 134 | | |
| |||
130 | 140 | | |
131 | 141 | | |
132 | 142 | | |
133 | | - | |
| 143 | + | |
| 144 | + | |
134 | 145 | | |
135 | 146 | | |
136 | 147 | | |
| |||
181 | 192 | | |
182 | 193 | | |
183 | 194 | | |
184 | | - | |
| 195 | + | |
| 196 | + | |
185 | 197 | | |
186 | 198 | | |
187 | 199 | | |
| |||
226 | 238 | | |
227 | 239 | | |
228 | 240 | | |
229 | | - | |
230 | | - | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
231 | 245 | | |
232 | | - | |
233 | | - | |
234 | | - | |
235 | | - | |
236 | | - | |
237 | | - | |
238 | | - | |
239 | | - | |
240 | | - | |
241 | | - | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
242 | 257 | | |
243 | | - | |
244 | | - | |
245 | | - | |
246 | | - | |
247 | | - | |
248 | | - | |
249 | | - | |
250 | | - | |
251 | | - | |
252 | | - | |
253 | | - | |
254 | | - | |
255 | | - | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
256 | 283 | | |
257 | 284 | | |
258 | 285 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
6 | 6 | | |
7 | 7 | | |
8 | 8 | | |
| 9 | + | |
9 | 10 | | |
10 | 11 | | |
11 | 12 | | |
| |||
43 | 44 | | |
44 | 45 | | |
45 | 46 | | |
46 | | - | |
47 | | - | |
48 | | - | |
49 | | - | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
50 | 56 | | |
51 | | - | |
52 | 57 | | |
53 | 58 | | |
54 | 59 | | |
| |||
59 | 64 | | |
60 | 65 | | |
61 | 66 | | |
62 | | - | |
63 | | - | |
64 | | - | |
65 | | - | |
66 | 67 | | |
67 | | - | |
68 | | - | |
69 | | - | |
70 | | - | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
71 | 91 | | |
72 | | - | |
73 | 92 | | |
74 | 93 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
5 | 5 | | |
6 | 6 | | |
7 | 7 | | |
| 8 | + | |
8 | 9 | | |
| 10 | + | |
9 | 11 | | |
10 | 12 | | |
| 13 | + | |
11 | 14 | | |
12 | 15 | | |
13 | 16 | | |
| |||
16 | 19 | | |
17 | 20 | | |
18 | 21 | | |
19 | | - | |
20 | | - | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
21 | 28 | | |
22 | | - | |
| 29 | + | |
23 | 30 | | |
24 | 31 | | |
25 | 32 | | |
26 | | - | |
| 33 | + | |
| 34 | + | |
27 | 35 | | |
| 36 | + | |
28 | 37 | | |
29 | 38 | | |
30 | 39 | | |
31 | 40 | | |
32 | 41 | | |
33 | | - | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
34 | 53 | | |
35 | 54 | | |
36 | 55 | | |
| |||
40 | 59 | | |
41 | 60 | | |
42 | 61 | | |
43 | | - | |
| 62 | + | |
| 63 | + | |
44 | 64 | | |
45 | 65 | | |
46 | 66 | | |
47 | 67 | | |
48 | 68 | | |
49 | 69 | | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
50 | 75 | | |
51 | | - | |
52 | | - | |
53 | | - | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
54 | 81 | | |
55 | | - | |
56 | | - | |
57 | | - | |
| 82 | + | |
58 | 83 | | |
59 | 84 | | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
60 | 90 | | |
61 | | - | |
| 91 | + | |
62 | 92 | | |
63 | 93 | | |
64 | 94 | | |
65 | | - | |
66 | | - | |
| 95 | + | |
67 | 96 | | |
68 | 97 | | |
69 | 98 | | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
70 | 108 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
16 | 16 | | |
17 | 17 | | |
18 | 18 | | |
19 | | - | |
| 19 | + | |
20 | 20 | | |
21 | 21 | | |
22 | 22 | | |
| |||
0 commit comments