Repository navigation
Commit 9f21db7
fix(spark): honor session vortex.splitFiles in V1 scans (#79)
## Summary
`VortexFileFormat.isSplitable` passed `VortexOptions.of(options.asJava)`
straight to `VortexScan.isSplitable`. The other V1 entry points
(`inferSchema`, `prepareWrite` and both reader builders) resolve session
defaults first, but this one didn't. As a result, V1 and catalog scans
ignored `vortex.splitFiles=false` when it was set in the session, under
either `vortex.splitFiles` or `spark.datasource.vortex.splitFiles`, and
split large files anyway. The V2 path was unaffected because
`VortexTable` already resolves session defaults.
The fix resolves session defaults in `isSplitable` using the class's
existing `withSessionDefaults` helper.
Found while reviewing #35.
Closes #51
## Tests
Added `VortexFileSplitTest.sessionSplitFilesTurnsSplittingOff`, which
runs for both V2 and V1. For each session key form it checks two things:
- `false` in the session plans one partition per file and reads every
row.
- An explicit `.option("vortex.splitFiles", "true")` overrides the
session setting and splits the file as usual.
## Checks
- `./gradlew -DsparkVersions=4.0 :vortex-spark-4.0_2.13:test --tests
'dev.vortex.spark.VortexFileSplitTest'` passes.
- With the fix reverted, the new test's `v1` case fails
(`AssertionFailedError` at `VortexFileSplitTest.java:266`) and its `v2`
case passes.
- Not run: the Spark 3.5 modules, formatting, and lint.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: Otis Golden <[email protected]>
Co-authored-by: Otis Golden <[email protected]>
Co-authored-by: Claude Opus 5.5 <[email protected]>1 parent fd929e9 commit 9f21db7
2 files changed
Lines changed: 31 additions & 2 deletions
File tree
- java/vortex-spark/common/src
- main/scala/dev/vortex/spark
- test/java/dev/vortex/spark
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
69 | 69 | | |
70 | 70 | | |
71 | 71 | | |
72 | | - | |
| 72 | + | |
| 73 | + | |
73 | 74 | | |
74 | 75 | | |
75 | 76 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
69 | 69 | | |
70 | 70 | | |
71 | 71 | | |
72 | | - | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
73 | 75 | | |
74 | 76 | | |
75 | 77 | | |
| |||
249 | 251 | | |
250 | 252 | | |
251 | 253 | | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
252 | 280 | | |
253 | 281 | | |
254 | 282 | | |
| |||
0 commit comments