Skip to content

Commit 1aa2ad7

Browse files
feat(scan): let a scan request bound the splits it decodes at once (#64)
Closes #63. Helps with #61, but doesn't fix it. ## Problem - A scan decodes `4 × available_parallelism()` splits at once. - In Spark, every task in an executor runs its own scan sized to the whole machine. - Each split holds the decoded chunks of every column it overlaps. - On wide, nested files that's GiBs of native memory per executor (#61). ## Change An absolute bound on splits decoded at once, settable per scan: | Layer | API | |---|---| | Rust | `ScanRequest::split_concurrency` → `ScanBuilder::with_some_max_splits_in_flight` | | JNI | `ScanOptions.splitConcurrency()` | | Spark | `vortex.splitConcurrency` read option | - It's a cap kept apart from the per-worker `with_concurrency`. A later `with_concurrency` (as the DataFusion opener calls) leaves it in place, and `concurrency()` keeps its meaning. - JNI also bounds the Arrow conversion it runs ahead of Java (`buffered(worker_count)`) by the same value. - It doesn't bound I/O: segment reads for later splits may still be coalesced ahead of them. The docs say so. ## Effect A full scan of a 180 MB list-layout file, peak live native bytes: | splits at once | peak live | time | |---|---|---| | 48 (default, 4 × 12 cores) | 3.65 GiB | 1.0 s | | 12 | 1.18 GiB | 1.0 s | ## Tests - **Rust:** a scan of 64 one-row splits decodes exactly 1, and exactly 2, at once with the bound set, against 48 by default. A later `with_concurrency` keeps the bound. - **JNI:** `ScanOptions` validation, and a scan with the bound reads every row. - **Spark:** option parsing, validation and serialisation, plus V1 and V2 reads with the option set. - **Suites:** Rust (vortex-layout, vortex-scan, vortex-jni, vortex-file), JNI, Spark 3.5 and Spark 4.0 all pass. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Christopher Wirt <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]>
1 parent 769938b commit 1aa2ad7

18 files changed

Lines changed: 424 additions & 57 deletions

File tree

‎docs/user-guide/spark.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -241,6 +241,7 @@ depends on, on the driver and executor classpath, for example
241241
| Option | Default | Description |
242242
|----------------------------|---------|-------------|
243243
| `vortex.readConcurrency` | `0` | Maximum Hadoop read upcalls per file; `0` uses the native default. |
244+
| `vortex.splitConcurrency` | `0` | Most row splits of a file decoded at once by one task; `0` uses the native default of four per core. Lowering this reduces memory held by concurrent split decoding and Arrow conversion. A scan with both a filter and a limit may also retain one trimmed array per split while preparing output. This does not cap total native memory or I/O read-ahead. |
244245
| `vortex.workerThreads` | `4` | Background threads driving Vortex futures. JVM-wide, and set once per JVM by the first read or write. |
245246
| `vortex.session.provider` | — | Fully-qualified name of a `VortexSessionProvider` supplying a custom session on the driver and on every executor. |
246247
| `vortex.aggregatePushdown` | `true` | Answer `COUNT(*)` from file footers. |

‎java/vortex-jni/src/main/java/dev/vortex/api/DataSource.java‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -238,6 +238,7 @@ public Scan scan(ScanOptions options) {
238238
byte[] selectionRoaringBitmap = selectionRoaringBitmap(options);
239239
long limit = options.limit().orElse(0L);
240240
boolean ordered = options.ordered();
241+
int splitConcurrency = options.splitConcurrency().orElse(0);
241242

242243
long scanPtr = dev.vortex.jni.NativeScan.create(
243244
pointer,
@@ -249,7 +250,8 @@ public Scan scan(ScanOptions options) {
249250
selectionRoaringBitmap,
250251
selectionMode.code(),
251252
limit,
252-
ordered);
253+
ordered,
254+
splitConcurrency);
253255
return Scan.fromPointer(session, scanPtr);
254256
}
255257

‎java/vortex-jni/src/main/java/dev/vortex/api/ScanOptions.java‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@
99
import java.io.UncheckedIOException;
1010
import java.util.Objects;
1111
import java.util.Optional;
12+
import java.util.OptionalInt;
1213
import java.util.OptionalLong;
1314
import org.immutables.value.Value;
1415
import org.roaringbitmap.longlong.Roaring64NavigableMap;
@@ -59,6 +60,25 @@ default boolean ordered() {
5960
return false;
6061
}
6162

63+
/**
64+
* The most splits of each partition the native scan decodes at once, in all, and the most of its batches converted
65+
* to Arrow ahead of the caller. Absent keeps the native default, which scales with the machine's cores.
66+
*
67+
* <p>Each split being decoded holds the decoded chunks of every column it overlaps, so a scan running alongside
68+
* others, such as one Spark task of several in an executor, can lower this to reduce memory held by concurrent
69+
* decoding and Arrow conversion. A scan with both a filter and a limit may also retain one trimmed array per split
70+
* while preparing output. This does not cap total native memory or I/O read-ahead.
71+
*/
72+
OptionalInt splitConcurrency();
73+
74+
@Value.Check
75+
default void validateSplitConcurrency() {
76+
if (splitConcurrency().isPresent() && splitConcurrency().getAsInt() <= 0) {
77+
throw new IllegalArgumentException("split concurrency must be positive, got "
78+
+ splitConcurrency().getAsInt());
79+
}
80+
}
81+
6282
@Value.Check
6383
default void validateSelectionPayload() {
6484
boolean hasIndices = selectionIndices().isPresent();

‎java/vortex-jni/src/main/java/dev/vortex/jni/NativeScan.java‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,7 @@ private NativeScan() {}
2727
* {@code selectionRoaringBitmap})
2828
* @param limit max rows to return, or {@code 0} for "no limit"
2929
* @param ordered true to preserve row order across partitions
30+
* @param splitConcurrency the most splits of each partition read at once, or {@code 0} for the native default
3031
*/
3132
public static native long create(
3233
long dataSourcePointer,
@@ -38,7 +39,8 @@ public static native long create(
3839
byte[] selectionRoaringBitmap,
3940
byte selectionInclude,
4041
long limit,
41-
boolean ordered);
42+
boolean ordered,
43+
int splitConcurrency);
4244

4345
/** Free a scan pointer. */
4446
public static native void free(long pointer);

‎java/vortex-jni/src/test/java/dev/vortex/api/ScanOptionsTest.java‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@
1212
import dev.vortex.api.ScanOptions.SelectionMode;
1313
import java.io.ByteArrayInputStream;
1414
import java.io.DataInputStream;
15+
import java.util.OptionalInt;
1516
import java.util.OptionalLong;
1617
import org.junit.jupiter.api.Test;
1718
import org.roaringbitmap.longlong.Roaring64NavigableMap;
@@ -39,6 +40,21 @@ public void defaultsReadEveryRowAndColumn() {
3940
assertTrue(options.selectionRoaringBitmap().isEmpty());
4041
assertEquals(SelectionMode.INCLUDE_ALL, options.selectionMode());
4142
assertFalse(options.ordered());
43+
assertEquals(OptionalInt.empty(), options.splitConcurrency());
44+
}
45+
46+
@Test
47+
public void splitConcurrencyMustBePositive() {
48+
for (int value : new int[] {0, -1}) {
49+
IllegalArgumentException exception = assertThrows(
50+
IllegalArgumentException.class,
51+
() -> ScanOptions.builder().splitConcurrency(value).build(),
52+
() -> "no exception for " + value);
53+
assertEquals("split concurrency must be positive, got " + value, exception.getMessage());
54+
}
55+
assertEquals(
56+
OptionalInt.of(2),
57+
ScanOptions.builder().splitConcurrency(2).build().splitConcurrency());
4258
}
4359

4460
@Test

‎java/vortex-jni/src/test/java/dev/vortex/api/TestMinimal.java‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -184,6 +184,18 @@ public void testProjectedScanWithFilter() throws Exception {
184184
assertEquals(List.of(new Person("John", BigDecimal.valueOf(10_000L, 2), "VA")), people);
185185
}
186186

187+
@Test
188+
public void testScanWithSplitConcurrencyReadsEveryRow() throws Exception {
189+
BufferAllocator allocator = ArrowAllocation.rootAllocator();
190+
Session session = Session.create();
191+
DataSource ds = DataSource.open(session, writePath);
192+
193+
ScanOptions options =
194+
ScanOptions.builder().splitConcurrency(1).ordered(true).build();
195+
List<Person> people = readAll(ds, options, allocator, TestMinimal::readFullBatch);
196+
assertEquals(MINIMAL_DATA, people);
197+
}
198+
187199
@Test
188200
public void testSelectionIncludesRows() throws Exception {
189201
BufferAllocator allocator = ArrowAllocation.rootAllocator();

‎java/vortex-spark/common/src/main/java/dev/vortex/spark/io/VortexIo.java‎

Lines changed: 28 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -30,31 +30,47 @@ public final class VortexIo implements Serializable {
3030

3131
private static final int DEFAULT_READ_CONCURRENCY = 0;
3232

33+
/**
34+
* Option bounding how many splits of a file the native scan reads at once, in all. Each split in flight holds the
35+
* decoded chunks of every column it overlaps, and every task in an executor runs its own scan, which by default
36+
* keeps several splits in flight per core. Lowering this reduces memory held by concurrent split decoding and Arrow
37+
* conversion. A scan with both a filter and a limit may also retain one trimmed array per split while preparing
38+
* output. This does not cap total native memory or I/O read-ahead. Zero keeps the native default.
39+
*/
40+
public static final String SPLIT_CONCURRENCY_OPTION = "vortex.splitConcurrency";
41+
42+
private static final int DEFAULT_SPLIT_CONCURRENCY = 0;
43+
3344
private final SerializableHadoopConf conf;
3445
private final int readConcurrency;
46+
private final int splitConcurrency;
3547

3648
/** Captures {@code hadoopConf} and the read settings found in the format options. */
3749
public static VortexIo create(VortexOptions options, Configuration hadoopConf) {
3850
Objects.requireNonNull(options, "options");
3951
Objects.requireNonNull(hadoopConf, "hadoopConf");
40-
return new VortexIo(new SerializableHadoopConf(hadoopConf), parseReadConcurrency(options));
52+
return new VortexIo(
53+
new SerializableHadoopConf(hadoopConf),
54+
nonNegative(options, READ_CONCURRENCY_OPTION, DEFAULT_READ_CONCURRENCY),
55+
nonNegative(options, SPLIT_CONCURRENCY_OPTION, DEFAULT_SPLIT_CONCURRENCY));
4156
}
4257

4358
/** Hadoop I/O over a default configuration, for callers that have no Spark session to draw one from. */
4459
public static VortexIo defaults() {
45-
return new VortexIo(new SerializableHadoopConf(new Configuration()), DEFAULT_READ_CONCURRENCY);
60+
return new VortexIo(
61+
new SerializableHadoopConf(new Configuration()), DEFAULT_READ_CONCURRENCY, DEFAULT_SPLIT_CONCURRENCY);
4662
}
4763

48-
private VortexIo(SerializableHadoopConf conf, int readConcurrency) {
64+
private VortexIo(SerializableHadoopConf conf, int readConcurrency, int splitConcurrency) {
4965
this.conf = conf;
5066
this.readConcurrency = readConcurrency;
67+
this.splitConcurrency = splitConcurrency;
5168
}
5269

53-
private static int parseReadConcurrency(VortexOptions options) {
54-
int parsed = options.getInt(READ_CONCURRENCY_OPTION, DEFAULT_READ_CONCURRENCY);
70+
private static int nonNegative(VortexOptions options, String key, int defaultValue) {
71+
int parsed = options.getInt(key, defaultValue);
5572
if (parsed < 0) {
56-
throw new IllegalArgumentException(
57-
String.format(Locale.ROOT, "%s must be >= 0, got %d", READ_CONCURRENCY_OPTION, parsed));
73+
throw new IllegalArgumentException(String.format(Locale.ROOT, "%s must be >= 0, got %d", key, parsed));
5874
}
5975
return parsed;
6076
}
@@ -68,6 +84,11 @@ public int readConcurrency() {
6884
return readConcurrency;
6985
}
7086

87+
/** The most splits of a file the native scan reads at once; zero keeps the native default. */
88+
public int splitConcurrency() {
89+
return splitConcurrency;
90+
}
91+
7192
/**
7293
* Opens a byte source for {@code file}, stating it only if the listing that produced it did not report a size. The
7394
* caller owns the result and must close it once the scan built on it is done.

‎java/vortex-spark/common/src/main/java/dev/vortex/spark/read/VortexPartitionReader.java‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -115,6 +115,9 @@ public VortexPartitionReader(
115115
projectedSchema, arrowFields, caseSensitive, file.toPath().toString());
116116

117117
var options = ScanOptions.builder();
118+
if (io.splitConcurrency() > 0) {
119+
options.splitConcurrency(io.splitConcurrency());
120+
}
118121
// Always project, an empty read schema included: a query that needs only partition columns or only a
119122
// row count must not pull every data column off storage. Struct columns that Spark pruned read only the
120123
// nested fields the query uses.

‎java/vortex-spark/common/src/test/java/dev/vortex/spark/VortexSessionOptionsTest.java‎

Lines changed: 17 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@
77
import static org.junit.jupiter.api.Assertions.assertThrows;
88
import static org.junit.jupiter.api.Assertions.assertTrue;
99

10+
import dev.vortex.spark.io.VortexIo;
1011
import java.nio.file.Path;
1112
import org.apache.spark.sql.Dataset;
1213
import org.apache.spark.sql.Row;
@@ -43,7 +44,9 @@ public void setUp() {
4344

4445
@AfterEach
4546
public void resetConf() {
46-
for (String option : new String[] {READ_CONCURRENCY, VortexSparkSession.PROVIDER_OPTION}) {
47+
for (String option :
48+
new String[] {READ_CONCURRENCY, VortexIo.SPLIT_CONCURRENCY_OPTION, VortexSparkSession.PROVIDER_OPTION
49+
}) {
4750
spark.conf().unset(option);
4851
spark.conf().unset(DATA_SOURCE_PREFIX + option);
4952
}
@@ -84,6 +87,19 @@ void aSessionSettingAppliesToV1Reads() {
8487
assertSessionSettingAppliesToReads(write("v1"), "FileScan vortex", READ_CONCURRENCY);
8588
}
8689

90+
@Test
91+
void aSplitConcurrencyReadsEveryRowThroughV1AndV2() {
92+
String path = write("split_concurrency");
93+
for (String useV1 : new String[] {"", "vortex"}) {
94+
spark.conf().set(USE_V1, useV1);
95+
Dataset<Row> read = spark.read()
96+
.format("vortex")
97+
.option(VortexIo.SPLIT_CONCURRENCY_OPTION, "1")
98+
.load(path);
99+
assertEquals(45L, read.selectExpr("sum(id)").first().getLong(0), "useV1List=" + useV1);
100+
}
101+
}
102+
87103
@Test
88104
void aSessionSettingAppliesToWrites() {
89105
assertSessionSettingAppliesToWrites("written", VortexSparkSession.PROVIDER_OPTION);

‎java/vortex-spark/common/src/test/java/dev/vortex/spark/io/VortexIoTest.java‎

Lines changed: 25 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -18,10 +18,30 @@
1818

1919
final class VortexIoTest {
2020
@Test
21-
void readConcurrencyDefaultsToNativeDefault() {
21+
void concurrenciesDefaultToNativeDefaults() {
2222
VortexIo io = VortexIo.create(VortexOptions.empty(), new Configuration());
2323

2424
assertEquals(0, io.readConcurrency());
25+
assertEquals(0, io.splitConcurrency());
26+
}
27+
28+
@Test
29+
void splitConcurrencyIsParsedAndValidated() {
30+
Configuration conf = new Configuration();
31+
32+
assertEquals(
33+
2,
34+
VortexIo.create(options(VortexIo.SPLIT_CONCURRENCY_OPTION, "2"), conf)
35+
.splitConcurrency());
36+
assertEquals(
37+
2,
38+
VortexIo.create(options("VORTEX.SPLITCONCURRENCY", "2"), conf).splitConcurrency());
39+
assertThrows(
40+
IllegalArgumentException.class,
41+
() -> VortexIo.create(options(VortexIo.SPLIT_CONCURRENCY_OPTION, "-1"), conf));
42+
assertThrows(
43+
IllegalArgumentException.class,
44+
() -> VortexIo.create(options(VortexIo.SPLIT_CONCURRENCY_OPTION, "many"), conf));
2545
}
2646

2747
@Test
@@ -54,11 +74,14 @@ void readConcurrencyIsFoundUnderAnyKeyCase() {
5474
void settingsAndConfigurationSurviveSerialization() throws IOException, ClassNotFoundException {
5575
Configuration conf = new Configuration();
5676
conf.set("fs.s3a.endpoint", "https://storage.example");
57-
VortexIo io = VortexIo.create(options(VortexIo.READ_CONCURRENCY_OPTION, "4"), conf);
77+
VortexIo io = VortexIo.create(
78+
VortexOptions.of(Map.of(VortexIo.READ_CONCURRENCY_OPTION, "4", VortexIo.SPLIT_CONCURRENCY_OPTION, "3")),
79+
conf);
5880

5981
VortexIo shipped = roundTrip(io);
6082

6183
assertEquals(4, shipped.readConcurrency());
84+
assertEquals(3, shipped.splitConcurrency());
6285
assertEquals("https://storage.example", shipped.hadoopConf().get("fs.s3a.endpoint"));
6386
}
6487

0 commit comments

Comments
 (0)