perf(spark): append rows straight into the Arrow batch (FileFormat connector) - #45

Summary

The FileFormat writer copied every row into a list and converted the list to Arrow per value when a batch filled, re-exported the Arrow schema for every batch without releasing it, and could strand other partition writers when one write failed. Now each row is appended straight into the Arrow vectors, the schema is exported once per writer, and a failed writer lets Spark close every other writer.

Changes

• ArrowFieldWriter: one typed writer per field, with struct and map children resolved once; strings are copied once from the row into the Arrow buffer, decimals of up to 18 digits are written from the unscaled long.
• VortexOutputWriter appends each row into vectors that start at min(batch, 64) slots and grow. The schema is exported once and released on close; a batch native never took is released on the Java side.
• After a write fails, later writes throw and close() releases everything without throwing, so Spark's concurrent partition writer, which stops at the first close that throws, still closes the other writers. The truncated file can't be committed: Spark commits only after every write returned normally.
• Arrow memory left at close is logged and the allocator left open, so a late native release is still freed (Arrow marks an allocator closed even when its close throws).
• Tests: the Spark 4.1 and 4.2 compat suites now compile the Scala test sources, and test JVMs open java.io as Spark's launcher does (Spark 4.2's serialization debugger needs it).

Testing

• A/B on 4M rows, 6 columns, 18 runs each, with nothing else building: median 15.2 s against 15.3 s with 4 tasks, 21.8 s against 21.9 s with 1 task. No difference.
• Memory: 2,000 one-row batches hold 1.4 MB of Arrow memory instead of 253 MB; 10,000 batches run where the old writer hit its direct-memory limit.
• Tests that fail without the change: VortexOutputWriterMemoryTest (2,980 B left after close), the row-reuse test (binary values from a reused row read back wrong), VortexOutputWriterAbortTest (28 healthy partition writers stranded, 36,344 B), and the batch-size-1 nested-nulls round trip (out of direct memory).
• Full check passes on Spark 3.5/2.12, 3.5/2.13, 4.0, and the 4.1 and 4.2 compat runs.
• Reviewed by Astra (one round; its finding is fixed here) and Sol (two rounds on the allocator change).

Known: the native writer still holds imported batches until it has 8,192 rows and buffers per column, so memory grows with rows per file for small batch sizes.

Checked locally.

I used AI tools to help write and review this change.