Repository navigation
[Spark] Dynamic partition overwrite deletes every partition #26
Description
With spark.sql.sources.partitionOverwriteMode=dynamic, an overwrite should replace only the partitions it writes. Vortex silently deletes every partition under the path.
Repro
Write id, id % 2 as p from range(10) partitioned by p. Then, in dynamic mode, overwrite with one row in p=1:
| Format | Rows per partition after the overwrite |
|---|---|
| parquet | p=0: 5, p=1: 1 |
| vortex | p=1: 1 (p=0 is gone) |
Why
VortexWriteBuilder implements only SupportsTruncate, so Spark turns every overwrite into a truncate, and VortexBatchWrite.createBatchWriterFactory deletes every file under the path before writing.
Impact
Rerunning one partition of a partitioned table, for example a day of a daily job, is the usual way to use dynamic overwrite. On Vortex it deletes the rest of the table, with no error.
Fix
Implement SupportsDynamicOverwrite: delete only the partition directories the job writes. Since the partitions aren't known until tasks run, delete them at job commit, from the partition paths in the commit messages. Until that exists, fail a dynamic overwrite instead of truncating.
Activity
ChristopherWirt commentedon Nov 2, 2026
Resolved by the Spark FileFormat connector merged in #35. Writes now use Spark partition-overwrite handling. VortexWriteCommitTest.anOverwriteReplacesOnlyThePartitionsItsModeSelects and insertOverwriteReplacesOnlyThePartitionsItsModeSelects cover both DataFrame and SQL writes in static and dynamic modes against Parquet, including preservation of untouched partitions. The completed write-lifecycle evaluation in #31 also records this fix. Confirmed against develop abb4830 by source and existing regression inspection; no tests were rerun for this audit.