You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Spark] Writes land at their final paths before the job commits #27
Vortex writes each task's file straight to its final path. Spark's own file sources write under _temporary and move files into place when the job commits.
What goes wrong
A reader running during a write lists and opens files that are still being written, or that belong to a job that later fails.
If the driver dies before the job commits or aborts, the files of every finished task stay behind, and the next read includes them. A rerun then duplicates those rows.
Failed task attempts are handled by #22, which deletes their files on abort. A job that never reaches commit or abort isn't.
Where
VortexDataWriterFactory.createWriter and PartitionedVortexDataWriter.createWriterForPartition build the final path directly. VortexBatchWrite.commit only logs.
Fix
Write to a staging location under the output path, for example _temporary/<job id>/, which #20 already hides from reads, and move or rename the committed files in VortexBatchWrite.commit. On object stores a rename is a copy, so this costs a second write per file. Spark's committers have the same trade-off, and vortex-data#9658 inherits Spark's committer when it moves to FileFormat.
Closing as superseded by the merged FileFormat connector migration in #35.
The old VortexDataWriterFactory, PartitionedVortexDataWriter, and logging-only VortexBatchWrite.commit implementation described here have been replaced. Writes now use Spark's file-source write and commit machinery: VortexTable is a FileTable, and the V1 prepareWrite entry point supplies an OutputWriterFactory to Spark.
Staging, task/job commit and abort behavior therefore belong to the configured Spark commit protocol, rather than the custom direct-to-final-path implementation this issue tracked. This does not imply transactional publication beyond that protocol's guarantees.
Verified by source review of current develop; no new tests were run.
Activity
ChristopherWirt commentedon Nov 4, 2026
Closing as superseded by the merged FileFormat connector migration in #35.
The old
VortexDataWriterFactory,PartitionedVortexDataWriter, and logging-onlyVortexBatchWrite.commitimplementation described here have been replaced. Writes now use Spark's file-source write and commit machinery:VortexTableis aFileTable, and the V1prepareWriteentry point supplies anOutputWriterFactoryto Spark.Staging, task/job commit and abort behavior therefore belong to the configured Spark commit protocol, rather than the custom direct-to-final-path implementation this issue tracked. This does not imply transactional publication beyond that protocol's guarantees.
Verified by source review of current
develop; no new tests were run.