Skip to content

[Spark] Writes land at their final paths before the job commits #27

Closed

Description

@gitRasheed

Vortex writes each task's file straight to its final path. Spark's own file sources write under _temporary and move files into place when the job commits.

What goes wrong

  • A reader running during a write lists and opens files that are still being written, or that belong to a job that later fails.
  • If the driver dies before the job commits or aborts, the files of every finished task stay behind, and the next read includes them. A rerun then duplicates those rows.

Failed task attempts are handled by #22, which deletes their files on abort. A job that never reaches commit or abort isn't.

Where

VortexDataWriterFactory.createWriter and PartitionedVortexDataWriter.createWriterForPartition build the final path directly. VortexBatchWrite.commit only logs.

Fix

Write to a staging location under the output path, for example _temporary/<job id>/, which #20 already hides from reads, and move or rename the committed files in VortexBatchWrite.commit. On object stores a rename is a copy, so this costs a second write per file. Spark's committers have the same trade-off, and vortex-data#9658 inherits Spark's committer when it moves to FileFormat.

Activity

  1. ChristopherWirt commented on Nov 4, 2026

    @ChristopherWirt
    Collaborator

    Closing as superseded by the merged FileFormat connector migration in #35.

    The old VortexDataWriterFactory, PartitionedVortexDataWriter, and logging-only VortexBatchWrite.commit implementation described here have been replaced. Writes now use Spark's file-source write and commit machinery: VortexTable is a FileTable, and the V1 prepareWrite entry point supplies an OutputWriterFactory to Spark.

    Staging, task/job commit and abort behavior therefore belong to the configured Spark commit protocol, rather than the custom direct-to-final-path implementation this issue tracked. This does not imply transactional publication beyond that protocol's guarantees.

    Verified by source review of current develop; no new tests were run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions