[Spark] Appends from separate applications overwrite each other's files #30

Appending to the same table from two Spark applications silently replaces the first application's files.

Repro

Run spark.range(10).write().format("vortex").mode("append").save(path) in one application, stop it, and run the same in a second application. Reading path returns 10 rows, not 20.

Why

Files are named part-<partition>-<taskId>.vortex (VortexDataWriterFactory.createWriter, PartitionedVortexDataWriter.createWriterForPartition). Task ids restart at 0 in every application, so the second job's part-00000-0.vortex overwrites the first one's in place. The same happens between two partitions' writers in different applications, and between concurrent applications writing to one path.

Impact

Any table appended to by more than one application run, like a scheduled daily append, keeps only the latest run's files wherever names repeat. No error is raised.

Fix

Put the query's unique id in the file name, as Spark does (part-00000-<uuid>-c000). LogicalWriteInfo.queryId() gives one per write. Keep the task id too, so retried attempts of one task still write different files (#22 relies on that).