Repository navigation
[Spark] Date and timestamp partitions are written as epoch counts #28
Description
Date and timestamp partition columns are written as epoch counts, and read back as numbers.
| Partition by | Spark and Hive directory | Vortex directory |
|---|---|---|
date 2026-10-29 |
p=2026-10-29 |
p=20755 |
timestamp |
p=2026-10-29 01%3A02%3A03 (session time zone) |
microseconds since epoch |
Schema inference (PartitionPathUtils.inferPartitionColumnType) then reads the column as IntegerType or LongType, so a table written with a date partition reads back with an int column. Spark and Hive can't read the values as dates either.
Where
PartitionedVortexDataWriter.extractIdentityValue writes row.getInt and row.getLong for DateType and TimestampType. PartitionPathUtils.createConstantVector parses them back as numbers.
Fix
Format dates as yyyy-MM-dd and timestamps the way Spark does (Cast to string in the session time zone), infer date and timestamp types from those strings as Spark's partition discovery does, and keep accepting the old numeric form when the table schema says the column is a date or timestamp. Best done after #24, which settles the escaping these strings go through.
Activity
ChristopherWirt commentedon Nov 4, 2026
Closing as superseded by the merged FileFormat connector migration in #35.
The custom
PartitionedVortexDataWriter.extractIdentityValueandPartitionPathUtilsimplementation described here has been removed. The connector now delegates partition directory formatting and discovery to Spark's file-source machinery:VortexTableextendsFileTable, and V1 writes provide anOutputWriterFactoryto Spark. New date/timestamp partition names therefore follow Spark's formatting instead of the old connector's raw epoch counts.This closure does not claim migration or special compatibility handling for directories already written in the old numeric format. Any required legacy-data support should be tracked separately with a concrete compatibility case.
Verified by source review of current
develop; no new tests were run.