Repository navigation
[Fix] Adapt Arrow timestamp conversion for DATETIME and TIMESTAMPTZ - #366
Merged
Merged
Conversation
JNSimba
changed the title
CalvinKirs
approved these changes
Jul 27, 2026
3 tasks done
manesioz
added a commit
to manesioz/doris-spark-connector
that referenced
this pull request
Aug 27, 2026
Keep the public getDateTime(row, vector) path from apache#366 for TIMESTAMPTZ and unknown types. Only DATETIME/DATETIMEV2 with a non-null Arrow timezone decode in ZoneId.of(tz). Signed-off-by: Zachary Manesiotis <[email protected]>
This was referenced Aug 27, 2026
JNSimba
pushed a commit
that referenced
this pull request
Sep 3, 2026
Closes #376. Doris DATETIME and DATETIMEV2 store wall-clock values without timezone semantics. At times, Doris BE nodes can return them through timezone-aware Arrow vectors. RowBatch treated these values like TIMESTAMPTZ instants. When the Arrow timezone differed from the Spark JVM timezone, the displayed clock shifted. For example, Doris 2026-08-24 10:00:09 could become 2026-08-24 02:00:09 when Spark ran in UTC but the BE configured tz was Shanghai/UTC+8. #366 supports Arrow timestamps without timezone metadata. It does not recover wall-clock values when older backends include this metadata.
2 of 3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Proposed changes
Background:
PR apache/doris#38215 added timezone support to
datetime. Versions prior to this PR hardcoded the use ofTimeStampMicroVector, whereas subsequent versions returnTimeStampTZVector. Consequently, to ensure compatibility, the connector inferred the time unit based on the timestamp itself.Changes:
Since
datetimeis inherently timezone-agnostic, PR apache/doris#65823 removed the timezone fromdatetime, causing it to return a native timestamp;timestamptzcontinues to returnTimeStampTZVector.Connector Adaptation:
No conversion is performed if there is no timezone; the existing default conversion logic is applied if a timezone is present.
Impact:
Reading
datetimedata may result in errors on versions prior to PR apache/doris#38215.Checklist(Required)
Further comments
None.