Repository navigation
[Fix] Adapt Arrow timestamp conversion for DATETIME and TIMESTAMPTZ - #676
Merged
Merged
Conversation
JNSimba
changed the title
There was a problem hiding this comment.
Pull request overview
Adapts the Flink Doris connector’s Arrow timestamp-to-LocalDateTime conversion to distinguish between timezone-aware (TIMESTAMPTZ) and timezone-less (DATETIME/DATETIMEV2) Arrow Timestamp vectors, aligning behavior with upstream Doris changes around datetime timezone handling.
Changes:
- Update timestamp conversion to use Arrow
Timestampmetadata (unit + presence/absence of timezone) rather than inferring precision from the numeric value. - Add
TIMESTAMPTZhandling alongsideDATETIMEV2inRowBatchconversion logic. - Extend/adjust unit tests to cover both non-TZ timestamp vectors and TZ timestamp vectors across second/milli/micro units.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| flink-doris-connector/flink-doris-connector-base/src/main/java/org/apache/doris/flink/serialization/RowBatch.java | Updates conversion logic for Arrow timestamps (unit-aware) and adds TIMESTAMPTZ support with “no conversion when timezone is absent” behavior. |
| flink-doris-connector/flink-doris-connector-base/src/test/java/org/apache/doris/flink/serialization/TestRowBatch.java | Updates datetime-related tests and adds coverage for timestamp vectors with and without timezone across multiple time units. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
CalvinKirs
approved these changes
Jul 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Proposed changes
Background:
PR apache/doris#38215 added timezone support to
datetime. Versions prior to this PR hardcoded the use ofTimeStampMicroVector, whereas subsequent versions returnTimeStampTZVector. Consequently, to ensure compatibility, the connector inferred the time unit based on the timestamp itself.Changes:
Since
datetimeis inherently timezone-agnostic, PR apache/doris#65823 removed the timezone fromdatetime, causing it to return a native timestamp;timestamptzcontinues to returnTimeStampTZVector.Connector Adaptation:
No conversion is performed if there is no timezone; the existing default conversion logic is applied if a timezone is present.
Impact:
Reading
datetimedata may result in errors on versions prior to PR apache/doris#38215.Checklist(Required)
Further comments
None.