Repository navigation
[Bug] ArrowFlight SQL result automatically converts timezone naive date to UTC (which is wrong) #65741
Description
Search before asking
- I had searched in the issues and found no similar issues.
Version
When using arrow to execute sql:
SELECT CAST('2026-07-02 01:36:22.069504' AS DATETIME(6)) AS ts
What's Wrong?
The result is automatically converted to UTC.
2026-07-02 01:36:22.069504+00:00
What You Expected?
The result sould be
2026-07-02 01:36:22.069504
This is the wrong assumption. the original timestamp might be local timezone timestamp.
How to Reproduce?
Execute
SELECT CAST('2026-07-02 01:36:22.069504' AS DATETIME(6)) AS ts
in python:
import adbc_driver_flightsql.dbapi as flightsql
from adbc_driver_manager import DatabaseOptions
from q_secrets import get_connection_details
details = get_connection_details("doris", flightsql=True)
connection = flightsql.connect(
uri=f"{details['driver']}://{details['hostname']}:{details['port']}",
db_kwargs={
DatabaseOptions.USERNAME.value: details["username"],
DatabaseOptions.PASSWORD.value: details["pwd"],
},
)
with connection.cursor() as cursor:
cursor.execute("SELECT CAST('2026-07-02 01:36:22.069504' AS DATETIME(6)) AS ts")
table = cursor.fetch_arrow_table()
field = table.schema.field("ts")
print("Arrow field:", field)
print("Field metadata:", field.metadata)
print("Value:", table["ts"].to_pylist())
connection.close()
Anything Else?
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
zclllyybb commentedon Jul 17, 2026
Breakwater-GitHub-Analysis-Slot: slot_a66f91e21aeb
This content is generated by AI for reference only.
Initial assessment: this looks like a real Arrow Flight SQL type-mapping bug for
DATETIMEV2, not only a Python client display issue.I checked current
upstream/masterbecause the issue does not specify an affected Doris version. The relevant path is:CAST(... AS DATETIME(6))is aDATETIMEV2(6)result.be/src/format/arrow/arrow_row_batch.cpp,convert_to_arrow_type()currently handlesTYPE_TIMESTAMPTZandTYPE_DATETIMEV2in the same branch and createsarrow::TimestampType(unit, timezone). The nearby TODO already notes that these types may need to be distinguished.state->timezone()inbe/src/exec/operator/result_sink_operator.cpp, so the Arrow field receives non-empty timezone metadata.DataTypeDateTimeV2SerDe::write_column_to_arrow()then readstimestamp_type->timezone()and, when it is non-empty, serializes the value using the session timezone. With a UTC session/default timezone, PyArrow/ADBC will naturally return a timezone-aware Python value such as2026-07-02 01:36:22.069504+00:00.That behavior is probably wrong for ordinary Doris
DATETIME/DATETIMEV2: it is a timezone-naive wall-clock type, while an Arrow timestamp with timezone has instant semantics. The timezone-aware Arrow representation is appropriate forTIMESTAMPTZ, but should not be reused forDATETIMEV2.Suggested fix direction:
TYPE_DATETIMEV2andTYPE_TIMESTAMPTZin Arrow schema conversion.TYPE_DATETIMEV2, emit an Arrow timestamp with the correct unit but without timezone metadata, or another explicitly timezone-naive representation.TYPE_TIMESTAMPTZ.regression-test/suites/arrow_flight_sql_p0/test_select.groovycoversdatetime(6)result values, but it appears to compare converted output and does not assert that the Arrow field timezone is empty.Missing information that would help confirm the exact user-facing scope:
@@time_zonewhen the query is run.Arrow field:line, plus PyArrow / ADBC driver versions.morningman commentedon Jul 17, 2026
link to #65615
wushilin commentedon Nov 7, 2026
Closing