Repository navigation
Timestamps having different timezone cannot be represented in the same field of a schema #5823
Description
Is your feature request related to a problem or challenge?
We should be able to handle the cases where timestamps with different timezone data are stored in the same field of a schema. However, Field struct holds a DataType info, and if we want to place a different timezone:
ArrowError(InvalidArgumentError("column types must match schema types, expected Timestamp(Millisecond, Some(\"+03:00\")) but found Timestamp(Millisecond, None) at column index 0"))
Describe the solution you'd like
Timezone check for timestamps can be ignored while matching columns with the schema
Describe alternatives you've considered
Additional context
Reacted by Christopher Watford
Activity
tustvold commentedon Apr 1, 2023
I don't believe this is possible with the arrow data model, a column must only contain data relative to a single timezone epoch (or none). That being said timezone aware parsing was added to arrow-rs in apache/arrow-rs#3795 and allows parsing a timestamp with a given timezone, into the timezone of the column, e.g. UTC, preserving the same instant in time
apache/arrow-rs#1936 is the related arrow-rs ticket to handle this in the cast kernel (in addition to the existing CSV and JSON support)