Querying datetime data in DataFusion with an embedded timezone always fails #153

Activity

1. alamb

   added

   datafusion

   on Apr 26, 2021
2. alamb commented on Apr 26, 2021

   

   alamb

   on Apr 26, 2021

   ContributorAuthor

   More actions

   Comment from Andrew Lamb(alamb) @ 2021-01-20T14:26:14.792+0000:

   [~m18e] I can try and take a look at fixing this -- do you have a reproducer (e.g. the input file) easily at hand?

   Comment from Max Burke(m18e) @ 2021-01-21T19:36:59.503+0000:

   Yup. I've attached a test file. For what it's worth, this is the change that we've applied locally to work around it: [https://github.com/urbanlogiq/arrow/commit/9be88cf2994fe55ae0d2f5ae137b9e73daac1ef0] 
   
    
   
    
   
    [^0100c909-2537-c4dc-ce1d-1b7a75d613e8.parquet]
3. andygrove

   added

   bugSomething isn't workingSomething isn't working

   on May 1, 2021
4. alamb commented on Feb 2, 2023

   

   alamb

   on Feb 2, 2023

   ContributorAuthor

   More actions

   This query appears to be working as of DataFusion 17.0.0 in datafusion-cli -- FYI @maxburke

   ❯ SELECT * FROM '0100c909-2537-c4dc-ce1d-1b7a75d613e8.parquet' WHERE ("timestamp" >= to_timestamp('2010-03-24T13:00:00.000000Z') AND "timestamp" <= to_timestamp('2010-03-25T00:00:00.000000Z')) ORDER BY timestamp ASC NULLS LAST;
   0 rows in set. Query took 0.026 seconds.
   ❯ SELECT * FROM '0100c909-2537-c4dc-ce1d-1b7a75d613e8.parquet' limit 10;
   +--------------------------+----------+-------+-----------------------------------------------+-----------+
   | ul_node_id               | id       | count | timestamp                                     | direction |
   +--------------------------+----------+-------+-----------------------------------------------+-----------+
   | x5uF7gCs0N2UvC4icn2y7g== | 11715577 | 0     | 2011-09-09T07:00:00 (Unknown Time Zone 'UTC') | EB        |
   | x5uF7gCs0N2UvC4icn2y7g== | 11715578 | 0     | 2011-09-09T08:00:00 (Unknown Time Zone 'UTC') | EB        |
   | x5uF7gCs0N2UvC4icn2y7g== | 11715579 | 0     | 2011-09-09T09:00:00 (Unknown Time Zone 'UTC') | EB        |
   | x5uF7gCs0N2UvC4icn2y7g== | 11715580 | 0     | 2011-09-09T10:00:00 (Unknown Time Zone 'UTC') | EB        |
   | x5uF7gCs0N2UvC4icn2y7g== | 11715581 | 0     | 2011-09-09T11:00:00 (Unknown Time Zone 'UTC') | EB        |
   | x5uF7gCs0N2UvC4icn2y7g== | 11715582 | 0     | 2011-09-09T12:00:00 (Unknown Time Zone 'UTC') | EB        |
   | x5uF7gCs0N2UvC4icn2y7g== | 11715583 | 0     | 2011-09-09T13:00:00 (Unknown Time Zone 'UTC') | EB        |
   | x5uF7gCs0N2UvC4icn2y7g== | 11715584 | 0     | 2011-09-09T14:00:00 (Unknown Time Zone 'UTC') | EB        |
   | x5uF7gCs0N2UvC4icn2y7g== | 11715585 | 0     | 2011-09-09T15:00:00 (Unknown Time Zone 'UTC') | EB        |
   | x5uF7gCs0N2UvC4icn2y7g== | 11715586 | 0     | 2011-09-09T16:00:00 (Unknown Time Zone 'UTC') | EB        |
   +--------------------------+----------+-------+-----------------------------------------------+-----------+
   10 rows in set. Query took 0.005 seconds.
   
5. alamb

   closed this as completedon Feb 2, 2023
6. unkloud

   added a commit that references this issue on Mar 23, 2025

   build: Support CI pipelines for Spark 3.2, 3.3 and 3.4 (apache#153)

   a3dd8df
7. HairstonE

   added a commit that references this issue on Oct 7, 2025

   Add DataFrame methods for accessing plans (apache#153)

   0e74732