Skip to content

Cannot import Time64 from CSV #3176

New issue
New issue
Closed
Closed
Cannot import Time64 from CSV#3176
Labels
bugSomething isn't working

Description

@waitingkuo
waitingkuo
opened on Aug 16, 2022

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

create this time.csv

c0,c1,c2
1,00:00:00,2000-01-01T00:00:00
2,00:01:00,2000-01-01T00:00:00
3,00:02:00,2000-01-01T00:00:00

the query for time doesn't work while timestamp works

DataFusion CLI v10.0.0
❯ create external table a (c0 int, c1 time, c2 timestamp) stored as csv with header row location 'time.csv';
0 rows in set. Query took 0.020 seconds.

❯ select c0 from a;
+----+
| c0 |
+----+
| 1  |
| 2  |
| 3  |
+----+
3 rows in set. Query took 0.039 seconds.

❯ select c1 from a;
ArrowError(ParseError("Unsupported data type Time64(Nanosecond)"))

❯ select c2 from a;
+---------------------+
| c2                  |
+---------------------+
| 1999-12-31 16:00:00 |
| 1999-12-31 16:00:00 |
| 1999-12-31 16:00:00 |
+---------------------+
3 rows in set. Query took 0.009 seconds.

Expected behavior
A clear and concise description of what you expected to happen.

Additional context
Add any other context about the problem here.

Activity

  1. waitingkuo
    added
    bugSomething isn't working
    on Aug 16, 2022

  2. Jefffrey commented on Nov 15, 2022

    @Jefffrey
    Jefffrey
    on Nov 15, 2022
    Contributor

    This should be fixed by apache/arrow-rs#3101 with the related arrow-rs version

  3. waitingkuo commented on Nov 18, 2022

    @waitingkuo
    waitingkuo
    on Nov 18, 2022
    ContributorAuthor

    thank you @Jefffrey i think we could add some test cases later after arrow-rs new version upgraded
    btw, i found the behavior for Timestamp parsing changed
    it's now

    ❯ select c2 from a;
+---------------------+
| c2                  |
+---------------------+
| 2000-01-01T00:00:00 |
| 2000-01-01T00:00:00 |
| 2000-01-01T00:00:00 |
+---------------------+

  4. Jefffrey commented on Dec 4, 2022

    @Jefffrey
    Jefffrey
    on Dec 4, 2022
    Contributor

    Issue resolved with arrow-rs version upgraded

    jeffrey:~/Code/arrow-datafusion/datafusion-cli$ cat time.csv
c0,c1,c2
1,00:00:00,2000-01-01T00:00:00
2,00:01:00,2000-01-01T00:00:00
3,00:02:00,2000-01-01T00:00:00
jeffrey:~/Code/arrow-datafusion/datafusion-cli$ cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 0.11s
     Running `/media/jeffrey/1tb_860evo_ssd/.cargo_target_cache/debug/datafusion-cli`
DataFusion CLI v15.0.0
❯ create external table a (c0 int, c1 time, c2 timestamp) stored as csv with header row location 'time.csv';
0 rows in set. Query took 0.001 seconds.
❯ select * from a;
+----+----------+---------------------+
| c0 | c1       | c2                  |
+----+----------+---------------------+
| 1  | 00:00:00 | 2000-01-01T00:00:00 |
| 2  | 00:01:00 | 2000-01-01T00:00:00 |
| 3  | 00:02:00 | 2000-01-01T00:00:00 |
+----+----------+---------------------+
3 rows in set. Query took 0.006 seconds.
❯

    Would you still prefer some test cases be added, say to datafusion/core/src/datasource/file_format/csv.rs for example? Because in my opinion it'd just be retesting the arrow-rs functionality

    @waitingkuo

  5. Jefffrey commented on Mar 9, 2023

    @Jefffrey
    Jefffrey
    on Mar 9, 2023
    Contributor

    @alamb this issue looks like it can be closed

  6. alamb
    closed this as completedon Mar 9, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions