Skip to content

Commit 2b746c7

Browse files
author
Evan Chan
committed
Add docs for to_timestamp() functions
1 parent 89bbf2f commit 2b746c7

3 files changed

Lines changed: 116 additions & 25 deletions

File tree

‎README.md‎

Lines changed: 29 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -197,6 +197,10 @@ DataFusion also includes a simple command-line interactive SQL utility. See the
197197
- [ ] Basic date functions
198198
- [ ] Basic time functions
199199
- [x] Basic timestamp functions
200+
- [x] [to_timestamp](docs/user-guide/book/sql/datafusion-functions.html#to_timestamp)
201+
- [x] [to_timestamp_millis](docs/user-guide/book/sql/datafusion-functions.html#to_timestamp_millis)
202+
- [x] [to_timestamp_micros](docs/user-guide/book/sql/datafusion-functions.html#to_timestamp_micros)
203+
- [x] [to_timestamp_seconds](docs/user-guide/book/sql/datafusion-functions.html#to_timestamp_seconds)
200204
- nested functions
201205
- [x] Array of columns
202206
- [x] Schema Queries
@@ -320,31 +324,31 @@ execution. The SQL types from
320324
[sqlparser-rs](https://github.com/ballista-compute/sqlparser-rs/blob/main/src/ast/data_type.rs#L57)
321325
are mapped to Arrow types according to the following table
322326

323-
| SQL Data Type | Arrow DataType |
324-
| ------------- | ------------------------------- |
325-
| `CHAR` | `Utf8` |
326-
| `VARCHAR` | `Utf8` |
327-
| `UUID` | _Not yet supported_ |
328-
| `CLOB` | _Not yet supported_ |
329-
| `BINARY` | _Not yet supported_ |
330-
| `VARBINARY` | _Not yet supported_ |
331-
| `DECIMAL` | `Float64` |
332-
| `FLOAT` | `Float32` |
333-
| `SMALLINT` | `Int16` |
334-
| `INT` | `Int32` |
335-
| `BIGINT` | `Int64` |
336-
| `REAL` | `Float64` |
337-
| `DOUBLE` | `Float64` |
338-
| `BOOLEAN` | `Boolean` |
339-
| `DATE` | `Date32` |
340-
| `TIME` | `Time64(TimeUnit::Millisecond)` |
341-
| `TIMESTAMP` | `Date64` |
342-
| `INTERVAL` | _Not yet supported_ |
343-
| `REGCLASS` | _Not yet supported_ |
344-
| `TEXT` | _Not yet supported_ |
345-
| `BYTEA` | _Not yet supported_ |
346-
| `CUSTOM` | _Not yet supported_ |
347-
| `ARRAY` | _Not yet supported_ |
327+
| SQL Data Type | Arrow DataType |
328+
| ------------- | --------------------------------- |
329+
| `CHAR` | `Utf8` |
330+
| `VARCHAR` | `Utf8` |
331+
| `UUID` | _Not yet supported_ |
332+
| `CLOB` | _Not yet supported_ |
333+
| `BINARY` | _Not yet supported_ |
334+
| `VARBINARY` | _Not yet supported_ |
335+
| `DECIMAL` | `Float64` |
336+
| `FLOAT` | `Float32` |
337+
| `SMALLINT` | `Int16` |
338+
| `INT` | `Int32` |
339+
| `BIGINT` | `Int64` |
340+
| `REAL` | `Float64` |
341+
| `DOUBLE` | `Float64` |
342+
| `BOOLEAN` | `Boolean` |
343+
| `DATE` | `Date32` |
344+
| `TIME` | `Time64(TimeUnit::Millisecond)` |
345+
| `TIMESTAMP` | `Timestamp(TimeUnit::Nanosecond)` |
346+
| `INTERVAL` | _Not yet supported_ |
347+
| `REGCLASS` | _Not yet supported_ |
348+
| `TEXT` | _Not yet supported_ |
349+
| `BYTEA` | _Not yet supported_ |
350+
| `CUSTOM` | _Not yet supported_ |
351+
| `ARRAY` | _Not yet supported_ |
348352

349353
# Architecture Overview
350354

‎docs/user-guide/src/SUMMARY.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,7 @@
2727
- [SELECT](sql/select.md)
2828
- [DDL](sql/ddl.md)
2929
- [CREATE EXTERNAL TABLE](sql/ddl.md)
30+
- [Datafusion Specific Functions](sql/datafusion-functions.md)
3031

3132
- [Distributed](distributed/introduction.md)
3233
- [Create a Ballista Cluster](distributed/deployment.md)
Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,86 @@
1+
<!---
2+
Licensed to the Apache Software Foundation (ASF) under one
3+
or more contributor license agreements. See the NOTICE file
4+
distributed with this work for additional information
5+
regarding copyright ownership. The ASF licenses this file
6+
to you under the Apache License, Version 2.0 (the
7+
"License"); you may not use this file except in compliance
8+
with the License. You may obtain a copy of the License at
9+
10+
http://www.apache.org/licenses/LICENSE-2.0
11+
12+
Unless required by applicable law or agreed to in writing,
13+
software distributed under the License is distributed on an
14+
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
KIND, either express or implied. See the License for the
16+
specific language governing permissions and limitations
17+
under the License.
18+
-->
19+
20+
# Datafusion-Specific Functions
21+
22+
These SQL functions are specific to DataFusion, or they are well known and have functionality which is specific to DataFusion. Specifically, the `to_timestamp_xx()` functions exist due to Arrow's support for multiple timestamp resolutions.
23+
24+
## `to_timestamp`
25+
26+
`to_timestamp()` is similar to the standard SQL function. It performs conversions to type `Timestamp(Nanoseconds, None)`, from:
27+
28+
- Timestamp strings
29+
- `1997-01-31T09:26:56.123Z` # RCF3339
30+
- `1997-01-31T09:26:56.123-05:00` # RCF3339
31+
- `1997-01-31 09:26:56.123-05:00` # close to RCF3339 but with a space er than T
32+
- `1997-01-31T09:26:56.123` # close to RCF3339 but no timezone et specified
33+
- `1997-01-31 09:26:56.123` # close to RCF3339 but uses a space and timezone offset
34+
- `1997-01-31 09:26:56` # close to RCF3339, no fractional seconds
35+
- An Int64 array/column, values are nanoseconds since Epoch UTC
36+
- Other Timestamp() columns or values
37+
38+
Note that conversions from other Timestamp and Int64 types can also be performed using `CAST(.. AS Timestamp)`. However, the conversion functionality here is present for consistency with the other `to_timestamp_xx()` functions.
39+
40+
## `to_timestamp_millis`
41+
42+
`to_timestamp_millis()` does conversions to type `Timestamp(Milliseconds, None)`, from:
43+
44+
- Timestamp strings, the same as supported by the regular timestamp() function (except the output is a timestamp of Milliseconds resolution)
45+
- `1997-01-31T09:26:56.123Z` # RCF3339
46+
- `1997-01-31T09:26:56.123-05:00` # RCF3339
47+
- `1997-01-31 09:26:56.123-05:00` # close to RCF3339 but with a space er than T
48+
- `1997-01-31T09:26:56.123` # close to RCF3339 but no timezone et specified
49+
- `1997-01-31 09:26:56.123` # close to RCF3339 but uses a space and timezone offset
50+
- `1997-01-31 09:26:56` # close to RCF3339, no fractional seconds
51+
- An Int64 array/column, values are milliseconds since Epoch UTC
52+
- Other Timestamp() columns or values
53+
54+
Note that `CAST(.. AS Timestamp)` converts to Timestamps with Nanosecond resolution; this function is the only way to convert/cast to millisecond resolution.
55+
56+
## `to_timestamp_micros`
57+
58+
`to_timestamp_micros()` does conversions to type `Timestamp(Microseconds, None)`, from:
59+
60+
- Timestamp strings, the same as supported by the regular timestamp() function (except the output is a timestamp of microseconds resolution)
61+
- `1997-01-31T09:26:56.123Z` # RCF3339
62+
- `1997-01-31T09:26:56.123-05:00` # RCF3339
63+
- `1997-01-31 09:26:56.123-05:00` # close to RCF3339 but with a space er than T
64+
- `1997-01-31T09:26:56.123` # close to RCF3339 but no timezone et specified
65+
- `1997-01-31 09:26:56.123` # close to RCF3339 but uses a space and timezone offset
66+
- `1997-01-31 09:26:56` # close to RCF3339, no fractional seconds
67+
- An Int64 array/column, values are microseconds since Epoch UTC
68+
- Other Timestamp() columns or values
69+
70+
Note that `CAST(.. AS Timestamp)` converts to Timestamps with Nanosecond resolution; this function is the only way to convert/cast to microsecond resolution.
71+
72+
## `to_timestamp_seconds`
73+
74+
`to_timestamp_seconds()` does conversions to type `Timestamp(Seconds, None)`, from:
75+
76+
- Timestamp strings, the same as supported by the regular timestamp() function (except the output is a timestamp of secondseconds resolution)
77+
- `1997-01-31T09:26:56.123Z` # RCF3339
78+
- `1997-01-31T09:26:56.123-05:00` # RCF3339
79+
- `1997-01-31 09:26:56.123-05:00` # close to RCF3339 but with a space er than T
80+
- `1997-01-31T09:26:56.123` # close to RCF3339 but no timezone et specified
81+
- `1997-01-31 09:26:56.123` # close to RCF3339 but uses a space and timezone offset
82+
- `1997-01-31 09:26:56` # close to RCF3339, no fractional seconds
83+
- An Int64 array/column, values are seconds since Epoch UTC
84+
- Other Timestamp() columns or values
85+
86+
Note that `CAST(.. AS Timestamp)` converts to Timestamps with Nanosecond resolution; this function is the only way to convert/cast to seconds resolution.

0 commit comments

Comments
 (0)