You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 93b7cf4
Browse filesBrowse the repository at this point in the historyBrowse files
#6471 makes array_distinct and array_union fall back for any element type
that holds a non-UTF8_BINARY collated string, and checks that before the
float gate. That covers the float-plus-collated-string case this branch
checked on its own, so ArraySetSupport keeps #6471's check and drops its
own collationReason. The routing test now expects #6471's reason, and the
floating-point compatibility page no longer describes the collation
fallback as specific to floats.
|`docs/source/contributor-guide/adding_a_new_expression.md`| The serde contract, support levels, when to set the return type explicitly, shimming |
36
-
|`docs/source/contributor-guide/sql-file-tests.md`| The test framework expression PRs are expected to use, and every directive it supports |
37
-
|`docs/source/contributor-guide/optimizing_expressions.md`| The benchmark workflow and the no-regression rule, for PRs that change an existing kernel |
|`docs/source/contributor-guide/adding_a_new_expression.md`| The serde contract, support levels, when to set the return type explicitly, shimming |
36
+
|`docs/source/contributor-guide/sql-file-tests.md`| The test framework expression PRs are expected to use, and every directive it supports |
37
+
|`docs/source/contributor-guide/optimizing_expressions.md`| The benchmark workflow and the no-regression rule, for PRs that change an existing kernel |
38
+
|`docs/source/contributor-guide/timezones.md`| The timestamp label rule, where the session timezone comes from, and how to test it, for datetime expressions, casts, and anything that returns a timestamp |
38
39
39
40
Hold the diff against these. If the PR does something a guide says to do differently, either the PR
40
41
is wrong or the guide is out of date, and you need to say which.
ordering and rendering for float, double, timestamp, timestamptz, binary and decimal.
114
+
-[ ]**Timestamp partition values are UTC.** Iceberg's `years`, `months`, `days` and `hours`
115
+
ignore the session timezone. For date and timestamp sources, `PartitionValueCalculator`
116
+
(`iceberg_partition_value.rs`) computes them with the same kernels in
117
+
`iceberg_funcs/temporal.rs` that key the sort in front of a clustered write, and those follow
118
+
iceberg-java's `DateTimeUtil`, including its pre-1970 rounding. A change that moves one of
119
+
these transforms back to iceberg-rust, or that makes the kernels read the column's timezone
120
+
label, breaks that agreement. iceberg-rust's `years` and `months` follow the label
121
+
([apache/iceberg-rust#3142](https://github.com/apache/iceberg-rust/issues/3142)). Ask for a
122
+
test in a non-UTC session for a change here. "Partition values" in `iceberg-writes.md` lists
123
+
where iceberg-rust differs, and `timezones.md` covers how Comet labels timestamps.
114
124
-[ ]**Partition paths use the Java renderers.** Directory names come from
115
125
`CometLocationGenerator` / `partition_to_path` in `iceberg_partition_path.rs`, which follow
116
126
iceberg-java's `partitionToPath`, with `java_float_string` for floats. A PR that calls
@@ -142,6 +152,14 @@ Read the ownership table in the contributor guide before reviewing any change ne
142
152
`SparkWrite.abort` keeps working.
143
153
-[ ] The user-visible exception type is still what Spark's own write path throws on each Spark
144
154
version ([#6143](https://github.com/apache/datafusion-comet/issues/6143)).
155
+
-[ ]**Every open file and every held-back row is reserved.** Files must be opened through
156
+
`MeteredParquetWriterBuilder`, and rows the writer holds outside iceberg-rust (the
157
+
`PartitionFeed`s, or anything a new feed holds back) must be added to what `run_write_task` reserves. A
158
+
builder that constructs `ParquetWriterBuilder` directly leaves its files out of the task's
159
+
memory reservation, so a wide fanout write grows past the pool instead of failing its task.
160
+
A storage scheme newly supported for writes needs its entry in
161
+
`StorageWrites::for_location`: a file reports its flushed row groups until its storage has
162
+
written them out, which a local file does at once and an object store only part by part.
145
163
146
164
## 5. Plan Shape
147
165
@@ -179,16 +197,16 @@ Read the ownership table in the contributor guide before reviewing any change ne
For such a PR, hold the diff against "The invariant" and "Guidelines" in
197
+
`docs/source/contributor-guide/timezones.md`. Look for these first:
198
+
199
+
- A `TimestampType` value labelled with the session timezone, or with no timezone. Inside a native
200
+
plan every `TimestampType` value is labelled exactly `"UTC"`, and every `TimestampNTZType` value
201
+
has no timezone. Check the declared type and the arrays the code builds, not only the values.
202
+
- A timezone taken from the JVM default or the host. An expression uses the `timeZoneId` Spark
203
+
stamped on it, passed through `CometTimeZone.nativeId`, not `SQLConf.get.sessionLocalTimeZone`.
204
+
- A path gated on a UTC session. `Etc/UTC` is the session default on Ubuntu and Debian images, so
205
+
check what the gate does with it, and that the output there is still labelled `"UTC"` rather than
206
+
`"Etc/UTC"`.
207
+
- A timezone applied to a `TimestampNTZType` value, other than to convert it to `TimestampType`.
208
+
- Tests that use a single session timezone, or only project the result. "Testing timezone-sensitive
209
+
code" in the same page says what to ask for.
210
+
211
+
`timezones.md` also describes specific code: where the `"UTC"` label is set, which serdes serialize
212
+
a timezone, how `CometTimeZone` rewrites timezone IDs, what `array_with_timezone` does, how the
213
+
scans adapt timestamps, and which expressions go through the codegen dispatcher. A PR that changes
214
+
any of these updates the page in the same PR. The page also documents some known limitations as
215
+
current behavior, such as chrono-tz's DST horizon and the timezone database versions. A fix for one
216
+
of the bugs tracked in [#6335](https://github.com/apache/datafusion-comet/issues/6335) usually
217
+
changes that text too.
218
+
179
219
### Configuration
180
220
181
221
New configs go in `CometConf.scala` and must follow
Copy file name to clipboardExpand all lines: .github/ISSUE_TEMPLATE/bug_report.yml
+9Lines changed: 9 additions & 0 deletions
Original file line number
Diff line number
Diff line change
@@ -42,3 +42,12 @@ body:
42
42
label: Additional context
43
43
placeholder: >
44
44
Add any other context about the problem here.
45
+
- type: dropdown
46
+
attributes:
47
+
label: Willingness to contribute
48
+
description: >
49
+
The Apache DataFusion Comet community encourages bug fix contributions. Would you or another member of your organization be willing to contribute a fix for this bug to the Apache DataFusion Comet codebase?
50
+
options:
51
+
- I can contribute a fix for this bug independently
52
+
- I would be willing to contribute a fix for this bug with guidance from the DataFusion community
53
+
- I cannot contribute a fix for this bug at this time
Copy file name to clipboardExpand all lines: .github/ISSUE_TEMPLATE/feature_request.yml
+9Lines changed: 9 additions & 0 deletions
Original file line number
Diff line number
Diff line change
@@ -36,3 +36,12 @@ body:
36
36
label: Additional context
37
37
placeholder: >
38
38
Add any other context or screenshots about the feature request here.
39
+
- type: dropdown
40
+
attributes:
41
+
label: Willingness to contribute
42
+
description: >
43
+
The Apache DataFusion Comet community encourages bug fix contributions. Would you or another member of your organization be willing to contribute a fix for this bug to the Apache DataFusion Comet codebase?
44
+
options:
45
+
- I can contribute a fix for this bug independently
46
+
- I would be willing to contribute a fix for this bug with guidance from the DataFusion community
47
+
- I cannot contribute a fix for this bug at this time
0 commit comments