Skip to content

Commit 93b7cf4

Browse files
committed
Merge apache/main into array-set-native-normalize
#6471 makes array_distinct and array_union fall back for any element type that holds a non-UTF8_BINARY collated string, and checks that before the float gate. That covers the float-plus-collated-string case this branch checked on its own, so ArraySetSupport keeps #6471's check and drops its own collationReason. The routing test now expects #6471's reason, and the floating-point compatibility page no longer describes the collation fallback as specific to floats.
2 parents 35440ba + f80042f commit 93b7cf4

251 files changed

Lines changed: 25680 additions & 1667 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.ai/skills/review-comet-expression-pr/SKILL.md‎

Lines changed: 23 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -30,11 +30,12 @@ bar, and the output format. This skill only covers what is specific to expressio
3030

3131
## Read the Contributor Guide First
3232

33-
| Doc | What you need from it |
34-
| ---------------------------------------------------------- | ----------------------------------------------------------------------------------------- |
35-
| `docs/source/contributor-guide/adding_a_new_expression.md` | The serde contract, support levels, when to set the return type explicitly, shimming |
36-
| `docs/source/contributor-guide/sql-file-tests.md` | The test framework expression PRs are expected to use, and every directive it supports |
37-
| `docs/source/contributor-guide/optimizing_expressions.md` | The benchmark workflow and the no-regression rule, for PRs that change an existing kernel |
33+
| Doc | What you need from it |
34+
| ---------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- |
35+
| `docs/source/contributor-guide/adding_a_new_expression.md` | The serde contract, support levels, when to set the return type explicitly, shimming |
36+
| `docs/source/contributor-guide/sql-file-tests.md` | The test framework expression PRs are expected to use, and every directive it supports |
37+
| `docs/source/contributor-guide/optimizing_expressions.md` | The benchmark workflow and the no-regression rule, for PRs that change an existing kernel |
38+
| `docs/source/contributor-guide/timezones.md` | The timestamp label rule, where the session timezone comes from, and how to test it, for datetime expressions, casts, and anything that returns a timestamp |
3839

3940
Hold the diff against these. If the PR does something a guide says to do differently, either the PR
4041
is wrong or the guide is out of date, and you need to say which.
@@ -77,6 +78,10 @@ Location: `spark/src/main/scala/org/apache/comet/serde/`
7778
- [ ] A change that routes a whole class of expressions to the JVM codegen dispatcher lists the new
7879
shapes it admits and tests one of each, including decimal results whose scale differs from the
7980
declared type and boolean inputs from sliced batches (#6424, #6425)
81+
- [ ] A timezone-aware expression serializes `CometTimeZone.nativeId(expr.timeZoneId)`, the
82+
timezone Spark stamped on it in a form native code can parse, rather than
83+
`SQLConf.get.sessionLocalTimeZone` or the JVM default, and returns
84+
`CometTimeZone.supportLevel` from `getSupportLevel` when `nativeId` gives `None`
8085

8186
### Registration in `QueryPlanSerde.scala`
8287

@@ -112,6 +117,13 @@ Location: `native/spark-expr/src/`, registered in `comet_scalar_funcs.rs`.
112117
- [ ] Comparisons, grouping and set operations treat `-0.0` and `0.0`, and NaN payloads, as Spark
113118
does, including inside arrays and structs. Where a DataFusion kernel treats them differently,
114119
the serde falls back (#5507, #5701).
120+
- [ ] A `TimestampType` result is labelled `"UTC"`, in `return_type()` or `data_type()` and in the
121+
arrays the kernel builds. Not the session timezone, and not `None`.
122+
- [ ] Local-time work uses the session timezone from the proto. DataFusion's own datetime functions
123+
take the timezone from the input's label, so wiring one in for a timezone-aware Spark
124+
expression evaluates it in UTC unless the session timezone is passed explicitly.
125+
- [ ] A UTC fast path matches a fixed list of UTC aliases, as `is_utc_timezone` in
126+
`extract_date_part.rs` does, and sends every other timezone ID down the general path
115127

116128
Before accepting a hand-written kernel, ask whether the function already exists upstream in
117129
DataFusion or the `datafusion-spark` crate. Comet prefers wiring an upstream function over carrying
@@ -169,8 +181,10 @@ single file by appending a substring of its name to the suite argument.
169181
- [ ] Edge cases tested: empty input, overflow, boundary values, negative values
170182
- [ ] Both literal and column arguments tested, in every combination for multi-argument
171183
expressions. They take different code paths.
172-
- [ ] Timezone handling tested for timestamp and datetime expressions, including a non-UTC session
173-
timezone and timestamps with and without timezone
184+
- [ ] Timestamp and datetime expressions are tested as "Testing timezone-sensitive code" in
185+
`timezones.md` describes. That means several session timezones through `ConfigMatrix`,
186+
including `Etc/UTC` and a zone with DST, both `TIMESTAMP` and `TIMESTAMP_NTZ` inputs, and a
187+
result that is compared or fed to another expression rather than only projected.
174188
- [ ] SQL syntax gated with `MinSparkVersion` when it only parses on newer Spark
175189
- [ ] For a function from another project, such as an Iceberg transform, the expected values come
176190
from that project's Java implementation, which is what Spark runs, not from iceberg-rust
@@ -254,3 +268,5 @@ reference in that doc is the only place they are documented.
254268
5. **Missing `getSupportLevel`**, divergences left undeclared rather than marked `Incompatible`
255269
6. **Version-specific Spark behavior implemented once**, with no shim
256270
7. **Name collides with a DataFusion built-in** and no explicit return type
271+
8. **Timestamp result mislabelled**, with the session timezone or no timezone instead of `"UTC"`.
272+
It only shows once the result is compared or fed to another expression.

‎.ai/skills/review-comet-ffi-pr/SKILL.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -138,6 +138,11 @@ a new subclass needs no new case as long as `getValueVector` returns an Arrow ve
138138
`ColumnarBatchArrowReader` decodes dictionary-encoded columns before export, and
139139
`reconcileStreamSchema` advertises the value type. A new reader that exports a dictionary
140140
would reach `ScanExec` as is, and only the cast in `build_record_batch` would unpack it.
141+
- [ ] **Timestamps cross unconverted.** Both directions pass the raw microseconds. JVM producers
142+
label `TimestampType` with `CometArrowStream.NATIVE_TIMEZONE`, which is `"UTC"`, and a new
143+
producer must use it too rather than the session timezone. A change that shifts values by a
144+
timezone at the boundary gives wrong answers. See "How Comet represents timestamps" in
145+
`docs/source/contributor-guide/timezones.md`.
141146

142147
## 5. Memory Accounting
143148

‎.ai/skills/review-comet-iceberg-write-pr/SKILL.md‎

Lines changed: 28 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -111,6 +111,16 @@ only the latest.
111111
`Float.compare` does not
112112
([#6138](https://github.com/apache/datafusion-comet/issues/6138)). Check equality, hashing,
113113
ordering and rendering for float, double, timestamp, timestamptz, binary and decimal.
114+
- [ ] **Timestamp partition values are UTC.** Iceberg's `years`, `months`, `days` and `hours`
115+
ignore the session timezone. For date and timestamp sources, `PartitionValueCalculator`
116+
(`iceberg_partition_value.rs`) computes them with the same kernels in
117+
`iceberg_funcs/temporal.rs` that key the sort in front of a clustered write, and those follow
118+
iceberg-java's `DateTimeUtil`, including its pre-1970 rounding. A change that moves one of
119+
these transforms back to iceberg-rust, or that makes the kernels read the column's timezone
120+
label, breaks that agreement. iceberg-rust's `years` and `months` follow the label
121+
([apache/iceberg-rust#3142](https://github.com/apache/iceberg-rust/issues/3142)). Ask for a
122+
test in a non-UTC session for a change here. "Partition values" in `iceberg-writes.md` lists
123+
where iceberg-rust differs, and `timezones.md` covers how Comet labels timestamps.
114124
- [ ] **Partition paths use the Java renderers.** Directory names come from
115125
`CometLocationGenerator` / `partition_to_path` in `iceberg_partition_path.rs`, which follow
116126
iceberg-java's `partitionToPath`, with `java_float_string` for floats. A PR that calls
@@ -142,6 +152,14 @@ Read the ownership table in the contributor guide before reviewing any change ne
142152
`SparkWrite.abort` keeps working.
143153
- [ ] The user-visible exception type is still what Spark's own write path throws on each Spark
144154
version ([#6143](https://github.com/apache/datafusion-comet/issues/6143)).
155+
- [ ] **Every open file and every held-back row is reserved.** Files must be opened through
156+
`MeteredParquetWriterBuilder`, and rows the writer holds outside iceberg-rust (the
157+
`PartitionFeed`s, or anything a new feed holds back) must be added to what `run_write_task` reserves. A
158+
builder that constructs `ParquetWriterBuilder` directly leaves its files out of the task's
159+
memory reservation, so a wide fanout write grows past the pool instead of failing its task.
160+
A storage scheme newly supported for writes needs its entry in
161+
`StorageWrites::for_location`: a file reports its flushed row groups until its storage has
162+
written them out, which a local file does at once and an object store only part by part.
145163

146164
## 5. Plan Shape
147165

@@ -179,16 +197,16 @@ Read the ownership table in the contributor guide before reviewing any change ne
179197

180198
## 7. Tests
181199

182-
| Suite | Covers |
183-
| ----------------------------------- | ------------------------------------------------------------------------------------ |
184-
| `CometIcebergWriteActionSuite` | Both layers end to end, parity with iceberg-java, DML, failure cleanup, AQE re-plans |
185-
| `CometIcebergWriteDetectionSuite` | One case per eligibility rule |
186-
| `CometIcebergSystemFunctionSuite` | Native partition transforms keeping partitioned writes native |
187-
| `CometIcebergRewriteActionSuite` | `rewrite_data_files` through the split plan and the native writer |
188-
| `IcebergWriteProtoTranslationSuite` | Property to `IcebergParquetWriteSettings` translation |
189-
| Rust tests in `iceberg_write.rs` | Rolling, fanout order, clustered checks, cleanup guard, manifest round trip |
190-
| `iceberg_partition_path.rs` tests | Partition path rendering against iceberg-java |
191-
| `CometIcebergWriteBenchmark` | Native versus iceberg-java throughput |
200+
| Suite | Covers |
201+
| ----------------------------------- | ----------------------------------------------------------------------------------------------- |
202+
| `CometIcebergWriteActionSuite` | Both layers end to end, parity with iceberg-java, DML, failure cleanup, AQE re-plans |
203+
| `CometIcebergWriteDetectionSuite` | One case per eligibility rule |
204+
| `CometIcebergSystemFunctionSuite` | Native partition transforms keeping partitioned writes native |
205+
| `CometIcebergRewriteActionSuite` | `rewrite_data_files` through the split plan and the native writer |
206+
| `IcebergWriteProtoTranslationSuite` | Property to `IcebergParquetWriteSettings` translation |
207+
| Rust tests in `iceberg_write.rs` | Rolling, fanout order, clustered checks, cleanup guard, manifest round trip, memory reservation |
208+
| `iceberg_partition_path.rs` tests | Partition path rendering against iceberg-java |
209+
| `CometIcebergWriteBenchmark` | Native versus iceberg-java throughput |
192210

193211
Ask specifically:
194212

‎.ai/skills/review-comet-pr/SKILL.md‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,11 @@ change.
4747
For a new operator, read `docs/source/contributor-guide/adding_a_new_operator.md` alongside this
4848
skill. There is no dedicated operator review skill yet.
4949

50+
For a PR that deals with timestamps or the session timezone, read
51+
`docs/source/contributor-guide/timezones.md` as well, whichever areas it touches. Timezone handling
52+
cuts across the serde, the native kernels, the scans and the JVM/native boundary, so no area skill
53+
owns it. "Timestamps and timezones" in step 5 says what to check.
54+
5055
If the PR falls outside all of these, for example build, CI, docs only, or release tooling, this
5156
skill on its own is the review.
5257

@@ -176,6 +181,41 @@ the minor version is wrong for every earlier patch. CI builds only the newest pa
176181
it can't catch that (#6042, #5701). The pull request CI also runs only the default Spark profile, so
177182
logic that depends on the Spark version needs the matching `run-spark-*` labels (#6156).
178183

184+
### Timestamps and timezones
185+
186+
Timezone bugs are easy to miss in review and in tests. A mislabelled timestamp column passes any
187+
test that only projects it, and a result computed in the wrong timezone looks plausible. A PR deals
188+
with timezones if it touches a datetime expression, a cast to or from a timestamp, how a scan reads
189+
timestamps, the timestamp type at the JVM/native boundary, or anything that reads the session
190+
timezone. Searching the diff flags most of these PRs:
191+
192+
```shell
193+
gh pr diff <pr> --repo apache/datafusion-comet | grep -inE 'time_?zone|zoneid|chrono_tz|timestamp(ntz)?type|timestampmicro|timestamp\('
194+
```
195+
196+
For such a PR, hold the diff against "The invariant" and "Guidelines" in
197+
`docs/source/contributor-guide/timezones.md`. Look for these first:
198+
199+
- A `TimestampType` value labelled with the session timezone, or with no timezone. Inside a native
200+
plan every `TimestampType` value is labelled exactly `"UTC"`, and every `TimestampNTZType` value
201+
has no timezone. Check the declared type and the arrays the code builds, not only the values.
202+
- A timezone taken from the JVM default or the host. An expression uses the `timeZoneId` Spark
203+
stamped on it, passed through `CometTimeZone.nativeId`, not `SQLConf.get.sessionLocalTimeZone`.
204+
- A path gated on a UTC session. `Etc/UTC` is the session default on Ubuntu and Debian images, so
205+
check what the gate does with it, and that the output there is still labelled `"UTC"` rather than
206+
`"Etc/UTC"`.
207+
- A timezone applied to a `TimestampNTZType` value, other than to convert it to `TimestampType`.
208+
- Tests that use a single session timezone, or only project the result. "Testing timezone-sensitive
209+
code" in the same page says what to ask for.
210+
211+
`timezones.md` also describes specific code: where the `"UTC"` label is set, which serdes serialize
212+
a timezone, how `CometTimeZone` rewrites timezone IDs, what `array_with_timezone` does, how the
213+
scans adapt timestamps, and which expressions go through the codegen dispatcher. A PR that changes
214+
any of these updates the page in the same PR. The page also documents some known limitations as
215+
current behavior, such as chrono-tz's DST horizon and the timezone database versions. A fix for one
216+
of the bugs tracked in [#6335](https://github.com/apache/datafusion-comet/issues/6335) usually
217+
changes that text too.
218+
179219
### Configuration
180220

181221
New configs go in `CometConf.scala` and must follow

‎.github/ISSUE_TEMPLATE/bug_report.yml‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,3 +42,12 @@ body:
4242
label: Additional context
4343
placeholder: >
4444
Add any other context about the problem here.
45+
- type: dropdown
46+
attributes:
47+
label: Willingness to contribute
48+
description: >
49+
The Apache DataFusion Comet community encourages bug fix contributions. Would you or another member of your organization be willing to contribute a fix for this bug to the Apache DataFusion Comet codebase?
50+
options:
51+
- I can contribute a fix for this bug independently
52+
- I would be willing to contribute a fix for this bug with guidance from the DataFusion community
53+
- I cannot contribute a fix for this bug at this time

‎.github/ISSUE_TEMPLATE/feature_request.yml‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,3 +36,12 @@ body:
3636
label: Additional context
3737
placeholder: >
3838
Add any other context or screenshots about the feature request here.
39+
- type: dropdown
40+
attributes:
41+
label: Willingness to contribute
42+
description: >
43+
The Apache DataFusion Comet community encourages bug fix contributions. Would you or another member of your organization be willing to contribute a fix for this bug to the Apache DataFusion Comet codebase?
44+
options:
45+
- I can contribute a fix for this bug independently
46+
- I would be willing to contribute a fix for this bug with guidance from the DataFusion community
47+
- I cannot contribute a fix for this bug at this time

‎.github/actions/rust-test/action.yaml‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -66,6 +66,15 @@ runs:
6666
run: |
6767
cargo install cargo-nextest --locked
6868
69+
- name: Build test UDF library (pre-requisite for native UDF tests)
70+
shell: bash
71+
run: |
72+
cd native
73+
# comet-test-udfs is `crate-type = ["cdylib"]` and has no test targets, so a
74+
# test build compiles it without emitting the shared library the c_udf loader
75+
# tests dlopen. Build it explicitly.
76+
cargo build -p comet-test-udfs
77+
6978
- name: Run Cargo test
7079
shell: bash
7180
run: |

0 commit comments

Comments
 (0)