You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 00a55ed
Browse filesBrowse the repository at this point in the historyBrowse files
Copy file name to clipboardExpand all lines: docs/source/contributor-guide/expression-audits/conditional_funcs.md
+3Lines changed: 3 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -27,13 +27,15 @@
27
27
- Spark 3.5.8 (audited 2026-05-27): baseline. `Coalesce(children) extends Expression with ComplexTypeMergingExpression`; returns the first non-null child, evaluated left-to-right with short-circuit. Result type is the merged child type. Comet routes via `CometCoalesce`, which serialises as nested `CaseWhen(IsNotNull(c1) -> c1, IsNotNull(c2) -> c2, ..., else cN)` so the native engine preserves the short-circuit semantics.
28
28
- Spark 4.0.1 (audited 2026-05-27): byte-for-byte identical to 3.5.8.
29
29
- Spark 4.1.1 (audited 2026-05-27): byte-for-byte identical to 3.5.8.
30
+
- Performance (tuned 2026-09-28, PR [#6350](https://github.com/apache/datafusion-comet/pull/6350)): runs through the `CASE WHEN` evaluation described under `when`. 12-13x faster for two columns. Benchmark: `benches/conditional.rs`.
30
31
31
32
## if
32
33
33
34
- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
34
35
- Spark 3.5.8 (audited 2026-05-27): baseline. `If(predicate, trueValue, falseValue) extends ComplexTypeMergingExpression`; standard ternary semantics with short-circuit, predicate must be `BooleanType`. Comet routes via `CometIf` to the native `IfExpr` proto.
35
36
- Spark 4.0.1 (audited 2026-05-27): adds an `override def withNewAlwaysEvaluatedInputs(...)` hook for the new optimizer phase; semantics unchanged. Error messages reformatted (`paramIndex` -> `ordinalNumber`).
36
37
- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
38
+
- Performance (tuned 2026-09-28, PR [#6350](https://github.com/apache/datafusion-comet/pull/6350)): runs through the `CASE WHEN` evaluation described under `when`, with an `IF` or `CASE WHEN` in the ELSE flattened into one merge. 1.5x faster with literal results, 6.6x with column results, 3.3-4.7x with string columns, 3.1-5.9x nested. Benchmark: `benches/conditional.rs`.
37
39
38
40
## ifnull
39
41
@@ -69,5 +71,6 @@
69
71
- Spark 3.5.8 (audited 2026-05-27): the `CASE WHEN ... THEN ...` SQL form lowers to `CaseWhen(branches: Seq[(Expression, Expression)], elseValue: Option[Expression])`. Spark evaluates left-to-right with short-circuit; result type is the merged branch type. Comet routes via `CometCaseWhen` to the native `CaseWhen` proto.
- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
74
+
- Performance (tuned 2026-09-28, PR [#6350](https://github.com/apache/datafusion-comet/pull/6350)): when no branch value and no WHEN after the first can fail, `CaseWhenExpr` evaluates them over the whole batch and takes each row's value with typed merges for primitive, boolean, string and binary results, instead of filtering the batch for each branch and merging the partial results with `MutableArrayData`. A branch is no longer wrapped in a cast to the type it already has, which hid literals and columns from DataFusion's fast paths. 2.8-4x faster with literal results and 2.2-7.8x with column results. When a branch can fail it is about 2% slower, because DataFusion's `CaseExpr` derives the result type of a bare branch for every batch. Benchmark: `benches/conditional.rs`.
These counters belong to the local TopK. An attached predicate may prune nothing, and a file's
122
+
schema adaptation can disable it to preserve conversion errors. Reader work remains on the scan:
123
+
`output_rows` counts rows emitted by the scan, `bytes_scanned` measures requested data/Bloom-filter
124
+
ranges (excluding footer and page-index reads), and page and decoder filtering use the existing
125
+
`page_index_rows_pruned` and `pushdown_rows_pruned` counters.
126
+
127
+
Row groups skipped by runtime pruning within a file increment `row_groups_pruned_dynamic_filter`.
128
+
When later files open, the reader can prune using the TopK threshold already available; those groups
129
+
increment `row_groups_pruned_statistics`. With one row group per file, the dynamic counter can remain
130
+
zero even when TopK skips most later groups. The statistics counter also includes pruning by other
131
+
predicates, so compare both counters and `bytes_scanned` with filtering disabled to assess TopK savings.
132
+
This feature adds no separate decoded-batch filter or evaluated/pruned row counters to the TopK.
133
+
110
134
### Exchange
111
135
112
136
Comet adds some additional metrics:
@@ -185,6 +209,7 @@ execution metrics. Counters accumulate per scan operator; they do not instrument
185
209
|`scan_io_object_store_response_bytes_read`| Response bytes actually consumed at that API, including bytes fetched between coalesced ranges. Not HTTP wire bytes. |
186
210
|`scan_io_metadata_cache_hits`| Successful, cache-eligible metadata opens requiring no storage reads. |
187
211
|`scan_io_metadata_cache_misses`| Successful, cache-eligible metadata opens requiring storage reads. Failed opens and encrypted opens, which bypass this shared cache, increment neither cache counter. |
212
+
|`row_groups_pruned_dynamic_filter`| Row groups skipped by a runtime predicate whose value changes during execution, such as a local TopK threshold. |
188
213
189
214
Reader-level and object-store bytes are two views of the same reads; do not add them together.
190
215
Likewise, footer bytes are a subset of metadata bytes, not a third reader-level category. A warm
0 commit comments