You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 7c8e995
Browse filesBrowse the repository at this point in the historyBrowse files
Copy file name to clipboardExpand all lines: docs/source/contributor-guide/release_process.md
-6Lines changed: 0 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -287,9 +287,6 @@ git tag 0.13.0-rc1
287
287
git push apache 0.13.0-rc1
288
288
```
289
289
290
-
Note that pushing a release candidate tag will trigger a GitHub workflow that will build a Docker image and publish
291
-
it to GitHub Container Registry at https://github.com/apache/datafusion-comet/pkgs/container/datafusion-comet
292
-
293
290
### Publishing Documentation
294
291
295
292
In `docs` directory:
@@ -468,9 +465,6 @@ git tag 0.13.0
468
465
git push apache 0.13.0
469
466
```
470
467
471
-
Note that pushing a release tag will trigger a GitHub workflow that will build a Docker image and publish
472
-
it to GitHub Container Registry at https://github.com/apache/datafusion-comet/pkgs/container/datafusion-comet
473
-
474
468
Reply to the vote thread to close the vote and announce the release. The announcement email should include:
Copy file name to clipboardExpand all lines: docs/source/user-guide/latest/compatibility/expressions/_category_template/cast.md
+16-18Lines changed: 16 additions & 18 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -22,17 +22,23 @@ under the License.
22
22
Cast operations in Comet fall into three levels of support:
23
23
24
24
-**C (Compatible)**: The results match Apache Spark
25
-
-**I (Incompatible)**: The results may match Apache Spark for some inputs, but there are known issues where some inputs
26
-
will result in incorrect results or exceptions. The query stage will fall back to Spark by default. Setting
27
-
`spark.comet.expression.Cast.allowIncompatible=true`will allow all incompatible casts to run natively in Comet, but this is not
28
-
recommended for production use.
29
-
-**U (Unsupported)**: Comet does not provide a native version of this cast expression and the query stage will fall back to
30
-
Spark.
25
+
-**I (Incompatible)**: Comet's native cast may produce different results from Apache Spark for some inputs. By
26
+
default, Comet instead runs Spark's own code-generated cast inside the Comet pipeline, so the results match Spark.
27
+
Setting `spark.comet.expression.Cast.allowIncompatible=true`selects the native cast, but this is not recommended
28
+
for production use.
29
+
-**U (Unsupported)**: Comet does not provide a native version of this cast, so it runs Spark's own code-generated
30
+
cast inside the Comet pipeline.
31
31
-**N/A**: Spark does not support this cast.
32
32
33
-
## ANSI Mode Fallback
33
+
Incompatible and unsupported casts fall back to Spark only when Comet's codegen dispatcher cannot handle them (for
34
+
example, casts involving `VariantType`) or when `spark.comet.exec.scalaUDF.codegen.enabled=false`.
34
35
35
-
Cast will fall back to Spark in some cases when ANSI mode is enabled. This can be enabled by setting `spark.comet.expression.Cast.allowIncompatible=true`. See the [Comet Supported Expressions Guide](../../../expressions.md) for more information on this configuration setting.
36
+
## ANSI Mode Support
37
+
38
+
Enabling ANSI mode does not by itself make a cast fall back to Spark or require
39
+
`spark.comet.expression.Cast.allowIncompatible=true`: every cast that Spark permits in ANSI mode has the same support
40
+
level as in legacy mode. See the [ANSI Mode](#ansi-mode) table below for per-pair support, and the
41
+
[Comet Supported Expressions Guide](../../../expressions.md) for more information on the `allowIncompatible` setting.
36
42
37
43
## Whitespace Trimming in Casts from String
38
44
@@ -143,20 +149,12 @@ Comet's native `CAST(date AS STRING)` is compatible with Spark. Years below 1000
143
149
zero-padded to four digits (e.g. year 999 renders as `0999-01-01`). Years above 9999 are
144
150
rendered without truncation. The cast is timezone-independent.
145
151
146
-
## String to TimestampNTZ
147
-
148
-
Comet's native `CAST(string AS TIMESTAMP_NTZ)` implementation matches Apache Spark's behavior.
149
-
Unlike `CAST(string AS TIMESTAMP)`, this cast is timezone-independent: any timezone offset in
150
-
the input string (e.g. `+08:00`, `Z`, `UTC`) is silently discarded, and the local date-time
151
-
components are preserved as-is. Time-only strings (e.g. `T12:34:56`, `12:34`) produce `NULL`.
152
-
The result is always a wall-clock timestamp with no timezone conversion or DST adjustment.
153
-
154
152
## Decimal with Negative Scale to String
155
153
156
154
Casting a `DecimalType` with a negative scale to `StringType` is marked as incompatible when
157
155
`spark.sql.legacy.allowNegativeScaleOfDecimal` is `false` (the default). When that config is
158
-
disabled, Spark cannot create negative-scale decimals, so Comet falls back to avoid running
159
-
native execution on unexpected inputs.
156
+
disabled, Spark cannot create negative-scale decimals, so by default Comet runs Spark's
157
+
code-generated cast instead of the native one.
160
158
161
159
When `spark.sql.legacy.allowNegativeScaleOfDecimal=true`, the cast is compatible. Comet matches
162
160
Spark's behavior of using Java `BigDecimal.toString()` semantics, which produces scientific
@@ -156,13 +148,3 @@ so users hunting an unexpected value have a single place to check:
156
148
`spark.sql.codegen.fallback` or when the generated code exceeds
157
149
`spark.sql.codegen.hugeMethodLimit`, where an intermediate overflow that later cancels out
158
150
returns `NULL` (or raises under ANSI) in Spark but the recovered value in Comet.
159
-
160
-
## Object store cache
161
-
162
-
When Comet's native scan reads Parquet files, it caches one object store instance per
163
-
`(scheme + host + port, hadoop-config-hash)` key. For `abfss://[email protected]/...`
164
-
URLs, the container lives in the URL userinfo, not the host, so two containers in one storage
165
-
account currently collide on the same cache entry. Within a single executor process, reading from
166
-
a second container after a first can be served by the first container's store instance and return
167
-
its data. S3, GCS, and HDFS are unaffected because their bucket / host lives in the URL host
168
-
component. Tracked by [#4993](https://github.com/apache/datafusion-comet/issues/4993).
- Floating-point values nested in array or struct `ORDER BY` keys are compared with Arrow's raw
111
+
total ordering, so ranks can differ from Spark when the data mixes `-0.0` and `+0.0` or more
112
+
than one NaN representation ([#5507](https://github.com/apache/datafusion-comet/issues/5507)).
113
+
Scalar `FLOAT` and `DOUBLE` keys are normalized and match Spark; see
Copy file name to clipboardExpand all lines: docs/source/user-guide/latest/datasources.md
+9-3Lines changed: 9 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -29,15 +29,17 @@ Arrow format, allowing the Comet pipeline to take over after that, but the proce
29
29
30
30
### Apache Iceberg
31
31
32
-
Comet accelerates Iceberg scans of Parquet files. See the [Iceberg Guide] for more information.
32
+
Comet accelerates Iceberg scans of Parquet files and has an experimental, opt-in native Iceberg writer.
33
+
See the [Iceberg Guide] and [Iceberg Writes](iceberg-writes.md) for more information.
33
34
34
35
[Iceberg Guide]: iceberg.md
35
36
36
37
### CSV
37
38
38
39
Comet provides experimental Rust-based CSV scan support. When `spark.comet.scan.csv.v2.enabled` is enabled, CSV files
39
40
are read in Rust for improved performance. This feature is experimental and performance benefits are
40
-
workload-dependent.
41
+
workload-dependent. Only Spark's DataSource V2 CSV scan is accelerated, and Spark reads CSV through the V1 API by
42
+
default, so also remove `csv` from `spark.sql.sources.useV1SourceList`.
41
43
42
44
Alternatively, when `spark.comet.convert.csv.enabled` is enabled, data from Spark's CSV reader is immediately
43
45
converted into Arrow format, allowing the Comet pipeline to take over after that.
@@ -308,6 +310,10 @@ credential providers and options documented above also apply to alias-scheme URL
308
310
translation feeds the native Iceberg scan; see
309
311
[Object store configuration (S3)](iceberg.md#object-store-configuration-s3) in the Iceberg guide.
310
312
313
+
A native Parquet scan whose alias-scheme paths span more than one bucket falls back to Spark. Alias
314
+
schemes apply to native scans only: a native Iceberg write to an alias-scheme location falls back to
315
+
iceberg-java.
316
+
311
317
### Examples
312
318
313
319
The following examples demonstrate how to configure S3 access using different authentication methods.
@@ -344,7 +350,7 @@ Comet's S3 support has the following limitations:
344
350
345
351
1.**Partial Hadoop S3A configuration support**: Not all Hadoop S3A configurations are currently supported. Only the configurations listed in the tables above are translated and applied to the underlying `object_store` crate.
346
352
347
-
2.**Custom credential providers**: Custom implementations of AWS credential providers are not supported. The implementation only supports the standard credential providers listed in the table above. We are planning to add support for custom credential providers through a JNI-based adapter that will allow calling Java credentialproviders from Rust code. See [#1829](https://github.com/apache/datafusion-comet/issues/1829) for more details.
353
+
2.**Custom credential providers**: Custom credential provider classes named in `fs.s3a.aws.credentials.provider`are not supported; only the standard providers listed in the table aboveare. To route credential requests through your own Java code, implement Comet's `CometS3CredentialProvider` SPI; see [S3 Credential Providers](s3-credential-providers.md). Broader Hadoop S3A integration is tracked in [#1829](https://github.com/apache/datafusion-comet/issues/1829).
0 commit comments