Skip to content

Commit d531350

Browse files
committed
docs: address review feedback on the 1.1.0 user guide
Document the Kryo registrator under "Additional Configuration" in the installation guide. Native broadcast needs it with spark.kryo.registrationRequired=true whether or not the in-memory cache is enabled, so link to it from the cache docs rather than describing it as a cache-only setting. Point the operator compatibility cache section and the operators table at the in-memory cache page that landed on main, instead of repeating it, which also drops a stale storage-format claim. List every source Comet checks for the Iceberg S3 region, and link the ANSI integral overflow divergence to #6217.
1 parent ad01b04 commit d531350

6 files changed

Lines changed: 38 additions & 28 deletions

File tree

‎docs/source/user-guide/latest/compatibility/index.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -114,7 +114,8 @@ divergence:
114114

115115
- Byte / Short `Add`, `Subtract`, and `Multiply` overflow raises `ARITHMETIC_OVERFLOW` (for
116116
example `byte overflow`) where Spark raises `BINARY_ARITHMETIC_OVERFLOW`, and integral
117-
`ARITHMETIC_OVERFLOW` messages omit Spark's `try_` suggestion.
117+
`ARITHMETIC_OVERFLOW` messages omit Spark's `try_` suggestion
118+
([#6217](https://github.com/apache/datafusion-comet/issues/6217)).
118119
- Wide-decimal arithmetic overflow, decimal divide-by-zero, and decimal-to-decimal cast overflow
119120
raise raw Arrow errors that bypass `SparkErrorConverter` and surface as `CometNativeException`
120121
rather than `SparkArithmeticException` with the proper error class and query context

‎docs/source/user-guide/latest/compatibility/operators.md‎

Lines changed: 8 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -35,23 +35,14 @@ readable empty output files and their schema metadata.
3535
## In-Memory Cache
3636

3737
Comet can store cached relations (`df.cache()`, `CACHE TABLE`) in Arrow format and scan them
38-
natively. This is experimental and disabled by default. Enable it with
39-
`spark.comet.exec.inMemoryCache.enabled=true` before the application starts: the value at startup
40-
decides whether Comet sets `spark.sql.cache.serializer` to its Arrow cache serializer, and
41-
because that is a static config the cache format is fixed for the application. Comet does not
42-
replace a `spark.sql.cache.serializer` that the application has already set. Disabling the
43-
setting later only sends cached scans back to Spark's execution path.
44-
45-
Relations whose schema Comet's Arrow writer does not support are cached in Spark's default
46-
format, and their scans fall back to Spark. Each cached column is stored as its own compressed
47-
Arrow IPC stream, so a scan decodes only the columns it projects. Reads that feed Spark operators
48-
rather than Comet operators still pay a row conversion that Spark's default format avoids, and
49-
can be slower than Spark's cache.
50-
51-
With `spark.kryo.registrationRequired=true`, also set
52-
`spark.kryo.registrator=org.apache.comet.CometKryoRegistrator` before creating the
53-
`SparkContext`. Otherwise caching fails as soon as a block is serialized, including the disk
54-
half of the default `MEMORY_AND_DISK` storage level.
38+
natively. This is experimental and disabled by default; see [In-Memory Cache](../in-memory-cache.md)
39+
for how to enable it. Comet does not replace a `spark.sql.cache.serializer` that the application
40+
has already set. Relations whose schema Comet's Arrow writer does not support are cached in
41+
Spark's default format, and their scans fall back to Spark. Reads that feed Spark operators rather
42+
than Comet operators can be slower than Spark's cache.
43+
44+
With Kryo and `spark.kryo.registrationRequired=true`, Comet needs its Kryo registrator whether or
45+
not the cache is enabled; see [Kryo serialization](../installation.md#kryo-serialization).
5546

5647
## Sampling
5748

‎docs/source/user-guide/latest/iceberg.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -172,7 +172,7 @@ For a custom S3-compatible endpoint, configure the catalog with the endpoint, pa
172172
--conf spark.sql.catalog.s3_cat.s3.secret-access-key=...
173173
```
174174

175-
These `s3.*` storage properties are not specific to the Hive catalog shown here. When `s3.access-key-id` / `s3.secret-access-key` are omitted, credentials come from the standard AWS chain (environment variables, instance profiles, and so on). `client.region` is not auto-detected: when neither it nor `AWS_REGION` is set, Comet uses `us-east-1`, so set it for AWS buckets in any other region. If your REST catalog vends temporary credentials, the native reader does not consume them automatically, and wiring that requires the credential provider bridge. See Iceberg's [S3 FileIO](https://iceberg.apache.org/docs/latest/aws/#s3-fileio) docs for the full property list, and [S3 Credential Providers](s3-credential-providers.md) for vended or per-request credentials.
175+
These `s3.*` storage properties are not specific to the Hive catalog shown here. When `s3.access-key-id` / `s3.secret-access-key` are omitted, credentials come from the standard AWS chain (environment variables, instance profiles, and so on). The region is not auto-detected: when neither the catalog (`client.region` or `s3.region`) nor the executor environment (`AWS_REGION` or `AWS_DEFAULT_REGION`) supplies one, Comet uses `us-east-1`, so set it for AWS buckets in any other region. If your REST catalog vends temporary credentials, the native reader does not consume them automatically, and wiring that requires the credential provider bridge. See Iceberg's [S3 FileIO](https://iceberg.apache.org/docs/latest/aws/#s3-fileio) docs for the full property list, and [S3 Credential Providers](s3-credential-providers.md) for vended or per-request credentials.
176176

177177
### Current limitations
178178

‎docs/source/user-guide/latest/in-memory-cache.md‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -159,7 +159,7 @@ Spark serializes a cached batch with `spark.serializer` whenever the block leave
159159
`_SER` storage levels, replication, cross-executor fetches, and the disk half of the default
160160
`MEMORY_AND_DISK`. So an ordinary `df.cache()` that spills is enough to reach it.
161161

162-
If you run with `spark.kryo.registrationRequired=true`, register Comet's classes:
162+
If you run Kryo with `spark.kryo.registrationRequired=true`, register Comet's classes:
163163

164164
```
165165
spark.serializer=org.apache.spark.serializer.KryoSerializer
@@ -171,7 +171,8 @@ Comet cannot set `spark.kryo.registrator` for you the way it sets `spark.sql.cac
171171
`KryoSerializer` reads it when `SparkEnv` builds the serializer, which happens before any plugin
172172
runs. Without it, caching fails with a "Class is not registered" error that does not name this
173173
feature. Comet's driver plugin warns at startup when it sees Kryo, `registrationRequired`, and no
174-
registrator.
174+
registrator. Native broadcast needs the same registrator even when the cache is disabled; see
175+
[Kryo serialization](installation.md#kryo-serialization).
175176

176177
## Limitations
177178

‎docs/source/user-guide/latest/installation.md‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -261,3 +261,20 @@ Some cluster managers may require additional configuration, see <https://spark.a
261261

262262
In addition to Apache Spark memory configuration parameters, Comet introduces additional parameters to configure memory
263263
allocation for native execution. See [Comet Memory Tuning](./tuning.md) for details.
264+
265+
### Kryo serialization
266+
267+
If the application uses Kryo (`spark.serializer=org.apache.spark.serializer.KryoSerializer`) with
268+
`spark.kryo.registrationRequired=true`, also register Comet's classes with Kryo:
269+
270+
```shell
271+
--conf spark.kryo.registrator=org.apache.comet.CometKryoRegistrator
272+
```
273+
274+
Without it, any query that uses Comet's native broadcast exchange, which is enabled by default,
275+
fails with Kryo's "Class is not registered" error, for example on the first broadcast hash join.
276+
The [in-memory cache](in-memory-cache.md#kryo) needs the same registrator. Set it before the
277+
`SparkContext` is created: `KryoSerializer` reads it before Comet's plugin runs, so Comet cannot
278+
add it for you. `spark.kryo.registrator` accepts a comma-separated list, so an application with
279+
its own registrator can list both. Comet logs a warning at startup when Kryo requires registration
280+
and this registrator is missing.

‎docs/source/user-guide/latest/operators.md‎

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -49,13 +49,13 @@ omitted from the tables below and may be reconsidered based on demand:
4949

5050
## Scans
5151

52-
| Operator | Status | Notes |
53-
| ----------------------- | ------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
54-
| `FileSourceScanExec` | ✅ | Parquet only. Some types and configurations fall back. See [Parquet Scan Compatibility](compatibility/scans.md). |
55-
| `BatchScanExec` | ✅ | Apache Iceberg Parquet scans run natively. Native CSV scans are experimental and disabled by default. DataSource V2 Parquet scans are not accelerated. See [Parquet Scan Compatibility](compatibility/scans.md) and the [Iceberg Guide](iceberg.md). |
56-
| `LocalTableScanExec` | ⚠️ | Disabled by default; there is no acceleration advantage and this operator is typically only used in test code. Can be opted into via config ([#4393](https://github.com/apache/datafusion-comet/pull/4393)). |
57-
| `EmptyRelationExec` | ✅ | Spark 4.0 and later. See [Empty Relations](compatibility/operators.md#empty-relations) for native-input support and writer fallback. |
58-
| `InMemoryTableScanExec` | ⚠️ | Experimental, disabled by default. Set `spark.comet.exec.inMemoryCache.enabled=true` before the application starts so Comet installs its Arrow cache serializer. Relations with unsupported column types stay in Spark's cache format and fall back. See [In-Memory Cache](compatibility/operators.md#in-memory-cache). |
52+
| Operator | Status | Notes |
53+
| ----------------------- | ------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
54+
| `FileSourceScanExec` | ✅ | Parquet only. Some types and configurations fall back. See [Parquet Scan Compatibility](compatibility/scans.md). |
55+
| `BatchScanExec` | ✅ | Apache Iceberg Parquet scans run natively. Native CSV scans are experimental and disabled by default. DataSource V2 Parquet scans are not accelerated. See [Parquet Scan Compatibility](compatibility/scans.md) and the [Iceberg Guide](iceberg.md). |
56+
| `LocalTableScanExec` | ⚠️ | Disabled by default; there is no acceleration advantage and this operator is typically only used in test code. Can be opted into via config ([#4393](https://github.com/apache/datafusion-comet/pull/4393)). |
57+
| `EmptyRelationExec` | ✅ | Spark 4.0 and later. See [Empty Relations](compatibility/operators.md#empty-relations) for native-input support and writer fallback. |
58+
| `InMemoryTableScanExec` | ⚠️ | Experimental, disabled by default. Set `spark.comet.exec.inMemoryCache.enabled=true` before the application starts so Comet installs its Arrow cache serializer. Relations with unsupported column types stay in Spark's cache format and fall back. See [In-Memory Cache](in-memory-cache.md). |
5959

6060
## Projection and filtering
6161

0 commit comments

Comments
 (0)