You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit d531350
Browse filesBrowse the repository at this point in the historyBrowse files
docs: address review feedback on the 1.1.0 user guide
Document the Kryo registrator under "Additional Configuration" in the
installation guide. Native broadcast needs it with
spark.kryo.registrationRequired=true whether or not the in-memory cache
is enabled, so link to it from the cache docs rather than describing it
as a cache-only setting. Point the operator compatibility cache section
and the operators table at the in-memory cache page that landed on main,
instead of repeating it, which also drops a stale storage-format claim.
List every source Comet checks for the Iceberg S3 region, and link the
ANSI integral overflow divergence to #6217.
These `s3.*` storage properties are not specific to the Hive catalog shown here. When `s3.access-key-id` / `s3.secret-access-key` are omitted, credentials come from the standard AWS chain (environment variables, instance profiles, and so on). `client.region` is not auto-detected: when neither it nor `AWS_REGION`is set, Comet uses `us-east-1`, so set it for AWS buckets in any other region. If your REST catalog vends temporary credentials, the native reader does not consume them automatically, and wiring that requires the credential provider bridge. See Iceberg's [S3 FileIO](https://iceberg.apache.org/docs/latest/aws/#s3-fileio) docs for the full property list, and [S3 Credential Providers](s3-credential-providers.md) for vended or per-request credentials.
175
+
These `s3.*` storage properties are not specific to the Hive catalog shown here. When `s3.access-key-id` / `s3.secret-access-key` are omitted, credentials come from the standard AWS chain (environment variables, instance profiles, and so on). The region is not auto-detected: when neither the catalog (`client.region` or `s3.region`) nor the executor environment (`AWS_REGION`or `AWS_DEFAULT_REGION`) supplies one, Comet uses `us-east-1`, so set it for AWS buckets in any other region. If your REST catalog vends temporary credentials, the native reader does not consume them automatically, and wiring that requires the credential provider bridge. See Iceberg's [S3 FileIO](https://iceberg.apache.org/docs/latest/aws/#s3-fileio) docs for the full property list, and [S3 Credential Providers](s3-credential-providers.md) for vended or per-request credentials.
Copy file name to clipboardExpand all lines: docs/source/user-guide/latest/installation.md
+17Lines changed: 17 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -261,3 +261,20 @@ Some cluster managers may require additional configuration, see <https://spark.a
261
261
262
262
In addition to Apache Spark memory configuration parameters, Comet introduces additional parameters to configure memory
263
263
allocation for native execution. See [Comet Memory Tuning](./tuning.md) for details.
264
+
265
+
### Kryo serialization
266
+
267
+
If the application uses Kryo (`spark.serializer=org.apache.spark.serializer.KryoSerializer`) with
268
+
`spark.kryo.registrationRequired=true`, also register Comet's classes with Kryo:
|`FileSourceScanExec`| ✅ | Parquet only. Some types and configurations fall back. See [Parquet Scan Compatibility](compatibility/scans.md). |
55
-
|`BatchScanExec`| ✅ | Apache Iceberg Parquet scans run natively. Native CSV scans are experimental and disabled by default. DataSource V2 Parquet scans are not accelerated. See [Parquet Scan Compatibility](compatibility/scans.md) and the [Iceberg Guide](iceberg.md). |
56
-
|`LocalTableScanExec`| ⚠️ | Disabled by default; there is no acceleration advantage and this operator is typically only used in test code. Can be opted into via config ([#4393](https://github.com/apache/datafusion-comet/pull/4393)). |
57
-
|`EmptyRelationExec`| ✅ | Spark 4.0 and later. See [Empty Relations](compatibility/operators.md#empty-relations) for native-input support and writer fallback. |
58
-
|`InMemoryTableScanExec`| ⚠️ | Experimental, disabled by default. Set `spark.comet.exec.inMemoryCache.enabled=true` before the application starts so Comet installs its Arrow cache serializer. Relations with unsupported column types stay in Spark's cache format and fall back. See [In-Memory Cache](compatibility/operators.md#in-memory-cache). |
|`FileSourceScanExec`| ✅ | Parquet only. Some types and configurations fall back. See [Parquet Scan Compatibility](compatibility/scans.md). |
55
+
|`BatchScanExec`| ✅ | Apache Iceberg Parquet scans run natively. Native CSV scans are experimental and disabled by default. DataSource V2 Parquet scans are not accelerated. See [Parquet Scan Compatibility](compatibility/scans.md) and the [Iceberg Guide](iceberg.md). |
56
+
|`LocalTableScanExec`| ⚠️ | Disabled by default; there is no acceleration advantage and this operator is typically only used in test code. Can be opted into via config ([#4393](https://github.com/apache/datafusion-comet/pull/4393)). |
57
+
|`EmptyRelationExec`| ✅ | Spark 4.0 and later. See [Empty Relations](compatibility/operators.md#empty-relations) for native-input support and writer fallback. |
58
+
|`InMemoryTableScanExec`| ⚠️ | Experimental, disabled by default. Set `spark.comet.exec.inMemoryCache.enabled=true` before the application starts so Comet installs its Arrow cache serializer. Relations with unsupported column types stay in Spark's cache format and fall back. See [In-Memory Cache](in-memory-cache.md). |
0 commit comments