Skip to content

Commit 684c1e0

Browse files
committed
docs: stop describing the in-memory cache as experimental
It is enabled by default from Spark 3.5. The operator table now marks InMemoryTableScanExec as supported, and the plan-node table no longer says the cache scan is disabled by default.
1 parent 7047b8a commit 684c1e0

4 files changed

Lines changed: 7 additions & 8 deletions

File tree

‎docs/source/user-guide/latest/compatibility/operators.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,7 @@ infer the schema from.
3737
## In-Memory Cache
3838

3939
Comet can store cached relations (`df.cache()`, `CACHE TABLE`) in Arrow format and scan them
40-
natively. This is experimental, and enabled by default from Spark 3.5 but not on Spark 3.4; see
40+
natively. This is enabled by default from Spark 3.5 but not on Spark 3.4; see
4141
[In-Memory Cache](../in-memory-cache.md) for how to turn it off or on. Comet does not replace a
4242
`spark.sql.cache.serializer` that the application has already set. Relations whose schema Comet's
4343
Arrow writer does not support are cached in Spark's default format, and their scans fall back to

‎docs/source/user-guide/latest/in-memory-cache.md‎

Lines changed: 4 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -25,8 +25,8 @@ format, which Comet operators cannot read. Under Comet's default settings the op
2525
cache scan then run on Spark. With `spark.comet.convert.inMemoryCache.enabled`, a
2626
`CometSparkColumnarToColumnar` above the scan converts each batch for Comet operators instead.
2727

28-
This feature is **experimental and enabled by default** from Spark 3.5. To turn it off, set the
29-
config at startup, alongside the rest of Comet's configuration:
28+
This feature is **enabled by default** from Spark 3.5. To turn it off, set the config at startup,
29+
alongside the rest of Comet's configuration:
3030

3131
```shell
3232
$SPARK_HOME/bin/spark-shell \
@@ -277,9 +277,8 @@ The fused reader is faster than Spark's own format for the narrowest reads and w
277277
the others. The row reader takes up to 1.6 times as long, and it is the only reader for relations
278278
wider than `spark.sql.codegen.maxFields`: reading every column of relations of 100, 200 and 1500
279279
nullable `bigint` columns took 2.2 to 2.5 times as long as from Spark's format. A Spark operator
280-
above Comet's native cache scan does not pay this; see [Performance](#performance). These gaps are
281-
the main reason the feature is still described as experimental;
282-
[#5485](https://github.com/apache/datafusion-comet/issues/5485) tracks them.
280+
above Comet's native cache scan does not pay this; see [Performance](#performance).
281+
[#5485](https://github.com/apache/datafusion-comet/issues/5485) tracks these gaps.
283282

284283
Comet's serializer exists because Spark's own Arrow cache format
285284
([SPARK-57268](https://issues.apache.org/jira/browse/SPARK-57268)) is only available from Spark

‎docs/source/user-guide/latest/operators.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -66,7 +66,7 @@ omitted from the tables below and may be reconsidered based on demand:
6666
| `LocalTableScanExec` | ⚠️ | Disabled by default; there is no acceleration advantage and this operator is typically only used in test code. Can be opted into via config ([#4393](https://github.com/apache/datafusion-comet/pull/4393)). |
6767
| `EmptyRelationExec` | ✅ | Spark 4.0 and later. See [Empty Relations](compatibility/operators.md#empty-relations) for native-input support, including native Parquet writes. |
6868
| `RangeExec` | ⚠️ | Disabled by default. Set `spark.comet.exec.range.enabled=true` to generate the rows of `spark.range` and SQL `range()` in native code, so the operators above them run natively. It can be slower than Spark when those operators are only cheap expressions, such as a filter, which Spark compiles together with the range into one loop. Ranges whose arithmetic overflows the `Long` range fall back to Spark. |
69-
| `InMemoryTableScanExec` | ⚠️ | Experimental, enabled by default from Spark 3.5. Comet installs its Arrow cache serializer as the application starts if `spark.comet.exec.inMemoryCache.enabled` is true, its default except on Spark 3.4. Relations with unsupported column types stay in Spark's cache format and fall back. See [In-Memory Cache](in-memory-cache.md). |
69+
| `InMemoryTableScanExec` | ✅ | Enabled by default from Spark 3.5. Comet installs its Arrow cache serializer as the application starts if `spark.comet.exec.inMemoryCache.enabled` is true, its default except on Spark 3.4. Relations with unsupported column types stay in Spark's cache format and fall back. See [In-Memory Cache](in-memory-cache.md). |
7070

7171
## Projection and filtering
7272

‎docs/source/user-guide/latest/understanding-comet-plans.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -298,7 +298,7 @@ by role. Names match what is shown in the plan output.
298298
| `CometNativeScan` | Parquet scan that runs entirely in Rust via DataFusion. |
299299
| `CometIcebergNativeScan` | Iceberg Parquet scan that runs entirely in Rust via DataFusion. |
300300
| `CometCsvNativeScan` | CSV scan that runs entirely in Rust via DataFusion (experimental). |
301-
| `CometInMemoryTableScan` | JVM-side scan of a table cached in Comet's Arrow format (experimental, disabled by default). |
301+
| `CometInMemoryTableScan` | JVM-side scan of a table cached in Comet's Arrow format (enabled by default from Spark 3.5). |
302302
| `CometLocalTableScan` | JVM-side scan of a `LocalTableScanExec` (disabled by default). |
303303
| `CometEmptyRelation` | Empty native input that replaces `EmptyRelationExec` (Spark 4.0 and later). |
304304
| `CometRange` | Generates the rows of `spark.range` and SQL `range()` in native code, replacing `RangeExec` (disabled by default). |

0 commit comments

Comments
 (0)