Skip to content

Commit a7744f2

Browse files
committed
docs(spark): shorten the pruneColumns comment and drop the user guide note
Signed-off-by: gitRasheed <[email protected]>
1 parent 93d7fa5 commit a7744f2

2 files changed

Lines changed: 2 additions & 5 deletions

File tree

‎docs/user-guide/spark.md‎

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -78,8 +78,6 @@ When pointed at a directory, the connector discovers all `.vortex` files and cre
7878
partition per file.
7979

8080
Column pruning is pushed down — only the columns referenced by the query are read from the file.
81-
Pruning applies to top-level columns: a query that uses one field of a struct, array or map
82-
column reads the whole column.
8381

8482
Filter pushdown is also supported. Comparisons (`=`, `<>`, `>`, `>=`, `<`, `<=`), `IS NULL`,
8583
`IS NOT NULL`, `IN`, the string predicates `STARTS_WITH`, `ENDS_WITH` and `CONTAINS`, and a bare

‎java/vortex-spark/src/main/java/dev/vortex/spark/read/VortexScanBuilder.java‎

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -130,9 +130,8 @@ public Scan build() {
130130
/**
131131
* Prunes the columns to only include those specified in the required schema.
132132
*
133-
* <p>Spark may also prune fields inside struct, array and map columns, but the native scan only projects top-level
134-
* columns and returns each one whole. Required columns therefore keep their table type, so that
135-
* {@link VortexScan#readSchema()} describes the data the reader returns.
133+
* <p>Required columns keep their table type because the native scan returns whole top-level columns, even when
134+
* Spark prunes nested fields.
136135
*
137136
* @param requiredSchema the schema specifying which columns are required
138137
*/

0 commit comments

Comments
 (0)