Skip to content

Commit 894a914

Browse files
committed
docs: bring the scan guide and review skill up to date with main
Review feedback: the immediate and deferred conversion rejections, per-store packing of multi-bucket scans, where CometNativeScan.isSupported runs, the gate table, the field id options, the Iceberg scheme list that is loaded from native, what the eager page index factory hosts, and the configs that affect the scan. Link the Parquet user guide.
1 parent fee4e1e commit 894a914

2 files changed

Lines changed: 181 additions & 103 deletions

File tree

  • .ai/skills/review-comet-scan-pr
  • docs/source/contributor-guide

‎.ai/skills/review-comet-scan-pr/SKILL.md‎

Lines changed: 44 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,7 @@ of it crashes. Review it expecting the failure mode to be quiet.
3939
| ------------------------------------------------------ | ------------------------------------------------------------------------------------- |
4040
| `docs/source/contributor-guide/scan.md` | Which path runs, the fallback gate model, the schema triple, the adapter, the gotchas |
4141
| `docs/source/user-guide/latest/datasources.md` | The user-facing contract for formats, storage, schemes, and credentials |
42+
| `docs/source/user-guide/latest/parquet.md` | The user-facing list of Parquet fallbacks, reads that differ from Spark, and tuning |
4243
| `docs/source/contributor-guide/plugin_overview.md` | Where `CometScanRule` sits relative to `CometExecRule` |
4344
| `docs/source/user-guide/latest/iceberg.md` | Iceberg PRs only |
4445
| `docs/source/contributor-guide/iceberg-spark-tests.md` | Iceberg PRs only. Those suites report in the merge queue, not on the PR |
@@ -95,13 +96,17 @@ Checklist for a new or moved gate:
9596
show it. This is easy to miss in review because the query still returns the right answer, just
9697
slowly, and the user has nothing to read.
9798
- [ ] **It matches the local return convention.** Some gates `return withFallbackReason(...)`
98-
immediately. Others (notably `CometNativeScan.isSupported` and the Iceberg arm) tag and
99-
continue so that every applicable reason accumulates before the single decision at the end.
100-
A gate that returns early where the surrounding code accumulates hides the other reasons.
99+
immediately. Others (notably `CometNativeScan.isSupported` and the table checks of the
100+
Iceberg arm) tag and continue so that every applicable reason accumulates before the single
101+
decision at the end. The Iceberg arm's checks on its configs, metadata columns, ordering,
102+
file scan tasks and metadata extraction return early. A gate that returns early where the
103+
surrounding code accumulates hides the other reasons.
101104
- [ ] **It is in the right function.** A guard hoisted above the contrib hook declines scans Comet
102105
does not own. A guard specific to the native Parquet reader belongs in `nativeScan`, not
103-
`transformV1Scan`, so contrib scans are unaffected. `scan.md` § The Fallback Gate Model has
104-
the placement rule.
106+
`transformV1Scan`, so contrib scans are unaffected. `CometNativeScan.isSupported` is also a
107+
planning-time gate: `nativeScan` calls it before any `CometScanExec` exists, while
108+
`CometNativeScan.convert`, which runs in `CometExecRule`, declines multi-store partitions and
109+
some ABFS layouts. `scan.md` § The Fallback Gate Model has the placement rule.
105110
- [ ] **A gate that duplicates an existing one is drift waiting to happen.** The AQE DPP check is
106111
deliberately in two places with a comment saying the second is a safety net. A new duplicate
107112
without that rationale should be one gate.
@@ -138,16 +143,22 @@ change here against Spark's source, not against what looks reasonable.
138143
- [ ] **The rejection matrix still mirrors `ParquetVectorUpdaterFactory.getUpdater`**, at the same
139144
Spark versions, with the same leaf-by-leaf walk and the same first-verdict-wins ordering. A
140145
change that short-circuits differently changes which error a user sees.
141-
- [ ] **Errors stay deferred.** A rejection becomes a `RejectOnNonEmpty` that fires only on a
142-
non-empty batch, so a file whose offending row groups are all pruned still reads, as it does
143-
in Spark. A PR that raises at plan time breaks queries Spark accepts, and a test that always
144-
reads the offending row group will not notice.
146+
- [ ] **Each rejection keeps its timing.** Spark rejects a shape mismatch it cannot clip when it
147+
opens the file, and every other pair only in `getUpdater`, while it decodes a row group. The
148+
first is an error from the check, the second a `ConversionCheck::RejectOnNonEmpty` that
149+
fires only on a non-empty batch, so a file whose offending row groups are all pruned still
150+
reads, as it does in Spark. Moving a pair from the deferred verdict to an error breaks
151+
queries Spark accepts, and a test that always reads the offending row group will not notice.
152+
Moving one the other way hides an error Spark raises at open.
145153
- [ ] **Case folding goes through `name_fold.rs`.** A new `to_lowercase()` or
146154
`eq_ignore_ascii_case()` anywhere in the scan path is a finding. Three call sites share the
147155
policy, and those drifting apart is what produced #5495.
148-
- [ ] **Field-id matching keeps Spark's gate**, which uses ids only when the conf is on _and_ the
149-
requested schema actually carries them. Removing either half changes behavior on schemas
150-
Spark reads by name.
156+
- [ ] **Field-id matching keeps Spark's gates.** Ids select fields (`use_field_id`) only when the
157+
conf is on _and_ the requested schema carries them, and the missing-id rejection
158+
(`require_field_ids`) applies whenever the schema carries ids, unless
159+
`spark.sql.parquet.fieldId.read.ignoreMissing` is set, whatever the read flag says. Removing
160+
either half of either gate changes behavior on schemas Spark reads by name. The precedence
161+
between id and name lives in `field_resolver.rs`, shared by the root and the nested fields.
151162
- [ ] **`is_pure_structural_narrowing` is an allow list on purpose.** Any PR touching
152163
`parquet_convert_array` should say whether that predicate needs a matching exclusion, because
153164
a case it does not know to exclude silently starts producing wrong results.
@@ -175,15 +186,19 @@ that `Filter` ever sees them. The three checks below follow from which side a ch
175186
## 6. Object Stores and Schemes
176187

177188
- [ ] **One object store is registered per `FilePartition`, keyed on the first file.** Everything
178-
about multi-bucket handling follows from this. A PR that widens which schemes are claimed
179-
needs to say what happens to a partition spanning two buckets. The existing V1-versus-Iceberg
180-
asymmetry is deliberate, so do not let a PR "fix" it silently.
189+
about multi-bucket handling follows from this. A non-bucketed V1 scan packs its files per
190+
store, the gates decline what packing cannot fix (bucketed scans over several stores, and
191+
settings that cannot be forwarded per store), and the planner rejects a mixed-store
192+
partition. A PR that widens which schemes are claimed needs to say what happens to each of
193+
the three. The Iceberg gate is stricter and declines data files in more than one bucket,
194+
which is deliberate, so do not let a PR "fix" it silently.
181195
- [ ] **A new scheme needs both sides.** The JVM gate and the native reader must agree. The `hdfs`
182196
default is the cautionary tale: the JVM default is `Set("hdfs")` specifically to mirror
183197
`is_hdfs_scheme`, because an empty default would silently fall back every plain `hdfs://` scan.
184-
- [ ] **Iceberg's scheme list tracks `storage_factory_for`.** `icebergReadableSchemes` must stay in
185-
lockstep with the match arms in `iceberg_common.rs`. Admitting a scheme iceberg-rust cannot
186-
build turns a clean JVM fallback into a native runtime error.
198+
- [ ] **Iceberg's scheme list comes from `builtin_storage_schemes`.** `icebergReadableSchemes` is
199+
loaded from `iceberg_common.rs` over JNI, so there is no JVM list to keep in lockstep. A PR
200+
that adds a hand-kept list needs a reason. Admitting a scheme iceberg-rust cannot build turns
201+
a clean JVM fallback into a native runtime error.
187202
- [ ] **A new Hadoop-surface key is read from the Hadoop config, not SQLConf**, or it will not work
188203
for users who configure through `core-site.xml`.
189204
- [ ] **New credential or endpoint handling is documented** in `datasources.md`, which carries the
@@ -210,9 +225,9 @@ This is the fail-closed rule applied to upstream defaults.
210225
options makes every future DataFusion option silently active.
211226
- [ ] **`coerce_int96` and `coerce_int96_tz` stay hardcoded.** Making them session-overridable
212227
breaks the adapter's ability to distinguish an INT96-derived TimestampLTZ.
213-
- [ ] **A DataFusion version bump is a scan review.** The allow list, the
214-
`EagerPageIndexReaderFactory` workaround and its stated exit condition, and the schema
215-
adapter's assumptions about `nested_struct::cast_column` all track upstream.
228+
- [ ] **A DataFusion version bump is a scan review.** The allow list, the forced page-index policy
229+
of `EagerPageIndexReaderFactory` and its stated exit condition, and the schema adapter's
230+
assumptions about `nested_struct::cast_column` all track upstream.
216231

217232
## 9. Tests
218233

@@ -249,11 +264,14 @@ invalidate. Two need naming individually:
249264
- The **Which Scan Path Runs** table. The claim that V2 Parquet is not read natively is the single
250265
statement in the doc most likely to become wrong.
251266
- The **Page Index and Metadata Caching** section, which describes a workaround with a stated exit
252-
condition. A DataFusion bump that fixes `apache/datafusion#23978` should delete it rather than
253-
edit it.
254-
255-
Check also whether `datasources.md` (user-facing formats, schemes, credentials) or the
256-
`### CometScanRule` section of `plugin_overview.md` still holds.
267+
condition. A DataFusion bump that fixes `apache/datafusion#23978` should drop the forced
268+
page-index policy and update the section. It should keep `EagerPageIndexReaderFactory`, which
269+
also hosts the field-id check, the Variant footer rewrite, the INT96 stamp, the deferred
270+
conversion check and the next-file prefetch.
271+
272+
Check also whether `datasources.md` (user-facing formats, schemes, credentials), `parquet.md` (the
273+
fallback list and the reads that differ from Spark) or the `### CometScanRule` section of
274+
`plugin_overview.md` still holds.
257275

258276
## Common Scan Review Findings
259277

0 commit comments

Comments
 (0)