@@ -39,6 +39,7 @@ of it crashes. Review it expecting the failure mode to be quiet.
3939| ------------------------------------------------------ | ------------------------------------------------------------------------------------- |
4040| ` docs/source/contributor-guide/scan.md ` | Which path runs, the fallback gate model, the schema triple, the adapter, the gotchas |
4141| ` docs/source/user-guide/latest/datasources.md ` | The user-facing contract for formats, storage, schemes, and credentials |
42+ | ` docs/source/user-guide/latest/parquet.md ` | The user-facing list of Parquet fallbacks, reads that differ from Spark, and tuning |
4243| ` docs/source/contributor-guide/plugin_overview.md ` | Where ` CometScanRule ` sits relative to ` CometExecRule ` |
4344| ` docs/source/user-guide/latest/iceberg.md ` | Iceberg PRs only |
4445| ` docs/source/contributor-guide/iceberg-spark-tests.md ` | Iceberg PRs only. Those suites report in the merge queue, not on the PR |
@@ -95,13 +96,17 @@ Checklist for a new or moved gate:
9596 show it. This is easy to miss in review because the query still returns the right answer, just
9697 slowly, and the user has nothing to read.
9798- [ ] ** It matches the local return convention.** Some gates ` return withFallbackReason(...) `
98- immediately. Others (notably ` CometNativeScan.isSupported ` and the Iceberg arm) tag and
99- continue so that every applicable reason accumulates before the single decision at the end.
100- A gate that returns early where the surrounding code accumulates hides the other reasons.
99+ immediately. Others (notably ` CometNativeScan.isSupported ` and the table checks of the
100+ Iceberg arm) tag and continue so that every applicable reason accumulates before the single
101+ decision at the end. The Iceberg arm's checks on its configs, metadata columns, ordering,
102+ file scan tasks and metadata extraction return early. A gate that returns early where the
103+ surrounding code accumulates hides the other reasons.
101104- [ ] ** It is in the right function.** A guard hoisted above the contrib hook declines scans Comet
102105 does not own. A guard specific to the native Parquet reader belongs in ` nativeScan ` , not
103- ` transformV1Scan ` , so contrib scans are unaffected. ` scan.md ` § The Fallback Gate Model has
104- the placement rule.
106+ ` transformV1Scan ` , so contrib scans are unaffected. ` CometNativeScan.isSupported ` is also a
107+ planning-time gate: ` nativeScan ` calls it before any ` CometScanExec ` exists, while
108+ ` CometNativeScan.convert ` , which runs in ` CometExecRule ` , declines multi-store partitions and
109+ some ABFS layouts. ` scan.md ` § The Fallback Gate Model has the placement rule.
105110- [ ] ** A gate that duplicates an existing one is drift waiting to happen.** The AQE DPP check is
106111 deliberately in two places with a comment saying the second is a safety net. A new duplicate
107112 without that rationale should be one gate.
@@ -138,16 +143,22 @@ change here against Spark's source, not against what looks reasonable.
138143- [ ] ** The rejection matrix still mirrors ` ParquetVectorUpdaterFactory.getUpdater ` ** , at the same
139144 Spark versions, with the same leaf-by-leaf walk and the same first-verdict-wins ordering. A
140145 change that short-circuits differently changes which error a user sees.
141- - [ ] ** Errors stay deferred.** A rejection becomes a ` RejectOnNonEmpty ` that fires only on a
142- non-empty batch, so a file whose offending row groups are all pruned still reads, as it does
143- in Spark. A PR that raises at plan time breaks queries Spark accepts, and a test that always
144- reads the offending row group will not notice.
146+ - [ ] ** Each rejection keeps its timing.** Spark rejects a shape mismatch it cannot clip when it
147+ opens the file, and every other pair only in ` getUpdater ` , while it decodes a row group. The
148+ first is an error from the check, the second a ` ConversionCheck::RejectOnNonEmpty ` that
149+ fires only on a non-empty batch, so a file whose offending row groups are all pruned still
150+ reads, as it does in Spark. Moving a pair from the deferred verdict to an error breaks
151+ queries Spark accepts, and a test that always reads the offending row group will not notice.
152+ Moving one the other way hides an error Spark raises at open.
145153- [ ] ** Case folding goes through ` name_fold.rs ` .** A new ` to_lowercase() ` or
146154 ` eq_ignore_ascii_case() ` anywhere in the scan path is a finding. Three call sites share the
147155 policy, and those drifting apart is what produced #5495 .
148- - [ ] ** Field-id matching keeps Spark's gate** , which uses ids only when the conf is on _ and_ the
149- requested schema actually carries them. Removing either half changes behavior on schemas
150- Spark reads by name.
156+ - [ ] ** Field-id matching keeps Spark's gates.** Ids select fields (` use_field_id ` ) only when the
157+ conf is on _ and_ the requested schema carries them, and the missing-id rejection
158+ (` require_field_ids ` ) applies whenever the schema carries ids, unless
159+ ` spark.sql.parquet.fieldId.read.ignoreMissing ` is set, whatever the read flag says. Removing
160+ either half of either gate changes behavior on schemas Spark reads by name. The precedence
161+ between id and name lives in ` field_resolver.rs ` , shared by the root and the nested fields.
151162- [ ] ** ` is_pure_structural_narrowing ` is an allow list on purpose.** Any PR touching
152163 ` parquet_convert_array ` should say whether that predicate needs a matching exclusion, because
153164 a case it does not know to exclude silently starts producing wrong results.
@@ -175,15 +186,19 @@ that `Filter` ever sees them. The three checks below follow from which side a ch
175186## 6. Object Stores and Schemes
176187
177188- [ ] ** One object store is registered per ` FilePartition ` , keyed on the first file.** Everything
178- about multi-bucket handling follows from this. A PR that widens which schemes are claimed
179- needs to say what happens to a partition spanning two buckets. The existing V1-versus-Iceberg
180- asymmetry is deliberate, so do not let a PR "fix" it silently.
189+ about multi-bucket handling follows from this. A non-bucketed V1 scan packs its files per
190+ store, the gates decline what packing cannot fix (bucketed scans over several stores, and
191+ settings that cannot be forwarded per store), and the planner rejects a mixed-store
192+ partition. A PR that widens which schemes are claimed needs to say what happens to each of
193+ the three. The Iceberg gate is stricter and declines data files in more than one bucket,
194+ which is deliberate, so do not let a PR "fix" it silently.
181195- [ ] ** A new scheme needs both sides.** The JVM gate and the native reader must agree. The ` hdfs `
182196 default is the cautionary tale: the JVM default is ` Set("hdfs") ` specifically to mirror
183197 ` is_hdfs_scheme ` , because an empty default would silently fall back every plain ` hdfs:// ` scan.
184- - [ ] ** Iceberg's scheme list tracks ` storage_factory_for ` .** ` icebergReadableSchemes ` must stay in
185- lockstep with the match arms in ` iceberg_common.rs ` . Admitting a scheme iceberg-rust cannot
186- build turns a clean JVM fallback into a native runtime error.
198+ - [ ] ** Iceberg's scheme list comes from ` builtin_storage_schemes ` .** ` icebergReadableSchemes ` is
199+ loaded from ` iceberg_common.rs ` over JNI, so there is no JVM list to keep in lockstep. A PR
200+ that adds a hand-kept list needs a reason. Admitting a scheme iceberg-rust cannot build turns
201+ a clean JVM fallback into a native runtime error.
187202- [ ] ** A new Hadoop-surface key is read from the Hadoop config, not SQLConf** , or it will not work
188203 for users who configure through ` core-site.xml ` .
189204- [ ] ** New credential or endpoint handling is documented** in ` datasources.md ` , which carries the
@@ -210,9 +225,9 @@ This is the fail-closed rule applied to upstream defaults.
210225 options makes every future DataFusion option silently active.
211226- [ ] ** ` coerce_int96 ` and ` coerce_int96_tz ` stay hardcoded.** Making them session-overridable
212227 breaks the adapter's ability to distinguish an INT96-derived TimestampLTZ.
213- - [ ] ** A DataFusion version bump is a scan review.** The allow list, the
214- ` EagerPageIndexReaderFactory ` workaround and its stated exit condition, and the schema
215- adapter's assumptions about ` nested_struct::cast_column ` all track upstream.
228+ - [ ] ** A DataFusion version bump is a scan review.** The allow list, the forced page-index policy
229+ of ` EagerPageIndexReaderFactory ` and its stated exit condition, and the schema adapter's
230+ assumptions about ` nested_struct::cast_column ` all track upstream.
216231
217232## 9. Tests
218233
@@ -249,11 +264,14 @@ invalidate. Two need naming individually:
249264- The ** Which Scan Path Runs** table. The claim that V2 Parquet is not read natively is the single
250265 statement in the doc most likely to become wrong.
251266- The ** Page Index and Metadata Caching** section, which describes a workaround with a stated exit
252- condition. A DataFusion bump that fixes ` apache/datafusion#23978 ` should delete it rather than
253- edit it.
254-
255- Check also whether ` datasources.md ` (user-facing formats, schemes, credentials) or the
256- ` ### CometScanRule ` section of ` plugin_overview.md ` still holds.
267+ condition. A DataFusion bump that fixes ` apache/datafusion#23978 ` should drop the forced
268+ page-index policy and update the section. It should keep ` EagerPageIndexReaderFactory ` , which
269+ also hosts the field-id check, the Variant footer rewrite, the INT96 stamp, the deferred
270+ conversion check and the next-file prefetch.
271+
272+ Check also whether ` datasources.md ` (user-facing formats, schemes, credentials), ` parquet.md ` (the
273+ fallback list and the reads that differ from Spark) or the ` ### CometScanRule ` section of
274+ ` plugin_overview.md ` still holds.
257275
258276## Common Scan Review Findings
259277
0 commit comments