Skip to content

Commit ba0d8d7

Browse files
committed
fix: answer both halves of the missing-field-id check from what Spark reads
Spark checks the pruned required schema for ids and the raw Parquet schema for their absence. The requested half now comes from `required_schema` at plan time, since the logical schema DataFusion hands the adapter is the full read schema, so an id on a column the query never projects no longer raises. The file half moves into the eager page index reader, which walks the raw schema from the footer. The Arrow schema the adapter sees has lost container metadata after INT96 coercion and never carries ids on repeated list or key value groups, so a file whose only id sat on a struct holding a timestamp raised where Spark reads. The JNI error conversion unwraps the Parquet external error so the Java side keeps the same exception, and the Iceberg scan no longer needs an opt-out since it does not use that reader. Tests cover the pruned projection, the timestamp struct, an id only on a list or key value group, a directory mixing files with and without ids, nested schema pruning, id zero, count(*), and the exception class on each version.
1 parent 0b5c4a4 commit ba0d8d7

7 files changed

Lines changed: 916 additions & 109 deletions

File tree

‎native/core/src/execution/operators/iceberg_scan.rs‎

Lines changed: 1 addition & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -230,13 +230,7 @@ impl IcebergScanExec {
230230
let scan_metrics = scan_result.metrics().clone();
231231
let stream = scan_result.stream();
232232

233-
let mut spark_options = SparkParquetOptions::new(EvalMode::Legacy, "UTC", false);
234-
// Iceberg resolves columns by id itself and its reader supplies the ids, so the
235-
// missing-id check that guards plain Parquet reads does not apply here. The same holds
236-
// for a migrated table read through a name mapping: the reader still resolves the
237-
// columns itself and hands back a schema of its own, so the Spark check has nothing to
238-
// say there either.
239-
spark_options.ignore_missing_field_id = true;
233+
let spark_options = SparkParquetOptions::new(EvalMode::Legacy, "UTC", false);
240234
let adapter_factory = SparkPhysicalExprAdapterFactory::new(spark_options, None);
241235

242236
let adapted_stream =

‎native/core/src/parquet/eager_page_index_reader_factory.rs‎

Lines changed: 124 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,13 @@
4545
//!
4646
//! Filed upstream as apache/datafusion#23978. Revert this once the opener merges its deferred
4747
//! page-index load back into `FileMetadataCache` instead of bypassing it.
48+
//!
49+
//! The reader also carries Spark's missing field id check, because the footer is first at hand
50+
//! here. Spark's `ParquetReadSupport` refuses to open a file whose raw schema carries no field
51+
//! id when the requested schema carries one, unless `ignoreMissing` is set, and it walks the
52+
//! raw `MessageType` to decide. The Arrow schema the schema adapter sees later cannot stand in
53+
//! for that walk: the INT96 coercion rebuilds container fields without their metadata, and an
54+
//! id on a `list` or `key_value` group, or on the message root, never reaches an Arrow field.
4855
4956
use arrow::datatypes::{DataType, FieldRef, Schema};
5057
use async_trait::async_trait;
@@ -58,6 +65,7 @@ use datafusion::execution::cache::cache_manager::FileMetadataCache;
5865
use datafusion::physical_plan::metrics::{
5966
Count, ExecutionPlanMetricsSet, MetricBuilder, MetricCategory, MetricType,
6067
};
68+
use datafusion_comet_common::SparkError;
6169
use datafusion_datasource::PartitionedFile;
6270
use futures::future::BoxFuture;
6371
use futures::{FutureExt, StreamExt, TryStreamExt};
@@ -77,7 +85,7 @@ use parquet::file::metadata::{FileMetaData, KeyValue, ParquetMetaDataBuilder};
7785
use parquet::file::metadata::{
7886
FooterTail, PageIndexPolicy, ParquetMetaData, ParquetMetaDataReader,
7987
};
80-
use parquet::schema::types::{ColumnDescPtr, SchemaDescriptor};
88+
use parquet::schema::types::{ColumnDescPtr, SchemaDescriptor, Type as ParquetType};
8189
use std::fmt::{Debug, Display, Formatter};
8290
use std::ops::Range;
8391
use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering};
@@ -163,6 +171,11 @@ pub struct EagerPageIndexReaderFactory {
163171
// Enable the footer workaround only for scans that project Variant.
164172
// https://github.com/apache/datafusion-comet/issues/5477
165173
spark_variant_schema: bool,
174+
// Whether the schema Spark asked the scan for carries a field id at any depth, computed
175+
// once by the planner. Together with `ignore_missing_field_id` it decides whether a file
176+
// whose Parquet schema carries no id is refused, as Spark's `ParquetReadSupport` does.
177+
requested_schema_has_field_ids: bool,
178+
ignore_missing_field_id: bool,
166179
}
167180

168181
impl EagerPageIndexReaderFactory {
@@ -191,13 +204,40 @@ impl EagerPageIndexReaderFactory {
191204
metadata_cache,
192205
scan_io_metrics,
193206
spark_variant_schema: false,
207+
requested_schema_has_field_ids: false,
208+
ignore_missing_field_id: false,
194209
}
195210
}
196211

197212
pub fn with_spark_variant_schema(mut self, enabled: bool) -> Self {
198213
self.spark_variant_schema = enabled;
199214
self
200215
}
216+
217+
/// Arm Spark's missing field id check. A file whose Parquet schema carries no field id
218+
/// is refused on open when `requested_schema_has_field_ids` is set and
219+
/// `ignore_missing_field_id` is not. Both default to off, so a factory that never calls
220+
/// this reads every file.
221+
pub fn with_missing_field_id_check(
222+
mut self,
223+
requested_schema_has_field_ids: bool,
224+
ignore_missing_field_id: bool,
225+
) -> Self {
226+
self.requested_schema_has_field_ids = requested_schema_has_field_ids;
227+
self.ignore_missing_field_id = ignore_missing_field_id;
228+
self
229+
}
230+
}
231+
232+
/// True when `node` or any node under it carries a field id, the way Spark's
233+
/// `containsFieldIds` answers it over the raw Parquet schema, message root included.
234+
fn parquet_schema_has_field_ids(node: &ParquetType) -> bool {
235+
node.get_basic_info().has_id()
236+
|| (node.is_group()
237+
&& node
238+
.get_fields()
239+
.iter()
240+
.any(|field| parquet_schema_has_field_ids(field)))
201241
}
202242

203243
impl ParquetFileReaderFactory for EagerPageIndexReaderFactory {
@@ -225,6 +265,8 @@ impl ParquetFileReaderFactory for EagerPageIndexReaderFactory {
225265
metadata_cache: Arc::clone(&self.metadata_cache),
226266
metadata_size_hint,
227267
spark_variant_schema: self.spark_variant_schema,
268+
requested_schema_has_field_ids: self.requested_schema_has_field_ids,
269+
ignore_missing_field_id: self.ignore_missing_field_id,
228270
}))
229271
}
230272
}
@@ -240,6 +282,8 @@ struct EagerPageIndexReader {
240282
metadata_cache: Arc<FileMetadataCache>,
241283
metadata_size_hint: Option<usize>,
242284
spark_variant_schema: bool,
285+
requested_schema_has_field_ids: bool,
286+
ignore_missing_field_id: bool,
243287
}
244288

245289
// Arrow infers ENUM as Binary, losing the distinction from raw binary that Spark needs.
@@ -439,6 +483,8 @@ impl AsyncFileReader for EagerPageIndexReader {
439483
let metadata_size_hint = self.metadata_size_hint;
440484
let scan_io_metrics = Arc::clone(&self.scan_io_metrics);
441485
let spark_variant_schema = self.spark_variant_schema;
486+
let require_file_field_ids =
487+
self.requested_schema_has_field_ids && !self.ignore_missing_field_id;
442488
async move {
443489
let file_decryption_properties = options
444490
.and_then(|o| o.file_decryption_properties())
@@ -498,6 +544,20 @@ impl AsyncFileReader for EagerPageIndexReader {
498544
}
499545

500546
let metadata = metadata?;
547+
// Spark's `ParquetReadSupport` refuses to open a file that carries no field ids when
548+
// the requested schema carries some, unless `ignoreMissing` is set, and it walks the
549+
// raw `MessageType` to decide. The same walk runs here over the footer's schema. The
550+
// error keeps its Spark type through `ParquetError::External`, which the JNI layer
551+
// unwraps, so the JVM sees the same exception Spark raises.
552+
if require_file_field_ids
553+
&& !parquet_schema_has_field_ids(
554+
metadata.file_metadata().schema_descr().root_schema(),
555+
)
556+
{
557+
return Err(ParquetError::External(Box::new(
558+
SparkError::ParquetMissingFieldIds,
559+
)));
560+
}
501561
if spark_variant_schema {
502562
with_spark_arrow_schema(metadata)
503563
} else {
@@ -992,6 +1052,69 @@ mod tests {
9921052
);
9931053
}
9941054

1055+
/// The raw schema walk answers like Spark's `containsFieldIds`: an id on the message root,
1056+
/// on a leaf, or on a repeated `list` group counts, and a schema without any does not.
1057+
#[test]
1058+
fn parquet_schema_has_field_ids_sees_ids_on_any_node() {
1059+
use parquet::basic::Type as PhysicalType;
1060+
use parquet::schema::types::TypePtr;
1061+
1062+
let leaf = |name: &str, id: Option<i32>| -> TypePtr {
1063+
Arc::new(
1064+
ParquetType::primitive_type_builder(name, PhysicalType::INT32)
1065+
.with_id(id)
1066+
.build()
1067+
.unwrap(),
1068+
)
1069+
};
1070+
let group = |name: &str, id: Option<i32>, fields: Vec<TypePtr>| -> TypePtr {
1071+
Arc::new(
1072+
ParquetType::group_type_builder(name)
1073+
.with_id(id)
1074+
.with_fields(fields)
1075+
.build()
1076+
.unwrap(),
1077+
)
1078+
};
1079+
1080+
assert!(!parquet_schema_has_field_ids(&group(
1081+
"schema",
1082+
None,
1083+
vec![]
1084+
)));
1085+
assert!(!parquet_schema_has_field_ids(&group(
1086+
"schema",
1087+
None,
1088+
vec![leaf("a", None)]
1089+
)));
1090+
assert!(parquet_schema_has_field_ids(&group(
1091+
"schema",
1092+
Some(1),
1093+
vec![leaf("a", None)]
1094+
)));
1095+
assert!(parquet_schema_has_field_ids(&group(
1096+
"schema",
1097+
None,
1098+
vec![leaf("a", Some(1))]
1099+
)));
1100+
let list_group_only = group(
1101+
"schema",
1102+
None,
1103+
vec![group(
1104+
"l",
1105+
None,
1106+
vec![group("list", Some(5), vec![leaf("element", None)])],
1107+
)],
1108+
);
1109+
assert!(parquet_schema_has_field_ids(&list_group_only));
1110+
let nested_without_ids = group(
1111+
"schema",
1112+
None,
1113+
vec![group("s", None, vec![leaf("a", None)])],
1114+
);
1115+
assert!(!parquet_schema_has_field_ids(&nested_without_ids));
1116+
}
1117+
9951118
#[test]
9961119
fn variant_policy_preserves_footer_metadata_and_indexes() {
9971120
let schema = Arc::new(Schema::new_with_metadata(

‎native/core/src/parquet/parquet_exec.rs‎

Lines changed: 11 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,7 @@ use crate::parquet::eager_page_index_reader_factory::{EagerPageIndexReaderFactor
2020
use crate::parquet::encryption_support::{CometEncryptionConfig, ENCRYPTION_FACTORY_ID};
2121
use crate::parquet::name_fold::fold_schema_names;
2222
use crate::parquet::parquet_support::{
23-
object_store_authority, ObjectStoreBackend, SparkParquetOptions,
23+
any_nested_field_has_id, object_store_authority, ObjectStoreBackend, SparkParquetOptions,
2424
};
2525
use crate::parquet::schema_adapter::SparkPhysicalExprAdapterFactory;
2626
use arrow::datatypes::{Field, FieldRef, SchemaRef};
@@ -102,6 +102,11 @@ pub(crate) fn init_datasource_exec(
102102
);
103103
spark_parquet_options.use_field_id = use_field_id;
104104
spark_parquet_options.ignore_missing_field_id = ignore_missing_field_id;
105+
// Spark runs its missing-id check against the pruned read schema it hands the reader, not
106+
// the full data schema that DataFusion later passes the schema adapter, so the answer is
107+
// taken from `required_schema` here, once per scan, and handed to the reader factory below.
108+
spark_parquet_options.requested_schema_has_field_ids =
109+
any_nested_field_has_id(required_schema.fields());
105110
// Spark can discard filtered-out values before timestamp conversion using statistics,
106111
// dictionary, and row-level filters. Comet cannot mirror every pruning path, so applying
107112
// checked conversion in a filtered scan can fail on values Spark never reads. Preserve the
@@ -194,7 +199,11 @@ pub(crate) fn init_datasource_exec(
194199
scan_io_source,
195200
parquet_source.metrics(),
196201
)
197-
.with_spark_variant_schema(projects_variant),
202+
.with_spark_variant_schema(projects_variant)
203+
.with_missing_field_id_check(
204+
spark_parquet_options.requested_schema_has_field_ids,
205+
spark_parquet_options.ignore_missing_field_id,
206+
),
198207
);
199208
parquet_source = parquet_source.with_parquet_file_reader_factory(reader_factory);
200209

‎native/core/src/parquet/parquet_support.rs‎

Lines changed: 24 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -104,6 +104,11 @@ pub struct SparkParquetOptions {
104104
/// requested schema does carry ids raises a runtime error rather than silently
105105
/// producing nulls (mirrors `spark.sql.parquet.fieldId.read.ignoreMissing`).
106106
pub ignore_missing_field_id: bool,
107+
/// Whether the schema Spark asked the scan for carries a Parquet field id at any depth.
108+
/// Spark's `ParquetReadSupport` runs its missing-id check against that pruned schema, so
109+
/// the planner computes this once from `required_schema`. The reader factory tests it when
110+
/// it reads a file's footer, against the Parquet schema found there.
111+
pub requested_schema_has_field_ids: bool,
107112
/// Whether type promotion (schema evolution) is allowed, e.g. INT32 -> INT64,
108113
/// FLOAT -> DOUBLE. Mirrors spark.comet.schemaEvolution.enabled.
109114
pub allow_type_promotion: bool,
@@ -132,6 +137,7 @@ impl SparkParquetOptions {
132137
return_null_struct_if_all_fields_missing: true,
133138
use_field_id: false,
134139
ignore_missing_field_id: false,
140+
requested_schema_has_field_ids: false,
135141
allow_type_promotion: false,
136142
allow_timestamp_ltz_to_ntz: false,
137143
checked_timestamp_overflow: true,
@@ -149,6 +155,7 @@ impl SparkParquetOptions {
149155
return_null_struct_if_all_fields_missing: true,
150156
use_field_id: false,
151157
ignore_missing_field_id: false,
158+
requested_schema_has_field_ids: false,
152159
allow_type_promotion: false,
153160
allow_timestamp_ltz_to_ntz: false,
154161
checked_timestamp_overflow: true,
@@ -416,15 +423,27 @@ fn field_id(field: &arrow::datatypes::Field) -> Option<i32> {
416423

417424
/// True when a field in `fields`, at any nesting depth, carries a Parquet field id. Spark's
418425
/// `containsFieldIds` walks the whole file schema the same way, and `ParquetUtils.hasFieldIds`
419-
/// walks the read schema. The root-only `schema_has_field_ids` in the schema adapter stays as the
420-
/// gate for id matching, which only ever renames root fields.
426+
/// walks the read schema. The planner runs this over the requested schema once, at plan time.
427+
/// The file side is not an Arrow walk at all: the reader factory checks the Parquet schema in
428+
/// the footer, because the Arrow schema the adapter sees can lose ids (the INT96 coercion
429+
/// rebuilds container fields without their metadata) and never shows an id that sits on a
430+
/// `list` or `key_value` group. The root-only `schema_has_field_ids` in the schema adapter
431+
/// stays as the gate for id matching, which only ever renames root fields.
421432
pub(crate) fn any_nested_field_has_id(fields: &Fields) -> bool {
422433
fields.iter().any(|f| field_holds_id(f))
423434
}
424435

425-
/// Whether `field` or anything nested under it carries a Parquet field id. Dictionary and
426-
/// run-end-encoded wrappers are not walked, because the Parquet read path never nests a struct,
427-
/// list or map inside them.
436+
/// Whether `field` or anything nested under it carries a Parquet field id.
437+
///
438+
/// This walks the requested schema, where only struct fields can hold the metadata: Spark's
439+
/// `hasFieldIds` recurses through `ArrayType` and `MapType` into their element and key or value
440+
/// types, only a `StructField` carries metadata, and the serde never populates the element or
441+
/// key and value fields. The walk still descends through list and map fields to reach the
442+
/// structs nested inside them. The file side is checked by the reader factory over the raw
443+
/// Parquet schema, where any node can carry an id, as Spark's `containsFieldIds` does.
444+
///
445+
/// Dictionary and run-end-encoded wrappers are not walked, because the Parquet read path never
446+
/// nests a struct, list or map inside them.
428447
fn field_holds_id(field: &Field) -> bool {
429448
field_id(field).is_some()
430449
|| match field.data_type() {

0 commit comments

Comments
 (0)