Skip to content

Commit 863f11d

Browse files
authored
fix: reject a file without field ids at any depth whether or not id matching is on (apache#6116)
* fix: reject a file without field ids at any depth whether or not id matching is on Spark's `ParquetReadSupport.getRequestedSchema` raises when the requested schema carries a Parquet field id at any depth and the file carries none at any depth, unless `spark.sql.parquet.fieldId.read.ignoreMissing` is set. The check runs on every read, for both readers, and does not consult `spark.sql.parquet.fieldId.read.enabled`. The native scan ran that check only with the read flag on, only over root fields, and only inside the name remap, which a case-sensitive session with the flag off never reaches. A read schema with ids over a file without ids returned rows where Spark raises, and a file whose ids sit only on nested fields was rejected where Spark reads it and null-fills the unmatched fields. The check now runs at the top of the expression adapter factory with a recursive predicate on both schemas. Id matching itself stays gated on the read flag and root ids, as Spark gates it per struct level. The Iceberg scan opts out, since its reader resolves columns by id and supplies the ids itself. * fix: answer both halves of the missing-field-id check from what Spark reads Spark checks the pruned required schema for ids and the raw Parquet schema for their absence. The requested half now comes from `required_schema` at plan time, since the logical schema DataFusion hands the adapter is the full read schema, so an id on a column the query never projects no longer raises. The file half moves into the eager page index reader, which walks the raw schema from the footer. The Arrow schema the adapter sees has lost container metadata after INT96 coercion and never carries ids on repeated list or key value groups, so a file whose only id sat on a struct holding a timestamp raised where Spark reads. The JNI error conversion unwraps the Parquet external error so the Java side keeps the same exception, and the Iceberg scan no longer needs an opt-out since it does not use that reader. Tests cover the pruned projection, the timestamp struct, an id only on a list or key value group, a directory mixing files with and without ids, nested schema pruning, id zero, count(*), and the exception class on each version. * test: find the missing field ids error anywhere in the cause chain On Spark 4.0 and 4.1 the RuntimeException Spark raises for a read schema with field ids over a file without any arrives wrapped once, in the FAILED_READ_FILE SparkException that collect raises, and the helper assumed one more layer above it. It now walks the cause chain and requires a plain RuntimeException carrying Spark's message, for Spark and for Comet alike, so the layer count no longer matters. * fix: let the serde say whether the file must carry field ids The requested side of the missing field ids check is Spark's own hasFieldIds over the required schema, so the serde now sends one flag, require_field_ids, in place of ignore_missing_field_id, and the native Arrow walk is gone. The reader factory takes that one flag through a builder and the parquet options carry nothing for it. The error names the file, and the module doc leads with the lasting reason the check reads the raw footer: ids on repeated groups and on the message root never reach Arrow. The Rust scan tests that had Scala twins are gone, the repeated group cases moved to Scala where they compare against Spark, and the Scala tests fold into the existing port and one pruned schema test with Spark's own assertion form.
1 parent bc4be39 commit 863f11d

10 files changed

Lines changed: 496 additions & 73 deletions

File tree

‎native/common/src/error.rs‎

Lines changed: 19 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -233,11 +233,12 @@ pub enum SparkError {
233233
matched_fields: String,
234234
},
235235

236-
/// The read schema requests Parquet field-id matching but the file carries no field ids.
237-
/// Mirrors the runtime error raised in Spark's `ParquetReadSupport` when
238-
/// `spark.sql.parquet.fieldId.read.ignoreMissing` is false.
236+
/// The read schema carries Parquet field ids but the file carries none. Mirrors the runtime
237+
/// error raised in Spark's `ParquetReadSupport` when
238+
/// `spark.sql.parquet.fieldId.read.ignoreMissing` is false. Spark's message names no file,
239+
/// so `file_path` travels as a parameter for the 4.x shim's `FAILED_READ_FILE` wrapper.
239240
#[error("Spark read schema expects field Ids, but Parquet file schema doesn't contain any field Ids. Please remove the field ids from Spark schema or ignore missing ids by setting `spark.sql.parquet.fieldId.read.ignoreMissing = true`")]
240-
ParquetMissingFieldIds,
241+
ParquetMissingFieldIds { file_path: String },
241242

242243
/// Schema mismatch when reading a Parquet column under a requested schema
243244
/// that's incompatible with the physical column type. Translated by the JVM
@@ -353,7 +354,7 @@ impl SparkError {
353354
SparkError::FileNotFound { .. } => "FileNotFound",
354355
SparkError::DuplicateFieldCaseInsensitive { .. } => "DuplicateFieldCaseInsensitive",
355356
SparkError::DuplicateFieldByFieldId { .. } => "DuplicateFieldByFieldId",
356-
SparkError::ParquetMissingFieldIds => "ParquetMissingFieldIds",
357+
SparkError::ParquetMissingFieldIds { .. } => "ParquetMissingFieldIds",
357358
SparkError::ParquetSchemaConvert { .. } => "ParquetSchemaConvert",
358359
SparkError::CannotReadFile { .. } => "CannotReadFile",
359360
SparkError::Arrow(_) => "Arrow",
@@ -604,6 +605,11 @@ impl SparkError {
604605
"matchedFields": matched_fields,
605606
})
606607
}
608+
SparkError::ParquetMissingFieldIds { file_path } => {
609+
serde_json::json!({
610+
"filePath": file_path,
611+
})
612+
}
607613
SparkError::ParquetSchemaConvert {
608614
file_path,
609615
column,
@@ -711,10 +717,11 @@ impl SparkError {
711717
// (Spark's `foundDuplicateFieldInFieldIdLookupModeError` returns SparkRuntimeException)
712718
SparkError::DuplicateFieldByFieldId { .. } => "org/apache/spark/SparkRuntimeException",
713719

714-
// ParquetMissingFieldIds - converted to a plain RuntimeException by the shim,
715-
// matching the `RuntimeException` Spark's ParquetReadSupport throws when the
716-
// file lacks field ids and `spark.sql.parquet.fieldId.read.ignoreMissing=false`.
717-
SparkError::ParquetMissingFieldIds => "java/lang/RuntimeException",
720+
// ParquetMissingFieldIds - converted to the plain RuntimeException Spark's
721+
// ParquetReadSupport throws when the file lacks field ids and
722+
// `spark.sql.parquet.fieldId.read.ignoreMissing=false`. The 4.x shim wraps it in
723+
// the FAILED_READ_FILE SparkException Spark 4 raises at the task boundary.
724+
SparkError::ParquetMissingFieldIds { .. } => "java/lang/RuntimeException",
718725

719726
// ParquetSchemaConvert - converted to SchemaColumnConvertNotSupportedException by the shim
720727
SparkError::ParquetSchemaConvert { .. } => {
@@ -814,8 +821,9 @@ impl SparkError {
814821
// Duplicate field id in id-lookup mode
815822
SparkError::DuplicateFieldByFieldId { .. } => Some("_LEGACY_ERROR_TEMP_2094"),
816823

817-
// ParquetMissingFieldIds is a plain RuntimeException with no error class.
818-
SparkError::ParquetMissingFieldIds => None,
824+
// ParquetMissingFieldIds is a plain RuntimeException with no error class. The 4.x
825+
// shim supplies FAILED_READ_FILE itself, so none is exposed here.
826+
SparkError::ParquetMissingFieldIds { .. } => None,
819827

820828
// Parquet schema mismatch — translated to SchemaColumnConvertNotSupportedException
821829
// by the JVM shim. The shim wraps it in the version-appropriate

‎native/core/src/execution/planner.rs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1743,7 +1743,7 @@ impl PhysicalPlanner {
17431743
self.session_ctx(),
17441744
common.encryption_enabled,
17451745
common.use_field_id,
1746-
common.ignore_missing_field_id,
1746+
common.require_field_ids,
17471747
)?;
17481748
Ok((
17491749
vec![],

‎native/core/src/parquet/eager_page_index_reader_factory.rs‎

Lines changed: 195 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -43,8 +43,22 @@
4343
//! fetch to encrypted scans that have no pruning predicate at all. Encrypted opens get exactly
4444
//! the caller's requested policy, unchanged from stock behavior.
4545
//!
46-
//! Filed upstream as apache/datafusion#23978. Revert this once the opener merges its deferred
47-
//! page-index load back into `FileMetadataCache` instead of bypassing it.
46+
//! Filed upstream as apache/datafusion#23978. Once the opener merges its deferred page-index
47+
//! load back into `FileMetadataCache` instead of bypassing it, the eager policy can go, but the
48+
//! factory cannot: `get_metadata` is the one per-file hook that sees the raw footer, and two
49+
//! other things hang off it.
50+
//!
51+
//! The first is Spark's missing field id check. `ParquetReadSupport` refuses to open a file
52+
//! whose Parquet schema carries no field id when the requested schema carries one, unless
53+
//! `ignoreMissing` is set, and it walks the raw `MessageType` to decide. That walk has to run
54+
//! over the Parquet schema rather than the Arrow schema the schema adapter is handed later,
55+
//! because an id on a repeated `list` or `key_value` group, or on the message root, never
56+
//! reaches an Arrow field. Until apache/datafusion#24790, which is not in DataFusion 55.1.0,
57+
//! the INT96 coercion also rebuilt container fields without their metadata, so a struct id could
58+
//! vanish on the way to Arrow as well.
59+
//!
60+
//! The second is the Variant footer rewrite, `with_spark_arrow_schema`, which replaces the
61+
//! Arrow schema hint in the footer for scans that project Variant.
4862
4963
use arrow::datatypes::{DataType, FieldRef, Schema};
5064
use async_trait::async_trait;
@@ -58,6 +72,7 @@ use datafusion::execution::cache::cache_manager::FileMetadataCache;
5872
use datafusion::physical_plan::metrics::{
5973
Count, ExecutionPlanMetricsSet, MetricBuilder, MetricCategory, MetricType,
6074
};
75+
use datafusion_comet_common::SparkError;
6176
use datafusion_datasource::PartitionedFile;
6277
use futures::future::BoxFuture;
6378
use futures::{FutureExt, StreamExt, TryStreamExt};
@@ -77,7 +92,7 @@ use parquet::file::metadata::{FileMetaData, KeyValue, ParquetMetaDataBuilder};
7792
use parquet::file::metadata::{
7893
FooterTail, PageIndexPolicy, ParquetMetaData, ParquetMetaDataReader,
7994
};
80-
use parquet::schema::types::{ColumnDescPtr, SchemaDescriptor};
95+
use parquet::schema::types::{ColumnDescPtr, SchemaDescriptor, Type as ParquetType};
8196
use std::fmt::{Debug, Display, Formatter};
8297
use std::ops::Range;
8398
use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering};
@@ -163,6 +178,9 @@ pub struct EagerPageIndexReaderFactory {
163178
// Enable the footer workaround only for scans that project Variant.
164179
// https://github.com/apache/datafusion-comet/issues/5477
165180
spark_variant_schema: bool,
181+
// Refuse a file whose Parquet schema carries no field id, as Spark's `ParquetReadSupport`
182+
// does when the requested schema carries one and `ignoreMissing` is not set.
183+
require_field_ids: bool,
166184
}
167185

168186
impl EagerPageIndexReaderFactory {
@@ -191,13 +209,32 @@ impl EagerPageIndexReaderFactory {
191209
metadata_cache,
192210
scan_io_metrics,
193211
spark_variant_schema: false,
212+
require_field_ids: false,
194213
}
195214
}
196215

197216
pub fn with_spark_variant_schema(mut self, enabled: bool) -> Self {
198217
self.spark_variant_schema = enabled;
199218
self
200219
}
220+
221+
/// Refuse a file whose Parquet schema carries no field id. Off by default, so a factory
222+
/// that never calls this reads every file.
223+
pub fn with_require_field_ids(mut self, enabled: bool) -> Self {
224+
self.require_field_ids = enabled;
225+
self
226+
}
227+
}
228+
229+
/// True when `node` or any node under it carries a field id, the way Spark's
230+
/// `containsFieldIds` answers it over the raw Parquet schema, message root included.
231+
fn contains_field_ids(node: &ParquetType) -> bool {
232+
node.get_basic_info().has_id()
233+
|| (node.is_group()
234+
&& node
235+
.get_fields()
236+
.iter()
237+
.any(|field| contains_field_ids(field)))
201238
}
202239

203240
impl ParquetFileReaderFactory for EagerPageIndexReaderFactory {
@@ -225,6 +262,7 @@ impl ParquetFileReaderFactory for EagerPageIndexReaderFactory {
225262
metadata_cache: Arc::clone(&self.metadata_cache),
226263
metadata_size_hint,
227264
spark_variant_schema: self.spark_variant_schema,
265+
require_field_ids: self.require_field_ids,
228266
}))
229267
}
230268
}
@@ -240,6 +278,7 @@ struct EagerPageIndexReader {
240278
metadata_cache: Arc<FileMetadataCache>,
241279
metadata_size_hint: Option<usize>,
242280
spark_variant_schema: bool,
281+
require_field_ids: bool,
243282
}
244283

245284
// Arrow infers ENUM as Binary, losing the distinction from raw binary that Spark needs.
@@ -439,6 +478,7 @@ impl AsyncFileReader for EagerPageIndexReader {
439478
let metadata_size_hint = self.metadata_size_hint;
440479
let scan_io_metrics = Arc::clone(&self.scan_io_metrics);
441480
let spark_variant_schema = self.spark_variant_schema;
481+
let require_field_ids = self.require_field_ids;
442482
async move {
443483
let file_decryption_properties = options
444484
.and_then(|o| o.file_decryption_properties())
@@ -498,6 +538,18 @@ impl AsyncFileReader for EagerPageIndexReader {
498538
}
499539

500540
let metadata = metadata?;
541+
// Spark's missing field id check, over the raw Parquet schema as in
542+
// `ParquetReadSupport`. The JNI layer unwraps the `External` error, so the JVM sees
543+
// the exception Spark raises.
544+
if require_field_ids
545+
&& !contains_field_ids(metadata.file_metadata().schema_descr().root_schema())
546+
{
547+
return Err(ParquetError::External(Box::new(
548+
SparkError::ParquetMissingFieldIds {
549+
file_path: object_meta.location.to_string(),
550+
},
551+
)));
552+
}
501553
if spark_variant_schema {
502554
with_spark_arrow_schema(metadata)
503555
} else {
@@ -833,7 +885,7 @@ mod tests {
833885
use arrow::{array::Int32Array, record_batch::RecordBatch};
834886
use object_store::memory::InMemory;
835887
use parquet::{
836-
arrow::ArrowWriter,
888+
arrow::{ArrowWriter, PARQUET_FIELD_ID_META_KEY},
837889
file::{
838890
properties::{EnabledStatistics, WriterProperties},
839891
reader::FileReader,
@@ -992,6 +1044,145 @@ mod tests {
9921044
);
9931045
}
9941046

1047+
/// The raw schema walk answers like Spark's `containsFieldIds`: an id on the message root,
1048+
/// on a leaf, or on a repeated `list` or `key_value` group counts, and a schema without
1049+
/// any does not.
1050+
#[test]
1051+
fn contains_field_ids_sees_ids_on_any_node() {
1052+
use parquet::basic::Type as PhysicalType;
1053+
use parquet::schema::types::TypePtr;
1054+
1055+
let leaf = |name: &str, id: Option<i32>| -> TypePtr {
1056+
Arc::new(
1057+
ParquetType::primitive_type_builder(name, PhysicalType::INT32)
1058+
.with_id(id)
1059+
.build()
1060+
.unwrap(),
1061+
)
1062+
};
1063+
let group = |name: &str, id: Option<i32>, fields: Vec<TypePtr>| -> TypePtr {
1064+
Arc::new(
1065+
ParquetType::group_type_builder(name)
1066+
.with_id(id)
1067+
.with_fields(fields)
1068+
.build()
1069+
.unwrap(),
1070+
)
1071+
};
1072+
1073+
assert!(!contains_field_ids(&group("schema", None, vec![])));
1074+
assert!(!contains_field_ids(&group(
1075+
"schema",
1076+
None,
1077+
vec![leaf("a", None)]
1078+
)));
1079+
assert!(contains_field_ids(&group(
1080+
"schema",
1081+
Some(1),
1082+
vec![leaf("a", None)]
1083+
)));
1084+
assert!(contains_field_ids(&group(
1085+
"schema",
1086+
None,
1087+
vec![leaf("a", Some(1))]
1088+
)));
1089+
let list_group_only = group(
1090+
"schema",
1091+
None,
1092+
vec![group(
1093+
"l",
1094+
None,
1095+
vec![group("list", Some(5), vec![leaf("element", None)])],
1096+
)],
1097+
);
1098+
assert!(contains_field_ids(&list_group_only));
1099+
let key_value_group_only = group(
1100+
"schema",
1101+
None,
1102+
vec![group(
1103+
"m",
1104+
None,
1105+
vec![group(
1106+
"key_value",
1107+
Some(6),
1108+
vec![leaf("key", None), leaf("value", None)],
1109+
)],
1110+
)],
1111+
);
1112+
assert!(contains_field_ids(&key_value_group_only));
1113+
let nested_without_ids = group(
1114+
"schema",
1115+
None,
1116+
vec![group("s", None, vec![leaf("a", None)])],
1117+
);
1118+
assert!(!contains_field_ids(&nested_without_ids));
1119+
}
1120+
1121+
/// Write a one-column `a: int32` file into `store`, with a field id on `a` when `id` is
1122+
/// set, and return the file as `PartitionedFile`.
1123+
async fn put_int_file(store: &InMemory, location: &str, id: Option<&str>) -> PartitionedFile {
1124+
let mut field = arrow::datatypes::Field::new("a", DataType::Int32, false);
1125+
if let Some(id) = id {
1126+
field = field
1127+
.with_metadata([(PARQUET_FIELD_ID_META_KEY.to_string(), id.to_string())].into());
1128+
}
1129+
let schema = Arc::new(Schema::new(vec![field]));
1130+
let batch = RecordBatch::try_new(
1131+
Arc::clone(&schema),
1132+
vec![Arc::new(Int32Array::from(vec![1, 2]))],
1133+
)
1134+
.unwrap();
1135+
let mut bytes = Vec::new();
1136+
let mut writer = ArrowWriter::try_new(&mut bytes, schema, None).unwrap();
1137+
writer.write(&batch).unwrap();
1138+
writer.close().unwrap();
1139+
let size = bytes.len() as u64;
1140+
store
1141+
.put(&Path::from(location), Bytes::from(bytes).into())
1142+
.await
1143+
.unwrap();
1144+
PartitionedFile::new(location.to_string(), size)
1145+
}
1146+
1147+
/// A reader made by a factory with `require_field_ids` set refuses a file without ids on
1148+
/// its first metadata fetch, naming the file, and reads one that carries an id. A factory
1149+
/// without it reads the file without ids.
1150+
#[tokio::test]
1151+
async fn get_metadata_refuses_a_file_without_ids_only_when_required() {
1152+
let store = Arc::new(InMemory::new());
1153+
let without_ids = put_int_file(&store, "no_ids.parquet", None).await;
1154+
let with_ids = put_int_file(&store, "ids.parquet", Some("1")).await;
1155+
let runtime = datafusion::execution::runtime_env::RuntimeEnv::default();
1156+
let metrics = ExecutionPlanMetricsSet::new();
1157+
let metadata_for = |require: bool, file: PartitionedFile| {
1158+
let factory = EagerPageIndexReaderFactory::new(
1159+
Arc::clone(&store) as Arc<dyn ObjectStore>,
1160+
runtime.cache_manager.get_file_metadata_cache(),
1161+
ScanIoSource::Local,
1162+
&metrics,
1163+
)
1164+
.with_require_field_ids(require);
1165+
let mut reader = factory.create_reader(0, file, None, &metrics).unwrap();
1166+
async move { reader.get_metadata(None).await }
1167+
};
1168+
1169+
let err = metadata_for(true, without_ids.clone())
1170+
.await
1171+
.expect_err("a file without ids must be refused");
1172+
match err {
1173+
ParquetError::External(inner) => assert!(
1174+
matches!(
1175+
inner.downcast_ref::<SparkError>(),
1176+
Some(SparkError::ParquetMissingFieldIds { file_path }) if file_path == "no_ids.parquet"
1177+
),
1178+
"unexpected error: {inner}"
1179+
),
1180+
other => panic!("unexpected error: {other}"),
1181+
}
1182+
assert!(metadata_for(true, with_ids).await.is_ok());
1183+
assert!(metadata_for(false, without_ids).await.is_ok());
1184+
}
1185+
9951186
#[test]
9961187
fn variant_policy_preserves_footer_metadata_and_indexes() {
9971188
let schema = Arc::new(Schema::new_with_metadata(

‎native/core/src/parquet/parquet_exec.rs‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -83,7 +83,7 @@ pub(crate) fn init_datasource_exec(
8383
session_ctx: &Arc<SessionContext>,
8484
encryption_enabled: bool,
8585
use_field_id: bool,
86-
ignore_missing_field_id: bool,
86+
require_field_ids: bool,
8787
) -> Result<Arc<DataSourceExec>, ExecutionError> {
8888
// Computed once and reused below for `try_pushdown_filters`. `copied_config()` clones only
8989
// `SessionConfig` (an `Arc<ConfigOptions>` plus a small extensions map); `SessionContext::
@@ -101,7 +101,6 @@ pub(crate) fn init_datasource_exec(
101101
&session_config.options().execution.parquet,
102102
);
103103
spark_parquet_options.use_field_id = use_field_id;
104-
spark_parquet_options.ignore_missing_field_id = ignore_missing_field_id;
105104
// Spark can discard filtered-out values before timestamp conversion using statistics,
106105
// dictionary, and row-level filters. Comet cannot mirror every pruning path, so applying
107106
// checked conversion in a filtered scan can fail on values Spark never reads. Preserve the
@@ -194,7 +193,8 @@ pub(crate) fn init_datasource_exec(
194193
scan_io_source,
195194
parquet_source.metrics(),
196195
)
197-
.with_spark_variant_schema(projects_variant),
196+
.with_spark_variant_schema(projects_variant)
197+
.with_require_field_ids(require_field_ids),
198198
);
199199
parquet_source = parquet_source.with_parquet_file_reader_factory(reader_factory);
200200

0 commit comments

Comments
 (0)