Native Parquet scan returns a non-null array or struct of NULLs where Spark returns NULL because no requested nested field is in the file #7030

Describe the bug

When none of the requested fields under a nested column exist in a Parquet file, Spark's legacy behavior returns the column as NULL, but the native Parquet scan returns a non-null array or struct that holds NULLs. Legacy is the only behavior on Spark 3.4, 3.5 and 4.0, and it is what Spark 4.1+ does with spark.sql.legacy.parquet.returnNullStructIfAllFieldsMissing=true. The shapes are an array whose element struct has no field in the file, a struct whose only requested child is such a struct, and a struct whose requested children all lead to missing fields. Schema evolution that replaced a nested struct's fields produces them. The results are wrong without an error: count(col) and col IS NULL both disagree with Spark.

Spark's ParquetColumnVector marks any struct, array or map vector all-null when every child vector is missing, and it applies this bottom-up (Spark 3.5.9 ParquetColumnVector.java lines 111-134). An array whose element struct has no field in the file is therefore missing itself, and so is a struct whose requested children are all missing at some depth. Comet's parquet_convert_struct_to_struct (native/core/src/parquet/parquet_support.rs lines 565-576) only nulls a struct when none of its direct requested fields matches a file field, and the list conversion keeps the list's own validity. So the list stays non-null with NULL elements, and a struct whose child exists by name but has no matching leaves stays non-null.

Steps to reproduce

In a CometTestBase suite:

spark.sql("""SELECT 1 AS id,
  |  array(named_struct('x', 1, 'y', 'a')) AS arrs,
  |  named_struct('a', 1, 'c', named_struct('d', 2)) AS s""".stripMargin)
  .coalesce(1).write.parquet(path)
val df = spark.read
  .schema("id int, arrs array<struct<w:int>>, s struct<c:struct<n:int>>")
  .parquet(path)
df.collect()
df.selectExpr("count(arrs)", "count(s)").collect()

On main at 66c63bd:

Over a wider generated dataset the same holds for sm struct<sa: array<struct<z9: int>>> (Spark NULL, Comet {[null, ...]}) and for sm struct<mm: map<string,int>, sa: array<struct<z9: int>>>, where Spark keeps sm but returns sa as NULL. A map whose value struct has no field in the file matches Spark, because the keys are present.

Expected behavior

With the legacy behavior in effect, a requested nested column, array element or struct field whose subtree has no field in the file reads as NULL at the highest level whose whole subtree is missing, as Spark's reader does.

Additional context

Found in a nested-type audit of the native scans. Spark 4.1's default behavior (SPARK-53535) matches already. A related detail: on Spark before 4.1, Comet honors spark.sql.legacy.parquet.returnNullStructIfAllFieldsMissing=false when a session sets it (CometNativeScan.populateScanConfFlags), although Spark 3.x ignores that 4.1-only key, so setting it there makes even the plain all-missing struct diverge.