Core, Parquet: Null counts are incorrect for child fields under a null struct #17561

Apache Iceberg version

None

Query engine

None

Please describe the bug 🐞

When an optional struct is null, OptionWriter writes a null directly to every leaf column it contains, so the writers for those columns never see the value and cannot count it. OptionWriter dropped its own null count in that case, with a comment saying nested null stats were not used. They are used: the counts it returns become DataFile.nullValueCounts.

Float and double are the only types whose writers report metrics, and ParquetMetrics prefers writer metrics over footer statistics, so the correct footer count was never used. A float or double under a nullable struct was reported as having 0 nulls even when the struct was null for some rows.

The incorrect counting of nulls could affect query engines that relay on this stats for optimization. For example, they could simply skip the file with null_count == 0 for predicate WHERE c.f_id IS NULL and produces wrong result.

Fix by PR: #17560

Willingness to contribute

• I can contribute a fix for this bug independently
• I would be willing to contribute a fix for this bug with guidance from the Iceberg community
• I cannot contribute a fix for this bug at this time