Skip to content

Iceberg write docs say the float/double null-count difference can't change IS NULL pruning, but it can #7050

Closed

Description

@andygrove

Describe the bug

docs/source/user-guide/latest/iceberg-writes.md says this about float and double fields nested under a nullable struct:

On Iceberg 1.9+, manifest value_counts / null_value_counts for float/double columns nested under a nullable struct count rows whose parent struct is null (they come from the parquet footer), while iceberg-java's writer-tracked counts do not. Both counts inflate by the same amount, so the derived null ratios and IS NULL / IS NOT NULL pruning decisions are unaffected.

The IS NULL half is wrong. When every present struct has a value for the field, iceberg-java records null_value_counts = 0 for it even though the struct is NULL in some rows. Iceberg's planner then prunes the file for WHERE s.x IS NULL and loses every NULL-struct row. That is an iceberg-java bug, apache/iceberg#17561, with a fix in apache/iceberg#17560. The native writer's footer counts include those rows, so its files are kept and the query returns the right rows. IS NOT NULL decisions do agree.

Steps to reproduce

With Comet disabled, on main at c230bf4:

CREATE TABLE t (id INT, s STRUCT<x: DOUBLE>) USING iceberg;
INSERT INTO t SELECT id, IF(id % 2 = 0, named_struct('x', CAST(id AS DOUBLE)), NULL) FROM range(10);
SELECT count(*) FROM t WHERE s.x IS NULL;
Iceberg null_value_counts of s.x in each file count(*)
1.5.2 (Spark 3.4.3) and 1.8.1 (Spark 3.5.9) 1 5
1.11.0 (Spark 4.1.3) 0 0

FLOAT behaves like DOUBLE. With an INT field, every version records 1 and returns 5.

Expected behavior

The bullet says that the native counts are the footer's, that iceberg-java's leave out the NULL-struct rows because of apache/iceberg#17561, and that iceberg-java's counts can prune files for IS NULL that the native counts keep. That is also why #6659 should stay closed: matching iceberg-java's counts would copy the wrong result.

Additional context

Found in a nested-type audit of the native writers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions