Skip to content

[REGRESSION] Parquet row group pruning incorrectly prunes out row groups when columns names have . in them #5708

Closed

Description

@alamb

Describe the bug

Parquet row group pruning incorrectly prunes out row groups when columns names have . in them

To Reproduce

Use this file: spans.zip

Run using datafusion-cli:

❯ SELECT "service.name" FROM 'spans.parquet';
+--------------+
| service.name |
+--------------+
| frontend     |
+--------------+
1 row in set. Query took 0.002 seconds.
❯ SELECT "service.name" FROM 'spans.parquet' WHERE "service.name" = 'frontend'
0 rows in set. Query took 0.002 seconds.

Expected behavior

However if I disable row group pruning the same query works as expected and returns a single row

❯ set datafusion.execution.parquet.pruning=false;
0 rows in set. Query took 0.000 seconds.
❯ SELECT "service.name" FROM 'spans.parquet' WHERE "service.name" = 'frontend';
+--------------+
| service.name |
+--------------+
| frontend     |
+--------------+

Additional context

@jacobmarble found this in IOx: https://github.com/influxdata/influxdb_iox/issues/7225

And has identified that it was a regression introduced in #5419 (see https://github.com/influxdata/influxdb_iox/issues/7225#issuecomment-1472546654) ❤️

Note this will only generate wrong results because there is a column named "name" and "service.name" in the same file (because the pruning logic incorrectly uses the statistics for "name" for the predicate on "service.name"

If there were no column named "name" the predicate would fail to resolve the statistics and they would be ignored

Activity

  1. self-assigned this
    on Mar 23, 2023
  2. changed the title [-]Parquet row group pruning incorrectly prunes out row groups when columns names have `.` in them[/-] [+][REGRESSION] Parquet row group pruning incorrectly prunes out row groups when columns names have `.` in them[/+] on Mar 23, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions