Repository navigation
select distinct on map types yields ArrowError NotYetImplemented #15428
Description
Describe the bug
querying a table/parquet file with the following schema and query fails:
create table t (
m MAP<VARCHAR, REAL> NOT NULL
);
let df = ctx.sql("SELECT distinct m FROM t LIMIT 10").await?;
fails with:
Error: ArrowError(NotYetImplemented("not yet implemented: Map(Field { name: \"entries\", data_type: Struct([Field { name: \"key\", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"value\", data_type: Float32, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, false)"), None)
it runs into the unimplemented error that comes from arrow-rs: https://github.com/apache/arrow-rs/blob/15db256bcdd739ce3233d90b3242af15ae3945fa/arrow-row/src/lib.rs#L434
To Reproduce
parquet file with the given format that fails the query attached for convenience
results_20250325_003444.parquet.zip
Expected behavior
query returns correct results
Additional context
No response
Reacted by Leonid Ryzhyk
Activity
LSUDOKO commentedon Mar 28, 2025
take
Jefffrey commentedon Feb 3, 2026
Upstream:
MapArrayinRowConverterarrow-rs#7879Jefffrey commentedon Jul 8, 2026
Rowconverter support is available on datafusion main now; to progress this issue we should add some test coverage in our sqllogictests with different distinct/aggregation queries (testing with group by too, etc.)
e.g. in here https://github.com/apache/datafusion/blob/main/datafusion/sqllogictest/test_files/map.slt
PG1204 commentedon Jul 8, 2026
take
Hi team, I'd like to pick this up. Following up on Jefffrey's note that RowConverter support is now on main, I'll add sqllogictest coverage in map.slt for:
SELECT DISTINCT on Map columns
GROUP BY with Map columns as grouping keys
Aggregations (COUNT, etc.) over Map-typed inputs
Planning to include edge cases like empty maps, maps with null values, and duplicate keys. Will confirm scope if there are specific scenarios you'd like prioritized.
test: add sqllogictest coverage for DISTINCT / GROUP BY / aggregation…
test: add sqllogictest coverage for DISTINCT / GROUP BY / aggregation…