Skip to content

Commit 53cb606

Browse files
adriangbclaude
andcommitted
refactor(proto): carry apache#23999 into the moved PartitionedFile decode
apache#23999 landed in `datafusion-proto` while this branch was open, fixing the duplicate-partition-statistics decode that apache#23998 described. The moved copy in `datafusion-datasource` predates it, so replaying the move on top of current main would have silently reverted the fix. Apply it to the copy that now owns the wire logic, so the delegating shim behaves exactly as `main` does today. Upstream's regression test, `partitioned_file_statistics_roundtrip_with_partition_values`, exercises the shim and therefore pins the moved impl. The round-trip test here asserted only that statistics survived decode, because on the old base they did not survive it faithfully. That caveat is gone: it now asserts the decoded statistics equal the originals, and that they span the full table schema. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
1 parent 8213bf9 commit 53cb606

1 file changed

Lines changed: 11 additions & 6 deletions

File tree

‎datafusion/datasource/src/proto.rs‎

Lines changed: 11 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -123,7 +123,10 @@ impl TryFrom<&protobuf::PartitionedFile> for PartitionedFile {
123123
pf = pf.with_range(range.start, range.end);
124124
}
125125
if let Some(proto_stats) = file.statistics.as_ref() {
126-
pf = pf.with_statistics(Arc::new(proto_stats.try_into()?));
126+
// The wire format carries statistics for the full table schema (file + partition
127+
// columns), so assign directly — `with_statistics` would append the partition
128+
// column stats a second time.
129+
pf.statistics = Some(Arc::new(proto_stats.try_into()?));
127130
}
128131
Ok(pf)
129132
}
@@ -191,11 +194,13 @@ mod tests {
191194
assert_eq!(decoded.partition_values, pf.partition_values);
192195
assert_eq!(decoded.range, pf.range);
193196
assert_eq!(decoded.arrow_schema.as_deref(), Some(schema.as_ref()));
194-
// Statistics on `PartitionedFile` span the full table schema, and
195-
// `PartitionedFile::with_statistics` re-derives the partition column
196-
// entries, so the decoded statistics are not compared field by field
197-
// here; this is pre-existing behavior of the wire format.
198-
assert!(decoded.statistics.is_some());
197+
// Statistics span the full table schema (file columns followed by one
198+
// entry per partition column), and survive the round trip intact.
199+
assert_eq!(
200+
pf.statistics.as_ref().unwrap().column_statistics.len(),
201+
schema.fields().len() + pf.partition_values.len()
202+
);
203+
assert_eq!(decoded.statistics, pf.statistics);
199204
Ok(())
200205
}
201206

0 commit comments

Comments
 (0)