Repository navigation
Commit 6644732
fix: render binary columns as hex in DataFrame::describe() (apache#21728)
## Which issue does this PR close?
- Closes apache#21496.
## Rationale for this change
`DataFrame::describe()` is the standard way to get a statistical summary
of a DataFrame (count, null_count, mean, std, min, max, median per
column). Today it handles binary-like columns poorly:
- For `Binary`, an exclusion filter in `min`/`max` aggregations caused
both to be reported as `null`, losing useful information for columns
that hold hashes, UUIDs, fingerprints, or other content-addressed
identifiers.
- For `LargeBinary`, `BinaryView`, and `FixedSizeBinary`, the filter did
not apply, so `min`/`max` ran successfully but then the display step
tried to `cast(column, Utf8)`, which Arrow correctly rejects, producing
an `ArrowError::CastError` that bubbled up and failed the whole
`describe()` call.
The fix in this PR is aligned with what the issue proposes: stop
filtering `Binary` from the aggregations and render binary outputs as
lowercase hex (matching Arrow's default display of binary arrays).
## What changes are included in this PR?
- `datafusion/core/src/dataframe/mod.rs`:
- Drop `DataType::Binary` from the `min`/`max` exclusion filter (now
only `Boolean` is excluded, which is still meaningful for a statistical
summary).
- Add a dedicated display branch for `Binary`, `LargeBinary`,
`BinaryView`, and `FixedSizeBinary` that uses
`arrow::util::display::ArrayFormatter` with default options, which
renders bytes as lowercase hex.
- Tidy a now-stale comment that referenced the previous binary filter.
- Drive-by: use the newly imported `FormatOptions` unqualified in
`DataFrame::to_string()` for consistency.
## Are these changes tested?
Yes, a new integration test `describe_binary_columns` in
`datafusion/core/tests/dataframe/describe.rs` builds an in-memory
`RecordBatch` with one column per binary-like type and asserts the full
`describe()` output via an inline `insta` snapshot. The test covers
non-null values and a null row per column, so it exercises both
`null_count` and the hex rendering path for `min`/`max`.
All existing `describe` tests continue to pass unchanged.
## Are there any user-facing changes?
Yes — this is a visible behavior change for `DataFrame::describe()`:
- Before: `min`/`max` on `Binary` columns were `null`; other binary-like
types caused a cast error.
- After: `min`/`max` on all binary-like types render as lowercase hex
strings (e.g. `"0001"`, `"ffee"`).
No public API changes.
---------
Co-authored-by: Jeffrey Vo <[email protected]>1 parent 7621299 commit 6644732
2 files changed
Lines changed: 88 additions & 10 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
48 | 48 | | |
49 | 49 | | |
50 | 50 | | |
| 51 | + | |
51 | 52 | | |
52 | 53 | | |
53 | 54 | | |
| |||
979 | 980 | | |
980 | 981 | | |
981 | 982 | | |
982 | | - | |
983 | | - | |
| 983 | + | |
| 984 | + | |
| 985 | + | |
| 986 | + | |
| 987 | + | |
| 988 | + | |
| 989 | + | |
984 | 990 | | |
985 | 991 | | |
986 | 992 | | |
| |||
1074 | 1080 | | |
1075 | 1081 | | |
1076 | 1082 | | |
1077 | | - | |
1078 | | - | |
1079 | | - | |
| 1083 | + | |
1080 | 1084 | | |
1081 | 1085 | | |
1082 | 1086 | | |
| |||
1085 | 1089 | | |
1086 | 1090 | | |
1087 | 1091 | | |
1088 | | - | |
1089 | | - | |
1090 | | - | |
| 1092 | + | |
1091 | 1093 | | |
1092 | 1094 | | |
1093 | 1095 | | |
| |||
1126 | 1128 | | |
1127 | 1129 | | |
1128 | 1130 | | |
| 1131 | + | |
| 1132 | + | |
| 1133 | + | |
| 1134 | + | |
| 1135 | + | |
| 1136 | + | |
| 1137 | + | |
| 1138 | + | |
| 1139 | + | |
| 1140 | + | |
| 1141 | + | |
| 1142 | + | |
| 1143 | + | |
| 1144 | + | |
| 1145 | + | |
| 1146 | + | |
1129 | 1147 | | |
1130 | 1148 | | |
1131 | 1149 | | |
1132 | 1150 | | |
1133 | 1151 | | |
1134 | 1152 | | |
1135 | 1153 | | |
1136 | | - | |
| 1154 | + | |
| 1155 | + | |
1137 | 1156 | | |
1138 | 1157 | | |
1139 | 1158 | | |
| |||
1517 | 1536 | | |
1518 | 1537 | | |
1519 | 1538 | | |
1520 | | - | |
| 1539 | + | |
1521 | 1540 | | |
1522 | 1541 | | |
1523 | 1542 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
15 | 15 | | |
16 | 16 | | |
17 | 17 | | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
18 | 24 | | |
19 | 25 | | |
20 | 26 | | |
| |||
112 | 118 | | |
113 | 119 | | |
114 | 120 | | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
115 | 174 | | |
116 | 175 | | |
117 | 176 | | |
| |||
0 commit comments