Repository navigation
Commit ba95460
committed
fix: keep extension types when skip_metadata clears Parquet field metadata
With `skip_metadata = true` (the default), Parquet schema inference cleared all
field metadata, including the `ARROW:extension:name` and
`ARROW:extension:metadata` keys. Columns with an extension type were read as
their storage type.
Keep the extension type keys and clear the rest: the extension type is part of
the type of a field, not a writer annotation. Only top-level fields are
affected: the metadata of nested fields was never cleared.
This is the fix proposed in apache#26517.
It changes how existing files with extension types (for example `arrow.uuid`)
are read.1 parent 27b047b commit ba95460
1 file changed
Lines changed: 13 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
21 | 21 | | |
22 | 22 | | |
23 | 23 | | |
| 24 | + | |
24 | 25 | | |
25 | 26 | | |
26 | 27 | | |
| |||
242 | 243 | | |
243 | 244 | | |
244 | 245 | | |
245 | | - | |
| 246 | + | |
| 247 | + | |
246 | 248 | | |
247 | 249 | | |
248 | 250 | | |
| |||
251 | 253 | | |
252 | 254 | | |
253 | 255 | | |
254 | | - | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
255 | 266 | | |
256 | 267 | | |
257 | 268 | | |
| |||
0 commit comments