Repository navigation
Commit 0eb1e18
apacheGH-39914: [Python] Skip unparseable complex dtype metadata instead of guessing ArrowDtype
Address review from @jorisvandenbossche: a parse failure on the
'numpy_type' metadata string shouldn't be read as "the user wants an
ArrowDtype" -- it can happen for reasons other than a complex/nested
Arrow type. Replace the schema-field-type fallback with `continue`,
leaving the column out of the metadata-driven extension-dtype handling
entirely, same as the existing KeyError case right above it.
Also broadened the except clause from `TypeError` to
`(TypeError, ValueError)`. Traced why: np.dtype() is tried before the
ArrowDtype parser, and for a 'numpy_type' string containing a comma
(struct, dictionary, map -- anything but a single-field list) it
misreads the comma as its own legacy multi-field format syntax and
raises ValueError, never reaching the TypeError the list case gets.
Verified directly against pandas_dtype() and np.dtype() on pandas 2.3.3:
the previous TypeError-only fix left struct/dictionary/map still
crashing.
Updated the regression test to match: it now asserts the column lands
as object (list/struct) or category (dictionary) dtype with no crash,
rather than an exact ArrowDtype round-trip -- which this fix no longer
attempts, per the review above. Added a dictionary-dtype case alongside
the existing list/struct ones since it exercises the same ValueError
path.
Co-Authored-By: Claude Sonnet 5 <[email protected]>1 parent b141fb3 commit 0eb1e18
2 files changed
Lines changed: 36 additions & 16 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
904 | 904 | | |
905 | 905 | | |
906 | 906 | | |
907 | | - | |
| 907 | + | |
908 | 908 | | |
909 | 909 | | |
910 | 910 | | |
911 | | - | |
912 | | - | |
913 | | - | |
| 911 | + | |
| 912 | + | |
| 913 | + | |
| 914 | + | |
| 915 | + | |
| 916 | + | |
| 917 | + | |
| 918 | + | |
914 | 919 | | |
915 | | - | |
916 | | - | |
917 | | - | |
918 | | - | |
919 | | - | |
| 920 | + | |
920 | 921 | | |
921 | 922 | | |
922 | 923 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
4570 | 4570 | | |
4571 | 4571 | | |
4572 | 4572 | | |
4573 | | - | |
4574 | | - | |
4575 | | - | |
4576 | | - | |
| 4573 | + | |
| 4574 | + | |
| 4575 | + | |
| 4576 | + | |
| 4577 | + | |
| 4578 | + | |
| 4579 | + | |
| 4580 | + | |
| 4581 | + | |
| 4582 | + | |
| 4583 | + | |
| 4584 | + | |
| 4585 | + | |
4577 | 4586 | | |
4578 | 4587 | | |
4579 | 4588 | | |
4580 | 4589 | | |
4581 | | - | |
4582 | 4590 | | |
4583 | 4591 | | |
4584 | | - | |
| 4592 | + | |
| 4593 | + | |
4585 | 4594 | | |
4586 | 4595 | | |
4587 | 4596 | | |
| |||
4594 | 4603 | | |
4595 | 4604 | | |
4596 | 4605 | | |
4597 | | - | |
| 4606 | + | |
| 4607 | + | |
| 4608 | + | |
| 4609 | + | |
| 4610 | + | |
| 4611 | + | |
| 4612 | + | |
| 4613 | + | |
| 4614 | + | |
| 4615 | + | |
| 4616 | + | |
4598 | 4617 | | |
4599 | 4618 | | |
4600 | 4619 | | |
| |||
0 commit comments