BUG: Preserve NumPy complex scalars in object data - #982

closes #976

Takes over #976 by @galafis, whose commit is kept unchanged, and implements the review comments on it.

The bug (from #976). as_float64 converted object data with astype(float64) first. NumPy complex scalars (np.complex128(1 + 2j), np.complex64) in object data were cast to their real parts with a ComplexWarning. The imaginary parts were therefore lost with allow_complex=True, and complex data were accepted with allow_complex=False. Python complex elements were already handled, since casting them to float64 raises.

The change (review). All object data now go through one helper, _object_values. It is used for object ndarrays (as_float64, check_array) and for the object columns of a Series or DataFrame (ensure1d, ensure2d).

• The values are cast to complex128 in one NumPy call. An element that is not a number raises the existing TypeError.
• If every imaginary part is 0, the real parts are returned as float64. Otherwise the data are complex128 if allow_complex, and a TypeError if not.
• There is no Python loop over the elements (the isinstance scans in BUG: Preserve NumPy complex scalars in object data #976 are gone), and no ComplexWarning.
• None is still rejected in an ndarray and still becomes NaN in pandas data, since pandas treats it as missing. It is found before the cast, since NumPy casts None to nan+nanj, which would look like complex data.
• In a DataFrame, if an object column turns out to be complex, every column is converted to complex128, as for a complex dtype column. Columns are matched by position, so duplicated labels and a duplicated index are kept.

Behaviour changes against main.

• NumPy complex scalars with a nonzero imaginary part become complex128 with allow_complex=True; on main they became float64 with a ComplexWarning. Without allow_complex they now raise a TypeError; on main they were accepted.
• Complex elements whose imaginary parts are all 0 (2 + 0j, np.complex128(1)) are real data: float64 for both values of allow_complex. On main, such Python complex values were rejected without allow_complex and became complex128 with it, and the NumPy ones became float64 with a ComplexWarning.
• The message for rejected complex object data is now "my_data must contain real numbers that can be converted to float64; got complex numbers in an object array with shape (2,) (ndarray)".

Everything else is unchanged on 21 kinds of object data (ndarray, Series and DataFrame, both values of allow_complex): numeric strings, Decimal/Fraction, large ints, longdouble and float32 scalars, empty arrays, NaN and None.

Tests. arch/tests/utility/test_dtype_policy.py adds:

• NumPy complex64 and complex128 scalars, 2-d object data and a complex NaN;
• zero imaginary parts, for the four helpers and three containers, with warnings as errors;
• None in pandas object data;
• a DataFrame with real, complex-object and real-object columns and duplicated labels.

The new tests fail on main, and the zero-imaginary ones also fail on the head of #976.

• test_dtype_policy.py: 519 passed with -W error, and also on Python 3.10 with NumPy 1.23.5 and pandas 1.5.3.
• Fast suite: 27,041 passed, 19 skipped (numba not installed, and inapplicable parametrizations).
• After merging main: arch/tests/utility and test_dtype_inputs.py, 1,487 passed.
• flake8, ruff, black, isort and numpydoc lint are clean, and mypy has no new errors.

🤖 Generated with Claude Code