BUG/ENH: fix casting and setitem nan behavior between StringDType and float16/32 by ngoldbaum · Pull Request #31825 · numpy/numpy

ngoldbaum · 2026-07-01T17:49:53Z

PR summary

This was originally part of #31682 but @seberg asked me to split it off. I also took his comments on that version about how I handled complex values into account.

Currently, we only support correctly parsing NaN values for float64 for casting and setitem. This leads to this (IMO) strange asymmetry:

>>> import numpy as np
>>> sdt = np.dtypes.StringDType(na_object=np.nan)
>>> arr = np.array([1, np.nan, 4.5], dtype=np.float32)
>>> arr.astype(sdt)
array(['1.0', 'nan', '4.5'], dtype=StringDType(na_object=nan))
>>> isinstance(arr.astype(sdt)[1], str)
True
>>> arr = np.array([1, np.nan, 4.5], dtype=np.float64)
>>> arr.astype(sdt)
array(['1.0', nan, '4.5'], dtype=StringDType(na_object=nan))
>>> arr.astype(sdt)[1] is np.nan
True

IMO this is just a straight-up bug in the implementation, which relies on coercing the value to a Python scalar, and fails to catch e.g. np.float32('nan') because float32 scalars aren't subtypes of PyFloat. That said, it is an observable behavior change.

We don't parse any complex-valued NaN as a NaN missing data value, leading to behavior like this:

>>> arr.astype(np.dtypes.StringDType(na_object=np.nan))
array(['(nan+0j)', 'nanj'], dtype=StringDType(na_object=nan))

This behavior is a bit more arguable to change, but under the principle that missing values should round-trip correctly via casting and that setitem should behave identically to casting, I also added behavior for complex values with nan real values to coerce to a null string if the na_object is NaN-like. This followes this existing behavior in NumPy:

>>> arr = np.array([complex(np.nan, 3.5), complex(0, np.nan), complex(3.5, np.nan), complex(np.nan, np.nan)])
>>> arr
array([nan+3.5j, 0. +nanj, 3.5+nanj, nan+nanj])
>>> arr.astype('float64')
<python-input-3>:1: ComplexWarning: Casting complex values to real discards the imaginary part
array([nan, 0. , 3.5, nan])

Note that the ComplexWarning happens unconditionally here for all complex values, even if the real part is zero.

I refactored StringDType internals to reflect this behavior. This refactoring included correcting some restructuring to reflect that the float to string casts might now produce null strings where has_nan_na was ignored before.

Added tests for this new behavior for floats and complex dtypes. Also adds a test that round-trips from float64 to float64 via complex, float64, and stringdtype, including all the permutations of that should round-trip.

AI Disclosure

An AI model initially noticed this issue. I also used an AI to brainstorm and review the code changes and tests.

…at16/32

seberg · 2026-07-14T09:24:06Z

Let me ping a random pandas maintainer @mroeschke in the hope of a quick thought.

To be honest, to me it seems very strange to distinguish complex NaNs, probably stanger than breaking the round-tripping of them.

ngoldbaum · 2026-07-14T10:45:22Z

FWIW Pandas doesn’t use StringDType so I’m not sure why they’d care here. Or were you just trying to ask about what behavior makes sense?

seberg · 2026-07-14T10:47:38Z

were you just trying to ask about what behavior makes sense?

Yes, mostly asking because I think pandas dev's have a better gut feeling for NaN/NA stuff then I do.

mroeschke · 2026-07-14T18:22:07Z

To be honest, to me it seems very strange to distinguish complex NaNs

I don't have much of a well informed opinion here so will tag in @jbrockmendel also.

My gut says a nan + nanj value seems more definitely a NaN missing data value with np.dtypes.StringDType(na_object=np.nan) as any non-nan real or imaginary part could be significant?

Otherwise in pandas, for this "several behaviors are possible" cases, we're trying to move toward raising and telling the user to preprocess the data to a more predictable state for the operation

jbrockmendel · 2026-07-14T19:32:51Z

I don't have a strong opinion on the exact API other than to aim for internal consistency. np.isnan returns True when either component is NaN, so I'd suggest treating that as NaN-like across the board.

ngoldbaum · 2026-07-14T20:06:22Z

OK, 3-against-1 on this one. I’ll go ahead and update this PR for the complex number case.

MaanasArora · 2026-07-15T09:32:06Z

Thanks Nathan, I went through in detail and couldn't find any correctness issues. Neither did two AI agents (nothing real).

I agree we shouldn't treat real/imaginary parts differently for NaN handling, but a minor concern I have with using either nan as the check is whether we would lose information in casts... (A little wild, but I can somewhat imagine someone trying to render partial complex NaNs into strings and just ending with lost non-nan real and imaginary parts!) To be fair though, e.g., real+nanj != real.nanj, so I could see the case for consistency (even with np.isnan).

Was just curious if this mattered, but don't block on me!

ngoldbaum · 2026-08-12T22:30:30Z

Sorry for the long delay here. I just pushed a commit implementing the suggestion to use the same rules as np.isnan for complex NaN.

seberg

Thanks, sorry about letting it hang, I am fine with basically just doing this definitely an improvement, it's all rather hard (what about quaddtype NaNs for example), but not sure we can fix this easily.

I am OK as is, although I would slightly lean to remove the ComplexWarning, so do with that what you think is right.

seberg · 2026-09-01T07:39:26Z

+        }
+        if (npy_gil_warning(npy_static_pydata.ComplexWarning, 1,
+                            "Casting complex values to real discards "
+                            "the imaginary part") < 0) {


I would honestly drop this warning (here and below). Yes, it's potentially lossy since we conflate real and complex part into a single "NA", but it isn't really clear to me and feels like it may just be annoying to deal with?

Agreed, thanks for pointing this out

seberg · 2026-09-01T07:42:09Z

+        return float_is_nan_na<npy_double>(value) ? 1 : 0;
+    }
+    if (PyComplex_Check(obj) || PyArray_IsScalar(obj, ComplexFloating)) {
+        Py_complex value = PyComplex_AsCComplex(obj);


My bot pointed out that we might overflow for longdoubles. Happy to ignore though, unless it warns (and I am not sure it does), it ends up at the same place anyway.

I simplified the error handling a bit and I think that satisfies the concern here. I thought about adding a test for complex longdouble but... meh.

ngoldbaum · 2026-09-01T22:17:49Z

Thanks for the review, let's bring this one in.

… float16/32 (numpy#31825)

BUG: fix casting and setitem nan behavior between StringDType and flo…

db24387

…at16/32

ngoldbaum added this to the 2.6.0 Release milestone Jul 1, 2026

ngoldbaum added component: numpy.strings String dtypes and functions 00 - Bug 01 - Enhancement labels Jul 1, 2026

Merge branch 'main' into fix-float-nan-behavior

7f2b862

ngoldbaum mentioned this pull request Jul 6, 2026

BUG: centralized helper for output coerce and na_object in stringdtype operations #31884

Merged

charris mentioned this pull request Jul 10, 2026

BUG: centralized helper for output coerce and na_object in stringdtype operations (#31884) #31952

Merged

ngoldbaum added 3 commits August 12, 2026 15:48

Merge branch 'main' into fix-float-nan-behavior

29b907f

MAINT: simplify, trim comments, and apply review comments

2d71ffe

DOC: add release note

5ee894e

ngoldbaum added 3 commits August 20, 2026 08:10

BUG: apply fix from numpygh-32273

67d6f2e

Merge branch 'main' into fix-float-nan-behavior

e256e63

BUG: add needed cast for longdouble

c05a532

seberg approved these changes Sep 1, 2026

View reviewed changes

ngoldbaum added 3 commits September 1, 2026 14:20

Merge branch 'main' into fix-float-nan-behavior

9efaf0c

MAINT: remove ComplexWarning and simplify error handling

222a5c9

BUG: fix non-nantive byteorder problems

84495e2

ngoldbaum mentioned this pull request Sep 1, 2026

CI: intermittent hard crash of cython while building the limited API test extensions on Windows #32473

Open

ngoldbaum merged commit 6ac3096 into numpy:main Sep 1, 2026
91 checks passed

ngoldbaum mentioned this pull request Sep 2, 2026

BUG: Backport StringDType byteorder fixes #32481

Merged

ngoldbaum mentioned this pull request Sep 2, 2026

BUG: write bytes to canonical byteorder in StringDType casts #32474

Merged

ngoldbaum added a commit to ngoldbaum/numpy that referenced this pull request Sep 8, 2026

BUG/ENH: fix casting and setitem nan behavior between StringDType and…

204411e

… float16/32 (numpy#31825)

ngoldbaum added a commit to ngoldbaum/numpy that referenced this pull request Sep 8, 2026

BUG/ENH: fix casting and setitem nan behavior between StringDType and…

dd9e6bf

… float16/32 (numpy#31825)

ngoldbaum mentioned this pull request Sep 21, 2026

NEP: update NEP 58 text to reflect current discussion/prototype #32745

Merged

Conversation

ngoldbaum commented Jul 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR summary

AI Disclosure

Uh oh!

seberg commented Jul 14, 2026

Uh oh!

ngoldbaum commented Jul 14, 2026

Uh oh!

seberg commented Jul 14, 2026

Uh oh!

mroeschke commented Jul 14, 2026

Uh oh!

jbrockmendel commented Jul 14, 2026

Uh oh!

ngoldbaum commented Jul 14, 2026

Uh oh!

MaanasArora commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ngoldbaum commented Aug 12, 2026

Uh oh!

seberg left a comment

Choose a reason for hiding this comment

Uh oh!

seberg Sep 1, 2026

Choose a reason for hiding this comment

Uh oh!

ngoldbaum Sep 1, 2026

Choose a reason for hiding this comment

Uh oh!

seberg Sep 1, 2026

Choose a reason for hiding this comment

Uh oh!

ngoldbaum Sep 1, 2026

Choose a reason for hiding this comment

Uh oh!

ngoldbaum commented Sep 1, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

ngoldbaum commented Jul 1, 2026 •

edited

Loading

MaanasArora commented Jul 15, 2026 •

edited

Loading