DEP: deprecate implicit casts and promotion between text and bytes by ngoldbaum · Pull Request #32757 · numpy/numpy

ngoldbaum · 2026-09-22T20:34:55Z

PR summary

Fixes #32880. Fixes #23899. Also addressed the mixed bytes/text detection requested in #32765.

Over in https://github.com/numpy/numpy/pull/32749/changes#r4072990536 @jorenham and I identified an inconsistency with how python handles mixed string and bytes operands in string and bytes methods.

First, this changes the default fillchar in np.strings.center, ljust, and rjust from " " to None. This lets us choose a space matching the input dtype without implicitly converting text to bytes.

Second, it deprecates implicit bytes/text promotion, safe and same_kind casts between bytes (np.bytes_) and text (np.str_ and StringDType), and mixed bytes/text arguments in string operations. For example, these now emit DeprecationWarning:

import numpy as np

np.strings.replace(b"abc", "a", "x")
np.array([b"abc", "def"])
np.array([b"abc"]).astype("U", casting="same_kind")

These checks live in NumPy's C casting and promotion machinery. The np.strings wrappers keep temporary compatibility handling for the deprecation period. Unsafe casts (the default for astype) and array construction with an explicit dtype are unchanged.

I added a new NPY_COERCION_POLICY environment variable that is read once during import. Setting NPY_COERCION_POLICY=strict before importing NumPy enables the future behavior: the deprecated operations raise, and np.can_cast("S1", "U1") returns False. By default, casting queries keep their existing answers without warnings during the deprecation period.

This affects downstream code outside np.strings too. In Astropy, dataframe conversion inserts b"" for missing text, while table operations and metadata merging rely on bytes/text promotion. I've opened astropy/astropy#20599 to handle these cases while preserving existing encoding behavior. I also looked at Pandas, H5Py, and scikit-learn and did not see any issues caused by this new deprecation.

Finally, this updates the type stubs to match these changes.

AI Disclosure

I iterated on this using an AI model.

seberg

Two comments. I'd love to fix this one level lower, but since I am not sure how viable it is and strings need some annoying extra handling anyway (e.g. to deal with the default), I am not opposed to this.

It does seem like there is absolutely no reason not to use None as default, though?

seberg · 2026-09-23T08:18:25Z

-            "The fill character must be exactly one character long")
+    fillchar = _get_fillchar(a, fillchar)

    if np.result_type(a, fillchar).char == "T":


Maybe it doesn't matter, but as a note. There is an underlying issue here that:

np.result_type() happily promotes stringes/bytes and everything else to strings. I would love to fix this, but IIRC there was some fun fallout (that may be fixable, IIRC a main thing was that some code relied on it working for later things to not fail).

That means that np.concatenate(([b"1", "2"])) just as happily works for example.

(I think that can_cast("U1", "S1", casting="safe") is true is probably even fine.)

Bringing this up not to stop it, but to say that there is more to this and I suppose I semi-care about this if we don't attack the rest. It might even be interesting to start with result_type(..., strict_strings=True) even if private.

It might even be interesting to start with result_type(..., strict_strings=True) even if private.

I went ahead and did that because I think it's a cleaner fix. I'm also about to file a followup issue citing your comment about this.

Co-authored-by: Joren Hammudoglu <[email protected]>

…result_type

ngoldbaum · 2026-09-23T17:34:39Z

I ended up applying part of the deeper structural solution @seberg was thinking about. It does make the diff bigger though, let me know what you think. I'll file a followup issue about the array coercion issues that Sebastian pointed out.

jorenham

Stub changes look right. I'll dot the "i"'s in #32749 once this is merged.

jorenham · 2026-10-05T08:57:37Z

How's this going? Anything I can do to help?

ngoldbaum · 2026-10-05T09:15:31Z

I think this just needs another review pass looking at the functional changes.

mhvk

I only had a short look, but this fix feels weird to me, and at the wrong level.

In particular, I do not see why one should have to adjust _array_converter -- the logic belongs in casting generally, where effectively you want the strings to only cast T to U and vice versa, and S only to itself. Logically (to me at least), this should happen for "same_kind":

Checking what actually happens seems to bring up an inconsistency (see #32880):
(NOTE: edited T->S to use "S1" instead of "S", which is allowed, while U->S is not.)

np.array(["a"], dtype="U").astype("S", casting="same_kind")
# OK TypeError: Cannot cast array data from dtype('<U1') to dtype('S1') according to the rule 'same_kind'
np.array(["a"], dtype="T").astype("S1", casting="same_kind")
# Huh, why is this one OK?
# array([b'a'], dtype='|S1')
np.array([b"a"]).astype("U", casting="same_kind")
# Huh?? array(['a'], dtype='<U1')
np.array([b"a"]).astype("T", casting="same_kind")
# huh?? array(['a'], dtype=StringDType())

Why is it possible to cast "S" to "U" or "T" and not vice versa with "same_kind"? Either things are the same kind or they are not!

I think if one were to fix that inconsistency, the fix here would be far simpler: just add casting="same_kind" to the relevant .astype() calls.

Alternatively, since the ufuncs do already seem to have the correct behaviour, can one rely on those? I.e., pass things through to the ufunc and only on failure raise a deprecation warning and retry?

a = np.array("a", dtype="T")
out = np.array("X")
np._core.umath._replace(a, "a", "b", 1, out)
# array('b', dtype='<U1')
np._core.umath._replace(a, "a", b"b", 1, out)
# UFuncTypeError: ufunc '_replace' did not contain a loop with signature matching types (<class 'numpy.dtypes.StrDType'>, <class 'numpy.dtypes.StrDType'>, <class 'numpy.dtypes.BytesDType'>, <class 'numpy.dtypes._PyLongDType'>) -> None

So, I would have expected can_cast to reflect this,

Logically, I would have thought that this would correspond to casting="same_kind" -- except that for reasons that I do not understand "S" and "U" are considered same kind...

ngoldbaum · 2026-10-05T15:12:28Z

Thanks for giving this a deep look @mhvk! I'll hopefully have time next week to dive into this and the other issue you opened.

seberg · 2026-10-05T16:29:50Z

I don't think this is a can_cast issue, but rather a result_type issue? np.result_type("S5", "U5") doesn't really make sense. By comparison casting is somewhat OK (yes, it exists the way it is because of Python 2 strings/unicode, but...).

That said, yes, I agree that the fix really be yet one level deeper, which is why I hadn't pushed it through, I guess.

My half-thought, by now is that we should maybe have a TLS strict_promotion (maybe also used for these casts). If set, we reject any of the crazy promotions we have:

anything to string (nope, bad!)
Integers to datetime (same, terrible!).

Last time I tried to actually do that change, there was weird astropy fallout with masked tables or so. But I now do wonder a bit if a TLS with warn/error/allow might be useful enough anyway? Even if we only use error in some narrow cases (e.g. the minmax ufunc promoter) and warn if we want to change an existing bad function.

(I don't like a TLS for this, but the reason why this ended up in _array_converter is that you'd need to pipe through multiple layers probably :/.)

ngoldbaum · 2026-10-13T15:41:53Z

I don't like a TLS for this

Well, there's ContextVars as well. Of course that doesn't play nicely with newly initialized threads. Hopefully someone (me? 😰) will need to take a close look at figuring out a way to unify the behavior of context variables and newly spawned threads on the free-threaded and GIL-enabled build. We tried to do it very late in the 3.15 cycle (you may have seen me thinking about it during the EuroPython sprints) and it opened a whole can of worms. But even with the behavior difference I think it might be OK for this purpose? I'll have to check.

I'll try to think all of these suggestions through. I think I'm in favor of moving in the direction of removing text/bytes casts and implicit promotion in favor of explicit encoding and decoding. We can establish that behavior from the outset for ByteStringDType's interactions with text dtypes, as proposed in NEP 58. Changing existing StringDType/S and S/U behavior would need a migration paths.

So to combine what Marten and Sebastian were saying, I think we want an opt-in mode that applies stricter rules to both promotion and text/bytes casting. That would let us gauge breakage before committing to a wider deprecation cycle. We may have to leave legacy S/U conversions alone if that is used all over the place in code that was last touched for the Python 2/3 transition. Ideally we'd be able to make StringDType/ByteStringDType behave sensibly to provide a long-term target for us to eventually land on.

seberg · 2026-10-14T09:35:58Z

I mentioned TLS, because I expect we can at least start with it only being private and C-side (effectively) and there it's good enough, I think. Otherwise, yeah, just use contextvars and live with the current limitation...

ngoldbaum · 2026-10-18T16:18:19Z

I just pushed a new version that deprecates implicit text/bytes promotion and starts a deprecation for the text/bytes casting safety. I also added a new NPY_COERCION_POLICY environment variable to enable and test the future behavior. I didn't end up using TLS storage because the version I added can only be enabled via an environment variable that is read once during import.

I also updated the PR description and title.

mhvk

Wow, that became quite a bit longer than perhaps hoped! But it all looks good; my comment is only in case there are still other changes; if not, just get it in!

mhvk · 2026-10-18T17:51:15Z

+     * Remove these branches once text/bytes casts are classified as unsafe;
+     * see void_to_void_resolve_descriptors_with_flags.
+     */
+    if (cast_flags != NULL && castingimpl->resolve_descriptors ==


Could combine the cast_flags != NULL pieces.
Actually, I assume the compiler will anyway... But perhaps just that bit clearer.

ngoldbaum · 2026-10-18T20:50:38Z

my comment is only in case there are still other changes; if not, just get it in!

The ASan CI flagged something so I pulled in your suggestion as well.

ngoldbaum · 2026-10-18T23:38:46Z

@seberg like Marten said this is a big change and I'd like to make sure you're on board as well.

seberg

Some of this is to at least consider slightly wider approaches. I would like to keep in mind that the timedelta/datetime promotion is just as big a problem as the string one.

I didn't end up using TLS storage because the version I added can only be enabled via an environment variable that is read once during import.

Yeah, the main reason for why I wanted that was that if it is a TLS, we could set the strict mode during ufunc promotion.
I.e. a few places call ResultType/CommonDType and at least one of those already works-around this (for datetimes though!).
So that would harden ufuncs and fix Iasons minimummaximum promoter in a subtle way (for datetimes, I guess strings all through here, currently).

I suppose we can't avoid the environment variable in order to support np.can_cast() to just return False (and maybe avoid a spurious warning for weird promotions)?
(In can_cast() itself, I guess this is technically a FutureWarning, but it is probably implausible to make that work, which is then the reason for the env variable.)

seberg · 2026-10-19T09:39:26Z

+    if (text_bytes && npy_coercion_is_strict()) {
+        PyErr_SetString(_npy_module_state->static_pydata.CoercionPolicyError,
+                "Promoting strings and bytes is not allowed.");
+        return NULL;


This isn't quite right here (I dunno what input you need to make it go, it's not trivial, this one does it):

np.result_type("S10", "U10", "S10", "O", "U10")

I.e. if we returned NotImplemented, than the above would end up working fine (compare with "T" rather than "U10").

Although, I wouldn't be surprised if I added similar issues, and we could actually make result_type() logic ignore errors (the warning is harder though).

FWIW, I think for np.array(...) things always work left-to-right and there is already no implicit object fallback (so the DeprecationWarning pans out, it actually goes to an error.)

I fixed the bug you pointed out. I also removed the private exception by making a small behavior change to the final strict promotion state we'd like to end up using. Removing the exception means that np.array([b"a", "b"]) ends up as an object array if strict promotion is turned on. Which makes sense to me and is consistent with NumPy's general promotion rules.

seberg · 2026-10-19T10:37:02Z

+        for a in arrays
+    ):
+        return False
+    return np.result_type(*arrays).char in _VARIABLE_LENGTH


So, the fact you avoid result_type() here and have custom _cast_string_arg() below, is basically all to give nicer warnings here?

Not sure I am excited about it, but OK. I can see a point in not giving two warnings (which requires a force-cast).

(I guess part of why I am wondering is, is that this is the only reason you need to expose the _STRICT to Python.)

These changes to the python wrappers are needed until we can change the casting safety in a future NumPy version. Once we switch the safety to "same_kind" then we can remove these wrappers. Would more comments or something else help to make that clear?

seberg · 2026-10-19T10:41:24Z

+_VARIABLE_LENGTH = "T"
+
+_STRING_WARNING_SKIP_PREFIXES = _get_warn_skip_file_prefixes(
+    __file__, os.path.join(os.path.dirname(__file__), "defchararray.py"),


I have to get used to using this, so side-question: Wouldn't it make sense to just always skip all of NumPy? (The main problem I see there, is that our tests are in-tree, but we don't test warning-levels, so...)

I think we probably could but we'd have to deal with the tests being in-tree. I just hate dealing with stacklevel so I wanted to use this and there's unfortunately an annoying bug we have to work around on 3.13 and older.

seberg · 2026-10-19T10:52:00Z

-    PyArray_Descr *from = PyArray_DESCR(arr);
-    PyArray_DTypeMeta *to_dtype = NPY_DTYPE(to);
+    return can_cast_to(PyArray_DESCR(arr), to, casting,
+            PyArray_FLAGS(arr) & NPY_ARRAY_WAS_PYTHON_LITERAL, 0);


So, this doesn't actually deprecate this for the PyArray_CanCastArrayTo public C-API? Is that intentional?

(but if we push this down, then that would be the case)

It is intentional to leave the C API alone. The internal versions return an int so callers can propagate warnings raised as errors.

seberg · 2026-10-19T11:21:36Z

    }
+    if (cast_flags != NULL && NPY_DT_is_text_bytes_cast(NPY_DTYPE(from), to_dtype)) {
+        *cast_flags |= NPY_CAST_TEXT_BYTES;
+    }


Hmmm, OK. So I can see that resolve_descriptors can't give the warning as such (as it can only pass out the actual cast safety).

But as much as I hate it as well, can't we add a _CAST_WILL_BE_UNSAFE kind of flag to the casts, then have PyArray_MinCastSafety() propagate it (together with save-value).
There is one explicit PyArray_MinCastSafety(...) == casting in the ufunc module that would have to deal with giving the right warning/error as well.

I am not sure how that plays out. The main reason to push this is that I think datetime promotions are just as a big an issue and I think that pattern may generalize better to them.
(I.e. the reason why you don't have to deal with the ufunc path there, is that right now ufuncs don't matter for strings.)

I adopted this approach, it ended up marginally less code and is a bit cleaner, thanks.

seberg · 2026-10-19T11:29:04Z

+                "is deprecated and will raise TypeError in a future "
+                "release. Use numpy.strings.encode or numpy.strings.decode "
+                "with an explicit encoding, or request casting='unsafe' "
+                "to retain the existing casting behavior.") < 0) {


Add the (Deprecated NumPy 2.6) also to the message (maybe also others).

ngoldbaum · 2026-11-04T18:33:35Z

@seberg @mhvk gentle ping since the 2.6 branching date is creeping up.

mhvk · 2026-11-09T17:20:28Z

I won't have time to review again, but was happy already before. I think the behaviour is all good; the main worry would be whether we are committing to a path the precludes solving other, related problems. I don't think so, but @seberg has a much better overview.

I do see the advantage of doing this in 2.6, so the ByteString PRs do not have to worry about it.

seberg · 2026-11-11T19:05:58Z

Thanks, I should look at this closer, but I think the solution now to check at the end should be OK, if maybe a little awkward.
It would be nice to have the TLS, because we need the "future" behavior in ufunc promotion (unconditionally, it never makes sense not to use it).

The casting, I have not thought about quite as much, but the flag seems sensible on first thought (I would have thought there it's actually possibly to bubble up the warning/error, but I am not sure).

ngoldbaum added 07 - Deprecation component: numpy.strings String dtypes and functions labels Sep 22, 2026

jorenham reviewed Sep 22, 2026

View reviewed changes

Comment thread numpy/_core/strings.pyi Resolved

jorenham reviewed Sep 22, 2026

View reviewed changes

Comment thread doc/release/upcoming_changes/32757.deprecation.rst Resolved

jorenham reviewed Sep 22, 2026

View reviewed changes

Comment thread doc/release/upcoming_changes/32757.deprecation.rst Resolved

jorenham reviewed Sep 22, 2026

View reviewed changes

Comment thread numpy/_core/strings.py Resolved

Comment thread numpy/_core/tests/test_deprecations.py Resolved

Comment thread doc/release/upcoming_changes/32757.deprecation.rst Resolved

seberg reviewed Sep 23, 2026

View reviewed changes

jorenham mentioned this pull request Sep 23, 2026

TYP: strings.* shape-typing #32749

Open

ngoldbaum and others added 7 commits September 23, 2026 11:27

DEP: deprecate implicit bytes to string conversion in np.strings

0c9d386

TYP: use NoValueType to fix stubtest

dacee94

DOC: add release note fragment

b957c4b

DOC: tweak release note title

c217300

Update 32757.deprecation.rst

0565fa4

Co-authored-by: Joren Hammudoglu <[email protected]>

MAINT: centralize string/bytes argument checking in _array_converter.…

d371f7c

…result_type

API: use None as the default value for fill characters

52514f5

ngoldbaum force-pushed the deprecate-implicit-conv branch from 41a0d05 to 52514f5 Compare September 23, 2026 17:28

ngoldbaum mentioned this pull request Sep 23, 2026

ENH: Detect mixed bytes/text inputs during array conversion #32765

Open

ngoldbaum added a commit to ngoldbaum/numpy that referenced this pull request Sep 23, 2026

DEP: include strict string argument conversion from numpy#32757

bc3dc6f

ngoldbaum added a commit to ngoldbaum/numpy that referenced this pull request Sep 23, 2026

ENH: integrate ByteStringDType strict promotion with numpy#32757

a53d0d2

ngoldbaum mentioned this pull request Sep 23, 2026

ENH: add ByteStringDType to NumPy #32772

Draft

2 of 4 tasks

ngoldbaum added a commit to ngoldbaum/numpy that referenced this pull request Sep 24, 2026

DEP: include strict string argument conversion from numpy#32757

49d4571

ngoldbaum added a commit to ngoldbaum/numpy that referenced this pull request Sep 24, 2026

ENH: integrate ByteStringDType strict promotion with numpy#32757

271a1b8

ngoldbaum added a commit to ngoldbaum/numpy that referenced this pull request Sep 24, 2026

DEP: include strict string argument conversion from numpy#32757

d077855

ngoldbaum added a commit to ngoldbaum/numpy that referenced this pull request Sep 24, 2026

ENH: integrate ByteStringDType strict promotion with numpy#32757

2b703b7

jorenham approved these changes Sep 29, 2026

View reviewed changes

mhvk mentioned this pull request Oct 5, 2026

BUG: same_kind casting between strings and bytes is inconsistent #32880

Open

mhvk reviewed Oct 5, 2026

View reviewed changes

ngoldbaum mentioned this pull request Oct 14, 2026

Avoid implicit bytes/text coercion in tables and metadata astropy/astropy#20599

Merged

1 of 2 tasks

ngoldbaum added 3 commits October 18, 2026 10:02

DEP: deprecate implicit text/bytes casting and promotion

9f2b4e0

Merge branch 'main' into deprecate-implicit-conv

4ed9a7b

MAINT: fixup patch application

a0bad13

ngoldbaum changed the title ~~DEP: deprecate implicit bytes to string conversion in np.strings~~ DEP: deprecate implicit casts and promotion between text and bytes Oct 18, 2026

ngoldbaum mentioned this pull request Oct 18, 2026

BUG: np.array() assumes that bytestrings are ASCII, if mixed with strings #23899

Open

mhvk approved these changes Oct 18, 2026

View reviewed changes

MAINT: delete redundant NULL checks; fix ASan

6d36a77

seberg reviewed Oct 19, 2026

View reviewed changes

ngoldbaum added 4 commits October 28, 2026 16:14

MAINT: respond to sebastian's review

7c13afc

Merge branch 'main' into deprecate-implicit-conv

7ce0974

MAINT: move flag definition to the same spot as the others

772dfa1

MAINT: drop unnecessary exception

f458732

ngoldbaum mentioned this pull request Oct 29, 2026

ENH: Fix promotion for odd integer dtypes and mixed floats #33105

Open

BUG: move _NPY_CAST_WILL_BE_UNSAFE into NPY_CASTING

85f4431

Merge branch 'main' into deprecate-implicit-conv

506e851

This comment was marked as resolved.

Sign in to view

Conversation

ngoldbaum commented Sep 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR summary

AI Disclosure

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

seberg left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ngoldbaum commented Sep 23, 2026

Uh oh!

jorenham left a comment

Choose a reason for hiding this comment

Uh oh!

jorenham commented Oct 5, 2026

Uh oh!

ngoldbaum commented Oct 5, 2026

Uh oh!

mhvk left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ngoldbaum commented Oct 5, 2026

Uh oh!

seberg commented Oct 5, 2026

Uh oh!

ngoldbaum commented Oct 13, 2026

Uh oh!

seberg commented Oct 14, 2026

Uh oh!

ngoldbaum commented Oct 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mhvk left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ngoldbaum commented Oct 18, 2026

Uh oh!

ngoldbaum commented Oct 18, 2026

Uh oh!

seberg left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

ngoldbaum commented Sep 22, 2026 •

edited

Loading

seberg left a comment •

edited

Loading

mhvk left a comment •

edited

Loading

ngoldbaum commented Oct 18, 2026 •

edited

Loading