Skip to content

Commit 58b94d6

Browse files
eendebakptclaude
andcommitted
PERF: avoid array scalars in the string-like casts
The legacy `STRING/UNICODE/VOID_to_<T>` cast loops build a full array scalar for every element and hand it to the destination's setitem. For S and U sources that scalar is needlessly expensive: `PyArray_Scalar` for NPY_UNICODE allocates a scratch buffer, builds a `str`, wraps it in an argument tuple and then runs `str.__new__` on the `np.str_` subtype, which copies the data a second time. The plain bytes/str returned by getitem behaves identically in every setitem and is much cheaper. Use getitem for S and U sources, keeping `PyArray_Scalar` for VOID sources (where `void.item()` and `bool(void)` disagree, numpygh-9847, the reason the scalar was introduced in numpygh-9856) and for VOID destinations (`VOID_setitem` reads the source through the buffer protocol, which `np.str_` supports but `str` does not). Measured on 2e5 elements: `U -> int64` 3.1x faster, `U -> float64` 2.5x, `U -> bool` 6.5x, `U -> S` 3.8x, `S -> int64` 1.4x, `S -> float64` 1.35x. The reproducer in numpygh-11014 is resolved: `np.array(string_array, dtype=int)` goes from 2.2x slower than `np.array(string_array.tolist(), dtype=int)` to slightly faster. Closes numpygh-11014 Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
1 parent 3274e70 commit 58b94d6

1 file changed

Lines changed: 16 additions & 1 deletion

File tree

‎numpy/_core/src/multiarray/arraytypes.c.src‎

Lines changed: 16 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1515,6 +1515,18 @@ BOOL_to_@TOTYPE@(void *input, void *output, npy_intp n,
15151515
/**end repeat**/
15161516

15171517

1518+
/*
1519+
* Per-element getters for the string-like casts below. For S/U sources the
1520+
* plain bytes/str from getitem behaves like the array scalar in setitem and is
1521+
* much cheaper (gh-11014). PyArray_Scalar stays for VOID sources (gh-9847)
1522+
* and for VOID destinations, where VOID_setitem needs the buffer protocol that
1523+
* np.str_ has but str does not.
1524+
*/
1525+
#define _GETOBJ_STRING(ip, aip) STRING_getitem(ip, aip)
1526+
#define _GETOBJ_UNICODE(ip, aip) UNICODE_getitem(ip, aip)
1527+
#define _GETOBJ_SCALAR(ip, aip) \
1528+
PyArray_Scalar(ip, PyArray_DESCR(aip), (PyObject *)(aip))
1529+
15181530
/**begin repeat
15191531
*
15201532
* #from = STRING*23, UNICODE*23, VOID*23#
@@ -1537,6 +1549,9 @@ BOOL_to_@TOTYPE@(void *input, void *output, npy_intp n,
15371549
* #oskip = 1*18,(PyArray_ITEMSIZE(aop))*3,1*2,
15381550
* 1*18,(PyArray_ITEMSIZE(aop))*3,1*2,
15391551
* 1*18,(PyArray_ITEMSIZE(aop))*3,1*2#
1552+
* #getobj = (_GETOBJ_STRING)*20, _GETOBJ_SCALAR, (_GETOBJ_STRING)*2,
1553+
* (_GETOBJ_UNICODE)*20, _GETOBJ_SCALAR, (_GETOBJ_UNICODE)*2,
1554+
* (_GETOBJ_SCALAR)*23#
15401555
*/
15411556

15421557
static void
@@ -1552,7 +1567,7 @@ static void
15521567
int oskip = @oskip@;
15531568

15541569
for (i = 0; i < n; i++, ip+=skip, op+=oskip) {
1555-
PyObject *temp = PyArray_Scalar(ip, PyArray_DESCR(aip), (PyObject *)aip);
1570+
PyObject *temp = @getobj@(ip, aip);
15561571
if (temp == NULL) {
15571572
return;
15581573
}

0 commit comments

Comments
 (0)