Repository navigation
np.array(string_array.tolist(), dtype=int) is faster than without the .tolist() #11014
Description
np.array(string_array.tolist(), dtype=int)
is almost twice as fast as
np.array(string_array, dtype=int)
Full demo script:
from timeit import timeit
import numpy as np
import sys
string_array = np.array(list(map(str, range(1000000))))
print(timeit(lambda: np.array(string_array, dtype=int), number=10))
print(timeit(lambda: np.array(string_array.tolist(), dtype=int), number=10))
print(np.__version__)
print(sys.version)Output:
3.470734696020372
2.058091988990782
1.14.2
3.6.2 (default, Jul 29 2017, 00:00:00)
[GCC 4.8.4]
This seems like an optimisation that numpy should be able to do automatically, and also possible a hint at an underlying performance problem.
Summary by @seberg 2021-11:
- As Eric notes at the end, the current timings should be mainly due to the weird casting functions.
- The solution will be to implement new-style casts (instead of the weird legacy cast function), for string to integer casts. Even the old functions are bad (they go via scalars!), but there is probably not much point in trying to improve them.
Activity
Dan-Patterson commentedon Apr 30, 2018
@alexmojaki I got a similar ratio of the times as you did using your script for numpy 1.13.3 but using IPython %timeit, the differences were less dramatic.
`%timeit -n10 -r10 np.array(string_array, dtype=int)
349 ms ± 4.07 ms per loop (mean ± std. dev. of 10 runs, 10 loops each)
%timeit -n10 -r10 np.array(string_array.tolist(), dtype=int)
317 ms ± 21.4 ms per loop (mean ± std. dev. of 10 runs, 10 loops each)`
Can you confirm your timing by other means?
alexmojaki commentedon Apr 30, 2018
Using more manual timing:
Dan-Patterson commentedon Apr 30, 2018
Something else must be going on. Using your script above, I ran it from n=2 to 10 and the ratios weren't as marked as yours. Here are a couple of the samples
`n = 2
plain took 0.6495838165283203 seconds
tolist took 0.622671365737915 seconds
n = 4
plain took 1.3122646808624268 seconds
tolist took 1.2245070934295654 seconds
n = 6
plain took 1.9206631183624268 seconds
tolist took 1.9051151275634766 seconds
n = 8
plain took 2.601041316986084 seconds
tolist took 2.455733299255371 seconds
n = 10
plain took 3.2278361320495605 seconds
tolist took 3.079383611679077 seconds`
alexmojaki commentedon May 1, 2018
You mentioned using numpy 1.13.3. Did you mean 1.14.3? If not, can you try with that? Also what version of Python are you using?
Dan-Patterson commentedon May 1, 2018
alexmojaki commentedon May 1, 2018
Well I just downgraded to 1.13.3, and now
tolist()is slower as one would expect. So I guess it's a new issue.eric-wieser commentedon May 1, 2018
@alexmojaki: Did
.tolist()get slower in 1.13 on your machine, or didarray(...)get faster (compared to 1.14)?alexmojaki commentedon May 1, 2018
It looks like in 1.14
arraygot slower. Here are some fresh runs with both scripts:1.13:
plain took 3.423121929168701 seconds
tolist took 3.9357383251190186 seconds
3.5675266589969397
3.9893020659219474
1.14:
plain took 6.254313945770264 seconds
tolist took 3.8215630054473877 seconds
6.432628321927041
3.8574256210122257
ahaldane commentedon May 1, 2018
I haven't checked, but perhaps #9978 is involved...
ahaldane commentedon May 1, 2018
Actually, we modified
arraytypes.c.srca couple times in the last few months, that was just the first one I remembered.My bet for this slowdown is now ##9856, since it modified the inner loop of
STRING_to_INT:for (i = 0; i < n; i++, ip+=skip, op+=oskip) { - temp = @from@_getitem(ip, aip); + PyObject *new; + PyObject *temp = PyArray_Scalar(ip, PyArray_DESCR(aip), (PyObject *)aip);eric-wieser commentedon May 1, 2018
Not sure how best to fix this. There's a horrible dance going on between doing the work in
scalartypes.c.srcand doing it inarraytypes.c.srcIdeally we'd do a conversion straight from array to array, rather than the round trip that currently happens of:
ndarray[np.str_]→np.str_→int→ndarray[np.int32]I'm pretty sure there are more steps there, but would need to add instrumentation to find them
ahaldane commentedon May 1, 2018
Agreed.
In that file we have a lot of functions like
INT_fromstr, which appear to avoid all that complication by usingPyOS_strtol, which should be fast. But that expects a null terminated string, and we have non-null terminated strings. C does not have a function to parse a non-null-terminated integer, so probably we'd have to make a null-terminated copy of the value first, or else roll our own strntol.That's ignoring any back-compat quirks we might have to support if we change things.
PERF: avoid array scalars in the string-like casts
PERF: avoid array scalars in the string-like casts
PERF: speed up the fixed-width string casts and unicode scalars