You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
numpy.testing accounts for almost 30 % import time #11457
I just ran the brand new python -X importtime -c "import numpy". Turns out that numpy.testing takes rougly 29 % (23 ms) of the total import time (80 ms), most of which (62 %, 14 ms) is the unittest module (no nose in my environment). In view of #10856, this is only going to get worse -- pytest weighs in with about 35 ms on my machine.
So there's an opportunity for an easy win in import time by making the import of numpy.testing lazy. That is, if such a change doesn't cause huge amounts of downstream breakage.
Alternatively, the main culprits could be imported lazily, with a much smaller downstream impact (likely none).
Interesting. I quickly looked through the main __init__.py, and that really refers to testing only in a few lines [1]. The important one, repeated elsewhere, is the import of testing._private.pytesttester.PytestTester. Here, PytestTester itself does not do any imports until it is used, but the problem is that along its path, testing.__init__ imports a lot of stuff. If I comment out the from .testing import Tester line in [1], and also remove all but the PyTestTester lines from testing.__init__ [2], the import time does indeed reduce by ~25%.
There would seem to be two options:
As you say, lazily initializing in testing.__init__.py (in python3.7, the module-level __getattr__ could be used...; are there other tricks?);
Move PytestTester to a different path.
The latter would seem a much lighter change (and one could obviously leave an import in its current location), but let me cc @charris (who can perhaps ping others who understand this part better).
[1] from numpy.__init__.py:
# We don't actually use this ourselves anymore, but I'm not 100% sure that
# no-one else in the world is using it (though I hope not)
from .testing import Tester
# Pytest testing
from numpy.testing._private.pytesttester import PytestTester
test = PytestTester(__name__)
del PytestTester
[2] almost everything from numpy.testing.__init__.py:
from unittest import TestCase
from ._private.utils import *
from ._private import decorators as dec
from ._private.nosetester import (
run_module_suite, NoseTester as Tester
)
__all__ = _private.utils.__all__ + ['TestCase', 'run_module_suite']
from ._private.pytesttester import PytestTester
test = PytestTester(__name__)
del PytestTester
IIRC we wanted to deprecate the automatic import of testing before, but didn't because there was no deprecation mechanism possible. Nathaniel probably put that comment there...
There are two issues: (1) there's lots of code that assumes it can do a bare import numpy and then refer to numpy.testing, so we need a way to automatically load it on first reference, (2) we have to somehow handle the re-exports like np.Tester and np.test. We have to solve all of them if we want to reduce import time.
For the lazy import, the portable way is to use https://github.com/njsmith/metamodule/; this lets you basically define a __getattr__ on module objects. There's some old PR to add this to numpy that I've been feeling guilty about not getting back to for like 4 years at this point... though that PR actually had a different motivation, which was to deprecate np.int and friends. (If anyone wants to revive that PR, I believe the main blocker did get fixed in metamodule's v1.2 release.)
metamodule is very clever about trying to avoid adding extra overhead to regular attribute lookup (e.g. np.array), but for obscure reasons it does add a little bit, at least on some versions of python. (It turns out there's a fast path inside the CPython interpreter that gets disabled for objects that have a __getattr__ fallback, even if you never actually call it?!) Because of this, using 3.7's new module-level __getattr__ hack might be faster, so we might want to use that where possible. People are pretty sensitive about the speed of attribute lookup on the numpy module object, because unless you're doing from numpy import * then most code does a lot of attribute lookups.
I think np.test is probably trivial, it can become a function or something. For Tester – remember that to support lazy loading of np.testing, we need to hook things up so that we can run arbitrary code when an attribute is accessed on the np object. I suspect the simplest thing to do is to re-use this for np.Tester. (And we can deprecate it at the same time!)
From that thread, tab completion is also important, it needs to stay working.
+1 for @mhvk's suggestion of pytest on a different import path, nicer than messing with lazy imports.
metamodule is very clever about trying to avoid adding extra overhead to regular attribute lookup (e.g. np.array), but for obscure reasons it does add a little bit, at least on some versions of python.
On my machine/configuration, numpy.add_newdocs is the most expensive import: 82% of the time. In total, 180ms.
Of which almost 100 % is from numpy.lib, of which north of 90 % is from numpy.core. At that point it spreads out more, but numpy.testing makes up about 30 % of the time spent importing numpy.core.
From that thread, tab completion is also important, it needs to stay working.
Tab completion is fine – all the mechanisms that let you override __getattr__ also let you override __dir__, so you can make sure dir(np) reports all the attributes you want to be tab-completable.
interesting. how much is "a little bit" [of overhead added by metamodule]?
On Python 3, it's ~100 ns per attribute lookup, and on Python 2 it's only ~10 ns per attribute lookup.
Details
Python 3.6:
In [1]: # Attribute lookup alone
...: %timeit np.array
58.3 ns ± 8.48 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)
In [2]: # Doing something simple
...: %timeit np.array([])
830 ns ± 41.3 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
In [3]: import metamodule
In [4]: metamodule.install("numpy")
In [5]: # Attribute lookup alone
...: %timeit np.array
165 ns ± 11.3 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
In [6]: # Doing something simple
...: %timeit np.array([])
970 ns ± 98.6 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
Python 2.7:
In [1]: %timeit np.array
10000000 loops, best of 3: 38.7 ns per loop
In [2]: %timeit np.array([])
The slowest run took 18.57 times longer than the fastest. This could mean that an intermediate result is being cached.
1000000 loops, best of 3: 590 ns per loop
In [3]: import metamodule
In [4]: metamodule.install("numpy")
In [5]: import numpy as np # required on py2 b/c we're hackily installing this from outside numpy/__init__.py
In [6]: %timeit np.array
10000000 loops, best of 3: 57.1 ns per loop
In [7]: %timeit np.array([])
The slowest run took 16.42 times longer than the fastest. This could mean that an intermediate result is being cached.
1000000 loops, best of 3: 610 ns per loop
Unfortunately when I was originally testing this it was on Python 2 so I didn't notice the issue. The underlying issue is probably not too hard to fix in CPython, but that wouldn't help until 3.8 comes out. I haven't tried timing the new module-level __getattr__ in 3.7, but someone should :-).
Wouldn't one use the __getattr__ in timing, though? In that case it wouldn't slow down anything in the main namespace (of course, that would not help for deprecating int).
@mhvk I assume you meant testing, not timing? And the answer is ... well, maybe :-). The simplest way to do this is to make it so that the testing subpackage doesn't get loaded at all until the first time someone accesses the numpy.testing attribute. For this, we'd need to add __getattr__ to the main numpy module. I guess it would also be possible to go through numpy.testing and make it so that loading the package itself is cheap, but then every attribute numpy.testing.<whatever> is auto-loaded on first access. But setting this up would be significantly more complex and error-prone though, just because there are a lot more attributes to sort out.
Yes, 'testing I meant.... Its __init__ is fairly simple, with an __all__ that just contains _private.utils and two other items, so writing a __getattr__ for that should be nearly trivial.
But perhaps we can also consider the even less intrusive option of moving PytestTester elsewhere? (maybe just move the _private directory one level up?)
I would vote for creating a clean path for importing only the minimum of NumPy needed to use the multiarray container, without any of testing, polynomial, ma, financial, fft, ... Users can always import numpy.testing to get the desired submodule. Yes I know this is a breaking change, but I think the wider community would welcome it, since it would siginficantly simplify packaging numpy as part of a pyinstall/nuitka package.
Yeah that's because the scipy namespace is empty except for numpy functions. You never do import scipy, you always do from scipy import some_submodule. scipy.stats import time alone weighs in at 3x more than all of numpy.
The point is, you can choose what you want (up to inter-dependencies), and then pay the price of the import. Some of the packages are actually not that heavy. Numbers from another (more crappy) machine (Numpy is around 160 ms):
package
time [ms]
Numpy
Comment
scipy.sparse
194
82 %
scipy.linalg
205
78 %
scipy.special
218
73 %
imports scipy.linalg
scipy.optimize
351
46 %
imports scipy.special, scipy.linalg and scipy.sparse
scipy.stats
466
34 %
imports kind of everything
... and so on. stats and signal are heavy, others less so.
Lazy loading has two main downsides. The intrinsic one is that it can be surprising: it can hide ImportErrors or make them appear at strange places, performance is less predictable, etc. Also, since it's just moving around there cost, it doesn't help for modules that you do eventually end up loading anyway. For numpy.testing in particular I don't think these are an issue.
The other one is that implementing it is pretty complicated, so most implementations have quirks and subtle tradeoffs. You have to look carefully at things like speed impact, and I don't trust that lazy-import package without reading its source. (Also it's GPL, which isn't something numpy can use by policy.) Or there are also methods that involve hooking the import system itself...
It's doable, some projects have done it for a long time, it's probably the right thing for numpy.testing, but it's just non-trivial enough that no one has ever put all the pieces together and gotten something merged.
Looking at the import times more carefully, 125ms / 195ms (65%) on my machine is spent importing builtin modules - so it's perhaps unfair to blame numpy for a large import time, especially if your code goes on to import the same builtins anyway.
I suggest we close this and if desired open a new issue reflecting the discussion here, alternatively we could retitle this one appropriately since the title is misleading
I suggest we close this and if desired open a new issue reflecting the discussion here, alternatively we could retitle this one appropriately since the title is misleading
Not sure opening a new issue here is helpful, since I don't think there's anything left to do. Except perhaps a decision on lazy loading of numpy/testing yes/no, which we can do in this issue. Once we have that, we can close this.
If you have a more descriptive title, please go ahead and edit it, that's always useful.
As stated, I think blaming numpy.testing alone is probably misleading.
There are quite a few other modules that take time.
One of those "innocent" looking ones is:
platform. The only location where it is truely necessary is to create the variable IS_PYPY. That said, it seems to import all of Threading, which accounts for a large chunk of the import time. If detecting PyPy was hard and inconvenient, it might be justified, but in fact, it is as easy as "PyPy" in sys.version.
Threading is also imported for a Lock in the random module, which is a Cython module. I made a proof of concept where I imported fastrlock (ok, I know that we don't need reentrant locks, but I wanted something that look API compatible easily). The random modules are already cython, and thus this is a "small micro optimization" that doesn't add any cost. We can use the same locking primitives that fastrlock uses to speed up the whole module.
As stated, by other UnitTest does take quite a bit of time. The warning in the comments indicates that it is likely only used by packages, that can find the relevant documentation to test numpy as needed.
shutils can be lazy imported in the two locations where it is used.
While some of these might be micro optimizations, I think many might be well justified to help improve numpy's import time in the near term especially seeing as these optimizations hit code that is considered soft deprecated or convenient for compatibility reasons that no longer exist (i.e. python 2 has been dropped).
According to those, on some computer somewhere, it might take about 900 ms to import numpy up from 700 ms in recent versions. While the benchmarks running on my laptop are not that slow, it also isn't the cheapest laptop around.
Here is a PR made to my own branch showing the changes in case anybody wanted to glance at them: #14083
And an image of the improvements as I made the changes.
I get that in many application the caller will likely import Threading, or pathlib or platform themselves, and thus their application will not see the overall benefit or removing all 3 imports, but they might see the slight improvement in removing one of the many dependencies that aren't critical, or, at the very least, they might have a nice way of lazy importing them themselves.
Activity
mhvk commentedon Jun 30, 2018
Interesting. I quickly looked through the main
__init__.py, and that really refers totestingonly in a few lines [1]. The important one, repeated elsewhere, is the import oftesting._private.pytesttester.PytestTester. Here,PytestTesteritself does not do any imports until it is used, but the problem is that along its path,testing.__init__imports a lot of stuff. If I comment out thefrom .testing import Testerline in [1], and also remove all but thePyTestTesterlines fromtesting.__init__[2], the import time does indeed reduce by ~25%.There would seem to be two options:
testing.__init__.py(in python3.7, the module-level__getattr__could be used...; are there other tricks?);PytestTesterto a different path.The latter would seem a much lighter change (and one could obviously leave an import in its current location), but let me cc @charris (who can perhaps ping others who understand this part better).
[1] from
numpy.__init__.py:[2] almost everything from
numpy.testing.__init__.py:seberg commentedon Jun 30, 2018
IIRC we wanted to deprecate the automatic import of
testingbefore, but didn't because there was no deprecation mechanism possible. Nathaniel probably put that comment there...eric-wieser commentedon Jun 30, 2018
Can we fix this by defining
Testeras:That way, only users and subclassers will incur the import
njsmith commentedon Jun 30, 2018
There are two issues: (1) there's lots of code that assumes it can do a bare
import numpyand then refer tonumpy.testing, so we need a way to automatically load it on first reference, (2) we have to somehow handle the re-exports likenp.Testerandnp.test. We have to solve all of them if we want to reduce import time.For the lazy import, the portable way is to use https://github.com/njsmith/metamodule/; this lets you basically define a
__getattr__on module objects. There's some old PR to add this to numpy that I've been feeling guilty about not getting back to for like 4 years at this point... though that PR actually had a different motivation, which was to deprecatenp.intand friends. (If anyone wants to revive that PR, I believe the main blocker did get fixed in metamodule's v1.2 release.)metamodule is very clever about trying to avoid adding extra overhead to regular attribute lookup (e.g.
np.array), but for obscure reasons it does add a little bit, at least on some versions of python. (It turns out there's a fast path inside the CPython interpreter that gets disabled for objects that have a__getattr__fallback, even if you never actually call it?!) Because of this, using 3.7's new module-level__getattr__hack might be faster, so we might want to use that where possible. People are pretty sensitive about the speed of attribute lookup on thenumpymodule object, because unless you're doingfrom numpy import *then most code does a lot of attribute lookups.I think
np.testis probably trivial, it can become a function or something. ForTester– remember that to support lazy loading ofnp.testing, we need to hook things up so that we can run arbitrary code when an attribute is accessed on thenpobject. I suspect the simplest thing to do is to re-use this fornp.Tester. (And we can deprecate it at the same time!)rgommers commentedon Jul 1, 2018
pyteston a different import path, nicer than messing with lazy imports.interesting. how much is "a little bit"?
kohr-h commentedon Jul 1, 2018
Here's @njsmith's PR for reference: #6103
horta commentedon Jul 1, 2018
On my machine/configuration,
numpy.add_newdocsis the most expensive import: 82% of the time. In total, 180ms.python3.7 -X importtime -c 'import numpy'Numpy version: 1.14.5
Python version: 3.7.0
Conda environment, on macos 10.13.5
kohr-h commentedon Jul 1, 2018
Of which almost 100 % is from
numpy.lib, of which north of 90 % is fromnumpy.core. At that point it spreads out more, butnumpy.testingmakes up about 30 % of the time spent importingnumpy.core.horta commentedon Jul 1, 2018
Graphically: https://transfer.sh/tUOuP/time.pdf
nschloe commentedon Jul 1, 2018
Another graph using tuna. Reproduce with
njsmith commentedon Jul 1, 2018
Tab completion is fine – all the mechanisms that let you override
__getattr__also let you override__dir__, so you can make suredir(np)reports all the attributes you want to be tab-completable.On Python 3, it's ~100 ns per attribute lookup, and on Python 2 it's only ~10 ns per attribute lookup.
Details
Python 3.6:
Python 2.7:
Unfortunately when I was originally testing this it was on Python 2 so I didn't notice the issue. The underlying issue is probably not too hard to fix in CPython, but that wouldn't help until 3.8 comes out. I haven't tried timing the new module-level
__getattr__in 3.7, but someone should :-).mhvk commentedon Jul 1, 2018
Wouldn't one use the
__getattr__intiming, though? In that case it wouldn't slow down anything in the main namespace (of course, that would not help for deprecatingint).njsmith commentedon Jul 1, 2018
@mhvk I assume you meant
testing, nottiming? And the answer is ... well, maybe :-). The simplest way to do this is to make it so that thetestingsubpackage doesn't get loaded at all until the first time someone accesses thenumpy.testingattribute. For this, we'd need to add__getattr__to the main numpy module. I guess it would also be possible to go throughnumpy.testingand make it so that loading the package itself is cheap, but then every attributenumpy.testing.<whatever>is auto-loaded on first access. But setting this up would be significantly more complex and error-prone though, just because there are a lot more attributes to sort out.mhvk commentedon Jul 1, 2018
Yes,
'testingI meant.... Its__init__is fairly simple, with an__all__that just contains_private.utilsand two other items, so writing a__getattr__for that should be nearly trivial.But perhaps we can also consider the even less intrusive option of moving
PytestTesterelsewhere? (maybe just move the_privatedirectory one level up?)mattip commentedon Jul 1, 2018
I would vote for creating a clean path for importing only the minimum of NumPy needed to use the multiarray container, without any of testing, polynomial, ma, financial, fft, ... Users can always
import numpy.testingto get the desired submodule. Yes I know this is a breaking change, but I think the wider community would welcome it, since it would siginficantly simplify packaging numpy as part of a pyinstall/nuitka package.13 remaining items
eric-wieser commentedon Jul 3, 2018
@njsmith: Not beyond making the tree easier to analyze - having add_newdocs look like 90% of numpy was just confusing.
I think a more balanced tree is less indicative of (allowable) circular imports though
kohr-h commentedon Jul 3, 2018
The point is, you can choose what you want (up to inter-dependencies), and then pay the price of the import. Some of the packages are actually not that heavy. Numbers from another (more crappy) machine (Numpy is around 160 ms):
scipy.sparsescipy.linalgscipy.specialscipy.linalgscipy.optimizescipy.special,scipy.linalgandscipy.sparsescipy.stats... and so on.
statsandsignalare heavy, others less so.horta commentedon Jul 3, 2018
Does anybody now what are the drawbacks of lazy loading modules? https://pypi.org/project/lazy-import/
horta commentedon Jul 3, 2018
Maybe lazy loading for py37 only is a good idea: https://snarky.ca/lazy-importing-in-python-3-7/
njsmith commentedon Jul 3, 2018
Lazy loading has two main downsides. The intrinsic one is that it can be surprising: it can hide
ImportErrors or make them appear at strange places, performance is less predictable, etc. Also, since it's just moving around there cost, it doesn't help for modules that you do eventually end up loading anyway. Fornumpy.testingin particular I don't think these are an issue.The other one is that implementing it is pretty complicated, so most implementations have quirks and subtle tradeoffs. You have to look carefully at things like speed impact, and I don't trust that
lazy-importpackage without reading its source. (Also it's GPL, which isn't something numpy can use by policy.) Or there are also methods that involve hooking the import system itself...It's doable, some projects have done it for a long time, it's probably the right thing for
numpy.testing, but it's just non-trivial enough that no one has ever put all the pieces together and gotten something merged.eric-wieser commentedon Jul 6, 2018
Looking at the import times more carefully, 125ms / 195ms (65%) on my machine is spent importing builtin modules - so it's perhaps unfair to blame numpy for a large import time, especially if your code goes on to import the same builtins anyway.
mattip commentedon Jul 18, 2018
I suggest we close this and if desired open a new issue reflecting the discussion here, alternatively we could retitle this one appropriately since the title is misleading
rgommers commentedon Jul 21, 2018
Not sure opening a new issue here is helpful, since I don't think there's anything left to do. Except perhaps a decision on lazy loading of
numpy/testingyes/no, which we can do in this issue. Once we have that, we can close this.If you have a more descriptive title, please go ahead and edit it, that's always useful.
mhvk commentedon Mar 22, 2019
Rereading the discussion I agree with @rgommers that, essentially, there is nothing more to do, so I'm closing this issue.
hmaarrfk commentedon Jul 23, 2019
As stated, I think blaming numpy.testing alone is probably misleading.
There are quite a few other modules that take time.
One of those "innocent" looking ones is:
platform. The only location where it is truely necessary is to create the variableIS_PYPY. That said, it seems to import all ofThreading, which accounts for a large chunk of the import time. If detectingPyPywas hard and inconvenient, it might be justified, but in fact, it is as easy as"PyPy" in sys.version.Threadingis also imported for aLockin the random module, which is a Cython module. I made a proof of concept where I importedfastrlock(ok, I know that we don't need reentrant locks, but I wanted something that look API compatible easily). The random modules are already cython, and thus this is a "small micro optimization" that doesn't add any cost. We can use the samelockingprimitives that fastrlock uses to speed up the whole module.secretsis quite slow to import. Since we only need it for a few random bits, we can import what we need ourselves. https://github.com/numpy/numpy/pull/14083/files#diff-89944aec176617da993c6de4d9529348R251UnitTestdoes take quite a bit of time. The warning in the comments indicates that it is likely only used by packages, that can find the relevant documentation to test numpy as needed.pickleis quite slow too.pickleis a strange one, since many other libraries will import it, but from what I found by removing it, almost everywhere it was used except fornumpy.core._methods, it is associated with a warning. Not sure if the omission of the warning there was an honest mistake or an API decision. But avoiding the import of pickle can speed things up for those that don't need it.https://github.com/numpy/numpy/blob/master/numpy/core/_methods.py#L241
textwrapis not a trivial import. It is only used in 2 location where it makes the code "indented to according to a certain style". It doesn't seem worthwhile to use it to sanitize static strings. https://github.com/numpy/numpy/blob/master/numpy/core/overrides.py#L166 https://github.com/numpy/numpy/blob/master/numpy/ma/core.py#L2448Decimaltakes time, but there isn't much you can do other than ruining code style.pathlibis also not a trivial import. In fact the one location where it is imported directly has a comment stating that it should not be the prefered method. https://github.com/numpy/numpy/blob/master/numpy/compat/py3k.py#L105shutilscan be lazy imported in the two locations where it is used.While some of these might be
micro optimizations, I think many might be well justified to help improve numpy's import time in the near term especially seeing as these optimizations hit code that is consideredsoft deprecatedorconvenient for compatibility reasons that no longer exist (i.e. python 2 has been dropped).I'm going to refer to numpy's own benchmarks regarding the amount of time it takes for numpy to import:
https://pv.github.io/numpy-bench/#bench_import.Import.time_numpy
According to those, on some computer somewhere, it might take about 900 ms to import numpy up from 700 ms in recent versions. While the benchmarks running on my laptop are not that slow, it also isn't the cheapest laptop around.
Here is a PR made to my own branch showing the changes in case anybody wanted to glance at them: #14083
And an image of the improvements as I made the changes.

I'm happy to cleanup the changes as required.
Other relevant discussion here:
https://news.ycombinator.com/item?id=16978932 linking to a post where python core devs are worried about import time as well.
I get that in many application the caller will likely import
Threading, orpathliborplatformthemselves, and thus their application will not see the overall benefit or removing all 3 imports, but they might see the slight improvement in removing one of the many dependencies that aren't critical, or, at the very least, they might have a nice way of lazy importing them themselves.