I just ran the brand new python -X importtime -c "import numpy". Turns out that numpy.testing takes rougly 29 % (23 ms) of the total import time (80 ms), most of which (62 %, 14 ms) is the unittest module (no nose in my environment). In view of #10856, this is only going to get worse -- pytest weighs in with about 35 ms on my machine.

So there's an opportunity for an easy win in import time by making the import of numpy.testing lazy. That is, if such a change doesn't cause huge amounts of downstream breakage.
Alternatively, the main culprits could be imported lazily, with a much smaller downstream impact (likely none).