Repository navigation
Description
Issue JP-1922 was created on JIRA by Vincent Geers:
I believe JWST Pipeline build 7.7 (0.18.3) and earlier builds may experience a blocker ImportError issue when Pipeline was installed as a DMS Operational Build (with explicitly pinned versions of dependencies), following instructions from:
I suspect the underlying issue is caused by Numpy that appears to have had a change in their API since their 1.20.0 release of January 30th, 2021.
Here are the steps to reproduce the issue on macOS:
Install DMS Build 0.18.3 with:
conda create -n jwst0.18.3_dms_test --file https://ssb.stsci.edu/releases/jwstdp/0.18.3/conda_python_macos-stable-deps.txt
conda activate jwst0.18.3_dms_test
pip install --no-cache-dir -r https://ssb.stsci.edu/releases/jwstdp/0.18.3/reqs_macos-stable-deps.txt```
The --no-cache-dir is relevant, as the local pip cache may otherwise serve up an version of the packages built prior to January 30th that likely wouldn't be affected, thus obscuring the problem.
Try to import Spec2D pipeline:
```java
python
from jwst.pipeline import Spec2Pipeline
RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/vincent.geers/work/software/miniconda3/envs/jwst0.18.3_dms_test/lib/python3.8/site-packages/jwst/pipeline/__init__.py", line 2, in <module>
from .calwebb_coron3 import Coron3Pipeline
File "/Users/vincent.geers/work/software/miniconda3/envs/jwst0.18.3_dms_test/lib/python3.8/site-packages/jwst/pipeline/calwebb_coron3.py", line 12, in <module>
from ..outlier_detection import outlier_detection_step
File "/Users/vincent.geers/work/software/miniconda3/envs/jwst0.18.3_dms_test/lib/python3.8/site-packages/jwst/outlier_detection/__init__.py", line 1, in <module>
from .outlier_detection_step import OutlierDetectionStep
File "/Users/vincent.geers/work/software/miniconda3/envs/jwst0.18.3_dms_test/lib/python3.8/site-packages/jwst/outlier_detection/outlier_detection_step.py", line 7, in <module>
from . import outlier_detection
File "/Users/vincent.geers/work/software/miniconda3/envs/jwst0.18.3_dms_test/lib/python3.8/site-packages/jwst/outlier_detection/outlier_detection.py", line 8, in <module>
from drizzle.cdrizzle import tblot
ImportError: numpy.core.multiarray failed to import```
-
The root cause appears to be this line:
```java
RuntimeError: module compiled against API version 0xe but this version of numpy is 0xdAs far as I can tell, by following the DMS build procedure, certain dependencies for "jwst", such as "drizzle" and "stsci.stimage", are getting built against the very latest available Numpy (currently 1.20.1), while the DMS JWST build 0.18.3 will subsequently installed the pinned (and lower) version 1.19.4 of Numpy.
One can reproduce the error with the individual packages, e.g.:
>>> from stsci import stimage
RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/vincent.geers/work/software/miniconda3/envs/jwst0.18.3_dms_test/lib/python3.8/site-packages/stsci/stimage/__init__.py", line 32, in <module>
from . import _stimage
ImportError: numpy.core.multiarray failed to importAs a test, if one then upgrade Numpy (to 1.20.0, or latest1.20.1) in this test conda environment:
pip install --no-cache-dir numpy==1.20.0then the ImportError goes away.
I have seen the same issue occur with JWST DMS build 7.6 (0.17.1), and I suspect it'll happen for any DMS build that has Numpy pinned to some version that is <1.20.0.
The issue will likely not affect anyone who uses "pip install jwst", as that will likely automatically install the latest Numpy (currently 1.20.1).
The reason I and the MIRI IDT ran into this is that we maintain the "MIRICLE" software distribution (for example for the MIRISim simulator), which is based on a DMS build of "jwst", to ensure we get pinned versions of dependencies. We are this month in the process of updating MIRICLE to Pipeline Build 7.7.
A pragmatic solution may be to increase the minimum version of Numpy to 1.20.0 for JWST (current set to >= 1.16).
Activity
jdavies-st commentedon Feb 12, 2021
Thanks for the report Vincent. This is a tough one, as it is the
pyproject.tomlin each of those dependencies (drizzle,tweakwcs, etc) that controls each package's build dependencies, which for those packages includenumpy.The fact that we are pinning
numpyin ourrequirements-sdp.txtfile seems to not affect this PEP517 build system behavior. Strange.jdavies-st commentedon Feb 12, 2021
Btw, we saw this yesterday on Travis for a separate package that was not using the DMS reqs_macos-stable-deps.txt file and so was not pinning
numpy==1.19.4. @bhilbert4 noticed this. We can see it herehttps://travis-ci.org/github/spacetelescope/jwst_reffiles/jobs/758536724#L405
In that case, it's not clear why Travis was building/installing with disparate versions of
numpy.jdavies-st commentedon Feb 12, 2021
The issue is reported here.
pypa/pip#9542
jdavies-st commentedon Feb 15, 2021
I think using
oldest-supported-numpymeta package instead ofnumpyin the build dependency listed in pyproject.toml fordrizzleand other packages that requirenumpyto build/compile is probably the best solution for this issue. It does fix the problem, though clearly this is a tooling issue inpipthat is much broader than only build dependencies onnumpy.Not sure the best way forward right now for people who want to use the
https://ssb.stsci.edu/releases/jwstdp/0.18.3/reqs_macos-stable-deps.txt
requirements file except to not to. Just
pip install jwst==0.18.3. Or if you really must use the requirements file and you're using isolated envs, installnumpy 1.19.4first into the env so that it then populates the pip cache and can then be used as a build dependency.The next patch release of
jwstdelivered to DMS will havenumpy 1.20.1, and this will be out in a few weeks. Of course the prospect for future problems next timenumpychanges their API remains, so it would be good to useoldest-supported-numpyinstead ofnumpyin the PEP518 build dependencies fordrizzle,stsci.*packages that have C code that depends onnumpyto compile.vcgeers commentedon Feb 16, 2021
Thank you for the follow up James.
Indeed, I called out Numpy's API change for causing the ImportError, but as you say, the root cause is the pip tooling issue. Until that's fixed, this kind of issue could happen again with Numpy (and/or other packages perhaps).
For our "MIRICLE" distribution, we'll first try out pinning Numpy to 1.20.1, but thank you for the idea of fetching 1.19.4 early to populate the pip cache.
jdavies-st commentedon May 26, 2021
To close this issue, it would be good modify
pyproject.tomlin the following repos to useoldest-supported-numpyinstead ofnumpyand then release:All of the above have the auto-delivery to PyPI on release.
jdavies-st commentedon Sep 17, 2021
This has been resolved by #6325.
While the DMS build spec
jwstdp/0.18.3/reqs_macos-stable-deps.txtis still broken, 0.18.3 can now be installed with the fixed dependency packages and it works as expected. I.e. the following:which allows usage with
numpy<1.20. Anyway, we shouldn't have this problem in the future, so I'm closing.