Skip to content

ImportError with B7.7 (0.18.3) when installed as DMS operational build (Numpy API change) #5744

Closed

Description

@stscijgbot-jp

Issue JP-1922 was created on JIRA by Vincent Geers:

I believe JWST Pipeline build 7.7 (0.18.3) and earlier builds may experience a blocker ImportError issue when Pipeline was installed as a DMS Operational Build (with explicitly pinned versions of dependencies), following instructions from:

https://github.com/spacetelescope/jwst#installing-a-dms-operational-build

I suspect the underlying issue is caused by Numpy that appears to have had a change in their API since their 1.20.0 release of January 30th, 2021.

Here are the steps to reproduce the issue on macOS:

Install DMS Build 0.18.3 with:

 

conda create -n jwst0.18.3_dms_test --file https://ssb.stsci.edu/releases/jwstdp/0.18.3/conda_python_macos-stable-deps.txt
conda activate jwst0.18.3_dms_test
pip install --no-cache-dir -r https://ssb.stsci.edu/releases/jwstdp/0.18.3/reqs_macos-stable-deps.txt```
 

The --no-cache-dir is relevant, as the local pip cache may otherwise serve up an version of the packages built prior to January 30th that likely wouldn't be affected, thus obscuring the problem.

Try to import Spec2D pipeline:

 
```java
python
from jwst.pipeline import Spec2Pipeline

RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/vincent.geers/work/software/miniconda3/envs/jwst0.18.3_dms_test/lib/python3.8/site-packages/jwst/pipeline/__init__.py", line 2, in <module>
    from .calwebb_coron3 import Coron3Pipeline
  File "/Users/vincent.geers/work/software/miniconda3/envs/jwst0.18.3_dms_test/lib/python3.8/site-packages/jwst/pipeline/calwebb_coron3.py", line 12, in <module>
    from ..outlier_detection import outlier_detection_step
  File "/Users/vincent.geers/work/software/miniconda3/envs/jwst0.18.3_dms_test/lib/python3.8/site-packages/jwst/outlier_detection/__init__.py", line 1, in <module>
    from .outlier_detection_step import OutlierDetectionStep
  File "/Users/vincent.geers/work/software/miniconda3/envs/jwst0.18.3_dms_test/lib/python3.8/site-packages/jwst/outlier_detection/outlier_detection_step.py", line 7, in <module>
    from . import outlier_detection
  File "/Users/vincent.geers/work/software/miniconda3/envs/jwst0.18.3_dms_test/lib/python3.8/site-packages/jwst/outlier_detection/outlier_detection.py", line 8, in <module>
    from drizzle.cdrizzle import tblot
ImportError: numpy.core.multiarray failed to import```
-

The root cause appears to be this line:
```java
RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd

As far as I can tell, by following the DMS build procedure, certain dependencies for "jwst", such as "drizzle" and "stsci.stimage", are getting built against the very latest available Numpy (currently 1.20.1), while the DMS JWST build 0.18.3 will subsequently installed the pinned (and lower) version 1.19.4 of Numpy.

One can reproduce the error with the individual packages, e.g.:

>>> from stsci import stimage
RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/vincent.geers/work/software/miniconda3/envs/jwst0.18.3_dms_test/lib/python3.8/site-packages/stsci/stimage/__init__.py", line 32, in <module>
    from . import _stimage
ImportError: numpy.core.multiarray failed to import

As a test, if one then upgrade Numpy (to 1.20.0, or latest1.20.1) in this test conda environment:

pip install --no-cache-dir numpy==1.20.0

then the ImportError goes away.

I have seen the same issue occur with JWST DMS build 7.6  (0.17.1), and I suspect it'll happen for any DMS build that has Numpy pinned to some version that is <1.20.0.

The issue will likely not affect anyone who uses "pip install jwst", as that will likely automatically install the latest Numpy (currently 1.20.1).

The reason I and the MIRI IDT ran into this is that we maintain the "MIRICLE" software distribution (for example for the MIRISim simulator), which is based on a DMS build of "jwst", to ensure we get pinned versions of dependencies. We are this month in the process of updating MIRICLE to Pipeline Build 7.7.

A pragmatic solution may be to increase the minimum version of Numpy to 1.20.0 for JWST (current set to >= 1.16).

Activity

  1. jdavies-st commented on Feb 12, 2021

    @jdavies-st
    Collaborator

    Thanks for the report Vincent. This is a tough one, as it is the pyproject.toml in each of those dependencies (drizzle, tweakwcs, etc) that controls each package's build dependencies, which for those packages include numpy.

    The fact that we are pinning numpy in our requirements-sdp.txt file seems to not affect this PEP517 build system behavior. Strange.

  2. jdavies-st commented on Feb 12, 2021

    @jdavies-st
    Collaborator

    Btw, we saw this yesterday on Travis for a separate package that was not using the DMS reqs_macos-stable-deps.txt file and so was not pinning numpy==1.19.4. @bhilbert4 noticed this. We can see it here

    https://travis-ci.org/github/spacetelescope/jwst_reffiles/jobs/758536724#L405

    In that case, it's not clear why Travis was building/installing with disparate versions of numpy.

  3. jdavies-st commented on Feb 12, 2021

    @jdavies-st
    Collaborator

    The issue is reported here.

    pypa/pip#9542

  4. jdavies-st commented on Feb 15, 2021

    @jdavies-st
    Collaborator

    I think using oldest-supported-numpy meta package instead of numpy in the build dependency listed in pyproject.toml for drizzle and other packages that require numpy to build/compile is probably the best solution for this issue. It does fix the problem, though clearly this is a tooling issue in pip that is much broader than only build dependencies on numpy.

    Not sure the best way forward right now for people who want to use the

    https://ssb.stsci.edu/releases/jwstdp/0.18.3/reqs_macos-stable-deps.txt

    requirements file except to not to. Just pip install jwst==0.18.3. Or if you really must use the requirements file and you're using isolated envs, install numpy 1.19.4 first into the env so that it then populates the pip cache and can then be used as a build dependency.

    The next patch release of jwst delivered to DMS will have numpy 1.20.1, and this will be out in a few weeks. Of course the prospect for future problems next time numpy changes their API remains, so it would be good to use oldest-supported-numpy instead of numpy in the PEP518 build dependencies for drizzle, stsci.* packages that have C code that depends on numpy to compile.

  5. vcgeers commented on Feb 16, 2021

    @vcgeers

    Thank you for the follow up James.

    Indeed, I called out Numpy's API change for causing the ImportError, but as you say, the root cause is the pip tooling issue. Until that's fixed, this kind of issue could happen again with Numpy (and/or other packages perhaps).

    For our "MIRICLE" distribution, we'll first try out pinning Numpy to 1.20.1, but thank you for the idea of fetching 1.19.4 early to populate the pip cache.

  6. jdavies-st commented on Sep 17, 2021

    @jdavies-st
    Collaborator

    This has been resolved by #6325.

    While the DMS build spec jwstdp/0.18.3/reqs_macos-stable-deps.txt is still broken, 0.18.3 can now be installed with the fixed dependency packages and it works as expected. I.e. the following:

    pip install --no-cache-dir -r https://ssb.stsci.edu/releases/jwstdp/0.18.3/reqs_macos-stable-deps.txt
    pip install --upgrade drizzle spherical_geometry stsci.image stsci.stimage stsci.imagestats
    

    which allows usage with numpy<1.20. Anyway, we shouldn't have this problem in the future, so I'm closing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions