Modify package template to use wheels - #37

@Cadair Thanks!

I tried this for gammapy (gammapy/gammapy#83) and it fails for scipy like so (only for the astropy dev version !?):

E   ImportError: numpy.core.multiarray failed to import
RuntimeError: module compiled against API version 9 but this version of numpy is 7

https://travis-ci.org/gammapy/gammapy/builds/19280515
https://travis-ci.org/gammapy/gammapy/jobs/19280524#L562

Should we extend the package-template with some function using the optional scipy dependency to make this issue show up here instead of in some affiliated packages only?

Are the builds that are failing ones that use old versions of NumPy?
On 20 Feb 2014 19:30, "Christoph Deil" [email protected] wrote:

@Cadair https://github.com/Cadair Thanks!

I tried this for gammapy and it fails for scipy like so (only for the
astropy dev version !?):

E ImportError: numpy.core.multiarray failed to import
RuntimeError: module compiled against API version 9 but this version of numpy is 7

https://travis-ci.org/gammapy/gammapy/builds/19280515
https://travis-ci.org/gammapy/gammapy/jobs/19280524#L562

Should we extend the package-template with some function using the
optional scipy dependency to make this issue show up here instead of in
some affiliated packages only?

Reply to this email directly or view it on GitHubhttps://github.com//pull/37#issuecomment-35659546
.

Are the builds that are failing ones that use old versions of NumPy?

No. It's the ones that are using the development version of astropy:
https://travis-ci.org/gammapy/gammapy/builds/19280515

OK, I am a little confused here now, as it looks like there is a numpy version conflict somewhere triggered by installing Astropy from the git egg. I have no idea what that git egg command does. @astrofrog Your up.

@Cadair @cdeil - I haven't looked through the logs in detail, but I think I can guess what is going on. If you build a scipy wheel using numpy 1.7.1, it won't work when you then install a numpy 1.5 wheel because the ABI is different. The solution is to basically build every version of modules that depend on numpy (scipy and astropy) once for every major version of numpy. That means we need e.g.

astropy-0.3-numpy-1.5
astropy-0.3-numpy-1.6
astropy-0.3-numpy-1.7
scipy-0.11-numpy-1.5
scipy-0.11-numpy-1.6
...

Does this make sense?

@Cadair - you could consider making one travis job per numpy/python version combination to handle this. That is, if you cover 3 numpy versions and 4 python versions, you'd have 12 jobs. Each job would then produce all the wheels for each python/numpy combination.

@astrofrog Yes, that is a problem and something that isn't really addressed by pip/wheels. I am not sure I have time time or the know-how to come up with a sane way of dealing with this.

I think that if you want to deal with more complex numpy/scipy pairs for now you might have to build from source or build wheels.

I think I can help this move forward, but have a few questions:

• What is the goal for test coverage for affiliated packages? i.e. what versions of numpy, astropy, python, etc. does it need to successfully build against?

Looking at the main astropy repository it looks like tests there cover (table cells are the numpy version number unless otherwise indicated):

It looks like the coverage for an affiliated package in this pull request would be:

It isn't clear to me based on this what the target should be - numpy version from 1.5.1 to 1.8.0? against both astropy master and the latest stable? Or a subset? Should we be testing against combinations like astropy-0.3.x/numpy-1.8.0 that are not currently covered by the astropy/astropy tests?

• For handling the fact that astropy and scipy wheels need to be built against the same version of numpy as the numpy wheels should the wheelhouse be organized more like this:
  • folder numpy-1.5.1, which contains wheels for numpy, astropy , scipy, all based on numpy-1.5.1
  • folder numpy-1.6.2, which contains wheels for numpy, astropy, scipy
  • etc.

then the setup of the travis file needs to change some, maybe like this:

# omitted stuff
env:
    global:
        - PIP_WHEEL_COMMAND="pip install --use-wheel --use-mirrors"
# more omitted stuff....numpy version is set in matrix env
install:
    - WHEEL_HOUSES="--find-links http://wheels.astropy.org/numpy-$NUMPY_VERSION --find-links http://wheels.astropy.org" # same for additional wheelhouses
    - $PIP_WHEEL_COMMAND $WHEEL_HOUSES "numpy==$NUMPY_VERSION"
    - $PIP_WHEEL_COMMAND $WHEEL_HOUSES "astropy"

• If that kind of approach makes sense the current wheelhouses would need to be rearranged a bit--but aren't those going to cause problems at some point anyway? The only reason astropy/astropy hasn't hit this same problem is that it only uses scipy-0.12.1 with numpy-1.7.1 and scipy-0.13.1 with numpy-1.8.0. It looks like there is no way to tell right now what numpy version a particular scipy wheel was built with.
• Is there an easier way to keep the python/numpy/astropy version requirements in sync between affiliated packages and main astropy?

I can build any wheels that are need and pass them on to either of the existing wheelhouses (I'm doing the builds on a local VM at the moment, but it would be easy enough to use travis, I suppose), and can take a shot at modifying .travis.yaml along the lines outlined above.

I'm going to give something like this a try because I want to get a reasonably full set of travis tests running for ccdproc before writing a lot more code for it. So I could try something like this and report back, I suppose...

I haven't attempted a Travis build in months because I've been waiting for this. When will this be done?

Closed by #44