Use wheels on travis - #44

Any time frame on when this will make it into package-template?

@mwcraig - could you look into the failures to see why this is failing? It looks like the failures are genuine and related to Cython being missing?

@astrofrog -- yes, will get to tonight or tomorrow

@astrofrog -- the issue was that I was only installing cython when astropy needed it to build but the package template itself requires cython too. I added an environmental variable to travis.yml to control whether the package needs cython and set it to true by default.

Should work, but travis is being poky tonight, os haven't confirmed yet.

@astrofrog -- travis works, but one more heads up -- this PR does not use the -w option to build_sphinx, which means that warnings are not elevated to errors. If you want that changed, let me know.

@mwcraig If you could add -w to the build_sphinx here, that would be good. If you don't want to do that now, just make a separate issue.

In the above description you still have this listed as a TODO:

Confirm what the minimum coverage expectations are for affiliated packages.

I would suggest to remove this here ... it's a separate discussion and it would delay this important update to .travis.yml.

+1 to add -w back to the sphinx build

@cdeil @astrofrog -- now I remember why I took out the -w option: any idea how to fix this: https://travis-ci.org/astropy/package-template/jobs/24633863

Funny thing is that in a local build I don't get that warning (I do get a warning about the astropy intersphinx inventory not being found because astropy/astropy#1841 wasn't in 0.3.1).

fyi, you can ignore the error in build 48.4 on travis -- I had forgotten to create an index.html in the numpy-1.8.1 directory

The fail is the -w optoin

couple of updates:

• The build failure here is the favicon.ico problem reported in Doc build gives warning about astropy_logo.ico #49
• This "hub and spoke" system for wheelhouses fixes the problem that @cdeil described in Modify package template to use wheels #37

@Cadair @astrofrog -- let me know which of you would like to host my wheels...they would reside in subdirectories named for the version of numpy, e.g. numpy-1.5.1, numpy-1.6.2. Each directory contains wheels for numpy, scipy and astropy built against that version of numpy.

With this structure my wheels won't interfere with anyone who relies on the current wheels in wheels.astropy.org because each of my numpy directories has its own index.

Once I know where the wheels will be hosted I'll update the PR with the correct URL.

@mwcraig - could you provide the scripts that you are using to generate the wheels? (just to make sure we know how to generate them for future versions)

I don't think scipy should be installed by default. How about having a section that is an 'additional dependencies' section that contains just scipy, commented out by default, and a note that any other dependencies can be installed this way (and that if a wheel is not available, it will be fetched from pypi)?

@mwcraig - if you have scripts to produce all the wheels in an automated way, could you add them (the scripts, not the wheels) to https://github.com/astropy/astropy-wheels? I think we should then also consider using these for astropy itself if we ever run into scipy/numpy version mismatches.

@mwcraig - I'm uploading the wheels to my server so they will appear at wheels.astropy.org/numpy-1.x.x (upload is slow, give it a few hours)

@mwcraig - here are the wheels:

http://wheels.astropy.org/numpy-1.5.1/
http://wheels.astropy.org/numpy-1.6.2/
http://wheels.astropy.org/numpy-1.7.1/
http://wheels.astropy.org/numpy-1.8.1/

Can you update this PR to use that URL?

@astrofrog -- this will need at least one more commit before it is ready. The commit I just pushed adds variables for dependencies that depend on a specific version of numpy and for dependencies that don't need a specific version.

I want to make sure a travis build succeeds with both variables set to something; after that I'll comment out the variables.

I'll get a PR in for the build scripts in the next day or two.

@astrofrog -- the numpy-1.X.Y directories need an index.html in each. The tarball at http://physics.mnstate.edu/craig/wheelhouse/numpy-1.X.Y-index.tgz should have what you need.

Or you can use this python script to generate the index files: https://gist.github.com/mwcraig/61291289b892b5014472

@mwcraig - I'm going to add the index.php I use for the main wheel directory and which dynamically generates the list. It works for the main directory, so should be ok for these too?

It should; after that can you restart the Travis test?—
Sent from Mailbox for iPhone

On Sun, May 18, 2014 at 3:52 PM, Thomas Robitaille
[email protected] wrote:

@mwcraig - I'm going to add the index.php I use for the main wheel directory and which dynamically generates the list. It works for the main directory, so should be ok for these too?

Reply to this email directly or view it on GitHub:
#44 (comment)

I restarted Travis

@mwcraig - some builds are working, but some are having issues finding astropy. Any ideas?

@astrofrog -- A few were breaking because the latest numpy version was set to 1.8.0.

Stumped at the moment as to why the astropy downloads are succeeding sometimes and failing others, will take another look later tonight or first thing in the morning.

@astrofrog -- looks like it is timing out on some of the requests to wheels.astropy.org; I've turned up the verbosity of the pip output in the latest travis builds for this.

@mwcraig - it looks like the URLs should have a trailing slash?

Skipping link http://wheels.astropy.org//numpy-1.8.1 (from -f); unknown archive format: .1

@astrofrog -- the trailing slash maybe helped a bit (though it still checks the URL itself first to see if ti is a wheel, I think). The only "real" fail is this one: https://travis-ci.org/astropy/package-template/jobs/25514971

The documentation build is failing because of the favicon.ico issue, separate from wheels.

It passed! Now to be annoying, I have one more comment - at the moment, the longest builds are at the end, which is not 100% efficient because often four builds can happen at one time, so it actually makes sense in terms of efficiency to start the long ones and they can keep running while another worker goes through the shorter builds after. So if it's easy to do, move the three long builds from the bottom higher up.

I think there are also maybe slightly too many configurations. For instance, I don't think we need to test all the Python/Numpy combinations. I would probably get rid of 2.6 with numpy 1.6 and 1.7 and 3.3 with numpy 1.7. What do you think?

One last comment - can you add an egg_info build which includes no dependencies, like the core package? We weren't able to do this before astropy-helpers, but it should work now (and is one of the main reasons for astropy-helpers existing). So the test is that with no dependencies, python setup.py egg_info should work.

@astrofrog -- can you please take a look at this travis fail, which came up when I tried doing the coverage run with stable (instead of development) astropy: https://travis-ci.org/astropy/package-template/jobs/25837833

Still a couple cleanup changes coming before this is ready, should be soon.

@mwcraig - I think coverage was not working for affiliated packages in 0.3.x, so it makes sense that you need to use the latest dev version.

@astrofrog -- can you please restart https://travis-ci.org/astropy/package-template/jobs/25879875 ? Thanks.

Done

@astrfrog - I think this is ready to go.

Summary of changes in response to your comments:

So if it's easy to do, move the three long builds from the bottom higher up.

I don't think it is possible to move the explicit builds (i.e. in the build matrix because of include) before the implicit ones. I did move the development builds to the top of the include section, and once 0.4 is released we can switch the coverage build to stable.

can you add an egg_info build which includes no dependencies, like the core package?

Done.

I would probably get rid of 2.6 with numpy 1.6 and 1.7 and 3.3 with numpy 1.7. What do you think?

Fine with me; I have no strong feelings about it, so I took these out.

Still to be done, after 0.4 is released:

Will create separate issues for these so they don't get forgotten...

• build 0.4 wheels
• Update .travis.yml to use stable for coverage test
• Add 3.4 to list of pythons tested

Thanks! I'm going to be a pain now and ask if you can squash these commits to make sure we keep the package-template history reasonably simple? (since all affiliated packages have to merge in the changes)

Squashed!

@mwcraig - thanks! This will be super helpful for affiliated packages.

@mwcraig - would it be possible for you to build wheels for the latest matplotlib stable version? Then I can copy them to wheels.astropy.org - thanks!

Can do, though maybe not until Monday...—
Sent from Mailbox

On Sun, Jun 1, 2014 at 4:25 PM, Thomas Robitaille
[email protected] wrote:

@mwcraig - would it be possible for you to build wheels for the latest matplotlib stable version? Then I can copy them to wheels.astropy.org - thanks!

Reply to this email directly or view it on GitHub:
#44 (comment)

Any particular backends that need to be built?

I think even just having Agg should be enough. Thanks!

matplotlibs for python 2.6, 2.7, 3.2 and 3.3 are at http://physics.mnstate.edu/craig/matplotlibs.tar.gz

@mwcraig - thanks! The issue is that the matplotlib builds depend on the numpy versions, so we need to build it similarly to astropy and scipy. Would it be easy to do this? We should make it available as part of out set of numpy-dependent-wheels.

Oops, was trying to get away with being lazy. Not a problem to build.

On Mon, Jun 2, 2014 at 10:42 AM, Thomas Robitaille <[email protected]

wrote:

@mwcraig https://github.com/mwcraig - thanks! The issue is that the
matplotlib builds depend on the numpy versions, so we need to build it
similarly to astropy and scipy. Would it be easy to do this? We should make
it available as part of out set of numpy-dependent-wheels.

—
Reply to this email directly or view it on GitHub
#44 (comment)
.

As long as I'm building, I'll also do scipy==0.14 and python3.4

@mwcraig - thanks! Could you update the script in https://github.com/astropy/astropy-wheels/pull/1 once you've done it? Then I can also try and do it in future.

Wheels are now ready for download at: http://physics.mnstate.edu/craig/new_wheels.tar.gz

Not all versions of numpy build on all versions of python, but everything that I could build is built (i.e. matplotlib=1.3.1 for all numpy/python combos on which numpy could build, same for scipy-0.14.0, and astropy on py3.4)

@mwcraig Thanks for building scipy 0.14 wheels!

I'm using something in gammapy that only became available in that version:
https://travis-ci.org/gammapy/gammapy/jobs/26755796#L882

Thanks @mwcraig!