Remove scipy tests from Travis

The requirement to have SciPy available for the tests in Travis has, in my opinion, made the Travis tests almost unusable. For one, it has inflated the build time for them immensely--I used to get results for pull requests pretty quickly.

Part of that may be due to growth in use of Travis by other projects and thus increased demand for its build servers. But the fact that we have to build all of SciPy as part of every single Astropy build has to be a major source of the slowness. Plus it has made the logs difficult to get to--it is necessary to download the log--it is not viewable directly on the Travis website. And even then it's huge and is just too much to scroll through.

I honestly don't see how it's worth the trouble. Can we not just let the Jenkins builds do the "all batteries included" tests and go back to using Travis as quick, if not imperfect feedback on most pull requests?

In general, I think I agree. What if as a middle ground we had one
configuration that just did apt-get install python-scipy (should be quick
since it's not building), which of course would imply the stock numpy as
well. All the other configurations would just be without scipy. We'd at
least get some basic scipy testing against the latest scipy in whatever
version of Ubuntu Travis' VM has without so many of the drawbacks Erik
points out.

Mike

On Wed, Apr 3, 2013 at 2:32 PM, Erik Bray [email protected] wrote:

The requirement to have SciPy available for the tests in Travis has, in my
opinion, made the Travis tests almost unusable. For one, it has inflated
the build time for them immensely--I used to get results for pull requests
pretty quickly.

Part of that may be due to growth in use of Travis by other projects and
thus increased demand for its build servers. But the fact that we have to
build all of SciPy as part of every single Astropy build has to be a major
source of the slowness. Plus it has made the logs difficult to get to--it
is necessary to download the log--it is not viewable directly on the Travis
website. And even then it's huge and is just too much to scroll through.

I honestly don't see how it's worth the trouble. Can we not just let the
Jenkins builds do the "all batteries included" tests and go back to using
Travis as quick, if not imperfect feedback on most pull requests?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/938
.

Michael Droettboom
http://www.droettboom.com/

I never followed closely the reasons we weren't just using the stock scipy from apt, but I guess it was just due to limitations in the Numpy versions it could be paired with? I agree as a middle ground to have one build configuration that uses the stock scipy, and have the rest just test without scipy.

I think scipy from apt will only work with Python 2.7. In general, I agree we've made the Travis builds a little bloated, and we could rely more on Jenkins to carry out the scipy tests.

@mdboom

There's a few different issues here, I'm going to try to separate them out:

Why not use apt-get for scipy?

This one's easy: you can't. Take a look at https://travis-ci.org/eteq/astropy/jobs/6029303 . That's a travis build where I use apt-get to install scipy, but not pip. You'll see in there that it says Scipy: not available in the test log. This makes sense, given the way travis does python builds - they're in isolated virtualenvs (http://about.travis-ci.org/docs/user/languages/python/#Travis-CI-Uses-Isolated-virtualenvs). The only reason we've got the apt-get in there at all is that it pulls in all the necessary dependencies needed to build scipy.

I agree this would make things better, but alas, Travis doesn't support it. We could ask travis people about this, though. We might be able to convince them to create a numpy/scipy environment. We couldn't control the version, presumably, but that's not necessarily a big deal if we use Jenkins for full version coverage, as @mdboom and @iguananaut suggest.

Scipy inflates build times.

I'm not sure that's true. Take a look at https://travis-ci.org/astropy/astropy/builds/6027883 , a recent build from master. You can see the "duration" time for the builds with scipy are roughly 3x the no-scipy build for 2.7 ... but it's roughly comparable for the 3.x builds. So in fact, I think it's really that having builds for every version we support is what's slowing down the build time.

You might also be getting tripped up by the 2.5 hr durations given for each build. As far as I can tell, that's just the sum of all build durations, which isn't wall time because we get 5 parallel builds. So in reality the time is ~30 min per build. I agree that's still a long time, but dropping the two scipy builds only shaves off ~6 min each. (~15 min per/5)

We don't need Scipy on Travis.

https://travis-ci.org/astropy/astropy/builds/6055696 is a good example for this, but there are plenty of other examples of PRs where features that are only on when scipy is present are affected. I think this is more common than the converse (PRs that pass with scipy, but not without). So if the point of Travis is to catch test failures before the PR is merged, we want the versions with Scipy.

Also, the build_sphinx build requires scipy (one of the examples uses scipy to generate a plot). And the build_sphinx build IMHO is the most important one, because it's very easy to accidentally write syntactically incorrect docstrings. Before putting that in, I spent a lot of time either running every PR I could through build_sphinx, or making changes in master to fix doc warnings that were missed. So I think that one is crucial.

Alternate Solution / tl;dr

I think we need scipy in travis for the above reasons. But it's true the Travis builds are taking a long time. So what about paring down travis to four builds:

2.7 w/scipy
3.3 (or whatever the latest 3.x is) w/scipy
build_sphinx (on 2.7)
2.7 w/o scipy (we could even dispense with this one, but it's the shortest, so it's not that much of an issue)

That would cut the wall build-time to ~ 10 min instead of 30. Then we can just rely Jenkins for catching version-related issues.

Sorry for the essay...

@eteq

@eteq - What about replacing your option (4) with 2.6 w/o scipy. It's pretty easy to accidentally use some feature that showed up in 2.7 and this gets caught by Travis. I find this useful.

I do agree in general with the pain in reading logs introduced by scipy.

@taldcroft

Just for the purposes of brainstorming, is there any way we could host binary eggs for the different scipy and numpy versions, and easy_install those into the virtualenvs? Maybe this is crazy, but it would save a lot of compile time if possible. Of course, PEP 427 would make this easier, but maybe we can already hack something together with eggs? Then we could keep all the builds we have.

I agree with @taldcroft we want to be testing 2.6 too.

I'd actually be in favor of getting rid of the sphinx build, if we have to get rid of any.

We should definitely keep the egg_info ones as they take no time at all and are required for making sure pip will work.

Just for info, I've tried to use binary eggs: https://travis-ci.org/astrofrog/astropy/jobs/6078504 - but I'm seeing issues with paths when it tries to install the egg...

I got further with the binary egg install: https://travis-ci.org/astrofrog/astropy/jobs/6078575 - the reason the build is to do with lapack (maybe I generated the egg incorrectly). Anyway, this is just an idea - if we could get this to work, this would make the builds a lot faster and we could keep them all. Thoughts?

EDIT: the issue is that the libraries on the machine I used to compile are in /lib64, and /usr/lib64, while the travis paths don't include '64'

This one's easy: you can't. Take a look at https://travis-ci.org/eteq/astropy/jobs/6029303 . That's a travis build where I use apt-get to install scipy, but not pip. You'll see in there that it says Scipy: not available in the test log. This makes sense, given the way travis does python builds - they're in isolated virtualenvs (http://about.travis-ci.org/docs/user/languages/python/#Travis-CI-Uses-Isolated-virtualenvs). The only reason we've got the apt-get in there at all is that it pulls in all the necessary dependencies needed to build scipy.

How does this make sense? We do have apt-get install python-numpy in there. Why would that work but not scipy?

Scratch that, I think I understand now--I see now that you later need to pip install it anyways. This is truly silly on Travis's part though.

@eteq

I like some combination of @eteq's suggestion of paring down the number of build configurations used in Travis, with @taldcroft's suggestion for pared down tests on Python 2.6, and @astrofrog's idea for binary installs of SciPy and numpy.

If we want to be daring we could even try using wheels (this would require using a development version of pip, though I think there should be a release with wheel support soon). But yes, if we just could have binary installs of those packages then most of my objections would go away.

@astrofrog

I really like @astrofrog's idea of a binary install - that would both speed things up and clean up the logs. We could even store those on github (in a new repo) to not have to worry about hosting. I'm also 👍 to @iguananaut's suggestion of using wheels.

So my feeling is that we should try to get that to work first, and if we can, then we can decide what to do about removing some versions - if we can speed things dramatically with just binaries, then @astrofrog is right we may just want to leave the matrix alone.

EDIT: oh, and @taldcroft's point that 2.6 may be better because of the missing-features related issues is a good one

@iguananaut re:Travis' python-numpy: It's not total nonesense - this gives you the assurance of complete control over your dependencies, and the Travis people don't have to be responsible for keeping everything up-to-date. Without something like that it'd be impossible to do our multi-version installs, for example. That said, you're right it's pretty silly that they don't allow the option of using the builtin packages. That's what readthedocs does, and it works great.

They should provide the option, and they should also make it possible to create package environments that persist between test runs.

Just for info, I'm currently working on making binaries for numpy/scipy for all the versions.

I just did a brief experiment in https://travis-ci.org/eteq/astropy/builds/6083723 - Travis works just fine if you use pip to install the version of pip that supports wheels (and then install wheel). So there's no reason we can't use wheels here.

@eteq

@eteq - oh cool, how does one make a wheel package?

@iguananaut

@iguananaut - the issue, as I discovered, is that one needs several wheel packages for e.g. linux, because numpy and scipy dynamically link to system libraries (e.g. ATLAS, LAPACK), and the location of these libraries is distribution-dependent. So one might need to produce many wheel files to cover all Python versions and Linux distributions. See https://bitbucket.org/dholth/wheel/issue/66/how-to-deal-with-extension-modules-that for a question I asked the wheel developer.

Looks like the answer was "either do it for all distros, or don't bother." I guess that's not too surprising, as that seems to be how people generally treat linux binaries.

That probably means we should just provide mac and windows binaries, then? After all, we can leave the binary packaging for distros to their respective package managers.

More on-topic: see https://travis-ci.org/eteq/astropy/builds/6168781 . That's a build where I tried for real to get the wheel stuff to work. There are some errors in the builds there, but the biggest problem is that scipy is failing due to apparently 403 errors. The real problem is revealed if you go to https://raw.github.com/astropy/wheels/master/wheelhouse/scipy-0.11.0-cp27-none-linux_x86_64.whl , though. You'll see "Error: blob is too big". Apparently github has limits on what they let you download via the web interface.

but, if you instead look at https://travis-ci.org/eteq/astropy/jobs/6170345 , you'll see that we can work around this by using the github API instead of the web site frontend to get the files. The trouble is, to do that you need the SHA of the BLOB for the file, and that's a very opaque string that isn't directly related to the filename. Another problem is that you need to muck around with HTTP GET headers to ,

So I see a few solutions:

write a little script that we include in the astropy repo that uses the github API to figure out the BLOB and download sit to the working directory. Then in .travis.yml we run that script to get scipy and pip install on the downloaded file.
put some sort of mapping in the .travis.yml script that we use to figure out the BLOB SHA. That would have to be manually updated when we add new wheels/scipy versions, but then everything is still restricted to .travis.yml
host the scipy wheel elsewhere. I'm not sure where "elsewhere" might be, though

I was never super crazy about hosting them on GitHub in the first place to be honest. I'd much rather we just had a simple package index somewhere, but I don't know where best to host that. I do run a package index at http://stsdas.stsci.edu/download/packages/, but that requires institutional support from STScI to depend on...

For testing purposes, why don't we just put it back in my web-space or one of yours? I'd much rather we host the somewhere relatively private in the sense we don't want random people downloading these wheels and then blaming us if they don't work.

nod Since they're just for testing purposes I think that's fine. We could even do both in case one of us goes down for some reason or gets hit by a truck :)

@eteq

Ok, well all the wheel files should already be available from http://www.mpia-hd.mpg.de/~robitaille/python/<wheel_file>/ - so @eteq, if you want to try testing that with the full Astropy testing suite, we can see how much time is gained to decide whether this is worthwhile.

@astrofrog

Alright, take a look at https://travis-ci.org/eteq/astropy/builds/6199778 - that uses @astrofrog's site only. Some major speedups are apparent (2.7 w/scipy is less than half the time), but some of that is because a lot of the builds failed.

It appears that 3.2 is failing I think due to some sort of linking problem for numpy - see https://travis-ci.org/eteq/astropy/jobs/6199780

2.6 is failing due to some sort of Cython linking problem - see https://travis-ci.org/eteq/astropy/jobs/6199785

@astrofrog - do you know what's going on with these two?

Are you still pushing forward with getting wheels to work? I'm working on some other projects that could use the speedup as well. Can anyone clarify why we should not host the wheels on github? Is it just the difficulty of tracking the SHA key?

Would it be possible to put a setup.py in the wheels repository, or something along those lines, to just install all / some subset of the wheels?

Just a side note - I think we should check if the open file testing on Travis is slowing down the builds, as that's something that could just live on Jenkins - it's not crucial, but useful.

@keflavich

@keflavich @astrofrog @iguananaut - take a look at https://travis-ci.org/eteq/astropy/builds/6523806 . I also tried @astrofrog's idea of turning off the open file testing in https://travis-ci.org/eteq/astropy/builds/6525068 (otherwise identical)

This took a fair bit more effort than I expected, but I've now got all the wheels working. It turned out the best thing was to just use a virutalbox VM of Ubuntu 12.04_64 server, the exact same thing Travis uses. That combined with the "dead snakes PPA" (which Travis' docs says they used) made it possible to generate wheels for all the necessary versions, that are now all building sucessfully.

A good thing, then, is to compare that to, say, https://travis-ci.org/astropy/astropy/builds/6517614 , which is a "typical" master build. You can see the wheels roughly halve the total build time (and the scipy builds are now sometimes faster than other builds). The open file detection seems to be basically unimportant, except maybe for the numpy 1.6.2 builds (which were each ~5 min shorter with it off). That may be coincidence, though - those builds may have just been running on slower Travis VMs.

It seems pretty obvious at this point that we want to do this. We may also want to cut down on the number of builds/versions in Travis, but I suggest we address that in another issue.

So unless there are any objections, I'll work on a PR to implement this, and we can close that issue when that's in (the builds above are getting their wheels from a server that shouldn't be treated as a "production" environment, so it'll need some more modification before merging.)

Oh yes, this looks much much better. Thanks for doing that. Any decision on where we want to store the built wheels for now?

I'd still like to also cut down on the Astropy build output (while still having it saved for examination of unsuccessful builds) but I will work on that as a separate issue. As it is this has already made the log files much more usable.

@astrofrog

@astrofrog had earlier suggested we put them on his server for now while we wait for a better solution. That seems fine to me. It's possible we'll be able to use a ci server that scipy is setting up, but that's not for sure yet.

This issue can be closed since #1029 now has SciPy installing from wheels.

Navigation Menu

Repository navigation

Remove scipy tests from Travis #938

24 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Remove scipy tests from Travis #938

Description

Activity

mdboom commented on Apr 3, 2013

embray commented on Apr 3, 2013

astrofrog commented on Apr 3, 2013

eteq commented on Apr 4, 2013

taldcroft commented on Apr 5, 2013

astrofrog commented on Apr 5, 2013

astrofrog commented on Apr 5, 2013

astrofrog commented on Apr 5, 2013

embray commented on Apr 5, 2013

embray commented on Apr 5, 2013

eteq commented on Apr 5, 2013

embray commented on Apr 5, 2013

astrofrog commented on Apr 5, 2013

eteq commented on Apr 5, 2013

astrofrog commented on Apr 5, 2013

24 remaining items

astrofrog commented on Apr 6, 2013

eteq commented on Apr 9, 2013

eteq commented on Apr 9, 2013

embray commented on Apr 9, 2013

astrofrog commented on Apr 9, 2013

embray commented on Apr 9, 2013

astrofrog commented on Apr 9, 2013

eteq commented on Apr 10, 2013

keflavich commented on Apr 20, 2013

astrofrog commented on Apr 21, 2013

eteq commented on Apr 22, 2013

embray commented on Apr 22, 2013

eteq commented on Apr 22, 2013

embray commented on May 30, 2013

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions