This might sound like a dumb question, but... do all packages that use the Python build system extend Python? For example, py-autopep8 does not; is that for real, or is it a mistake.

I believe the Spack package superclass should be based on the build system used, let's not get that confused with the contents of the package. If somebody wants to use Python setuptools to build a C++ package, they can go right ahead...

For that reason, I think that PythonPackage should probably NOT auto-extend python.

So in PythonPackage, I added:

extends('python')

But in some packages, I needed to add something like:

extends('python', ignore=r'bin/.*')

This is causing the documentation tests to fail with a DirectiveError because you can't extend a package more than once. How can I resolve this problem?

Would you like to try #2281 in this PR as well; or do you think that's best left for another day?

BTW... looking forward to this one!

do all packages that use the Python build system extend Python?

Well, all packages that need to run python setup.py have to depend on Python. And depends_on == extends. The only difference between extends and depends_on is that you are allowed to run spack activate py-package and it will symlink the contents into the Python installation directory.

For example, py-autopep8 does not

https://github.com/LLNL/spack/blob/develop/var/spack/repos/builtin/packages/py-autopep8/package.py#L38

Would you like to try #2281 in this PR as well; or do you think that's best left for another day?

Perhaps. I think this PR is going to need a lot of testing. And like your capitalization PR, it affects a lot of packages, so I would like to get it merged sooner rather than later (obviously after your capitalization PR which appears to be done). But I think it ought to be really easy to add a custom url_for_version to the PythonPackage base class. Or for a PyPiPackage base class that extends PythonPackage. It depends on how many Python packages are available on PyPi and how many aren't. I know py-phonopy and py-meep aren't.

By the way, setup.py has a lot of options:

$ python setup.py --name
# name of the package
$ python setup.py --version
# version of the tarball
$ python setup.py --description
# short description of the package
$ python setup.py --requires
# supposed to list dependencies, but often left blank

In combination with the url_for_version, it may be possible to fill in everything needed in a package.py during spack create.

I'll make a similar PR for the R-build system after #2475 is merged. R seems even simpler.

Ok, I have almost all of the packages that call setup_py converted to PythonPackage. I just have a couple types remaining.

The first type is like py-rtree, where environment variables need to be set during install:

lib = os.path.join(spec['libspatialindex'].prefix, 'lib')
os.environ['SPATIALINDEX_LIBRARY'] = os.path.join(lib, 'libspatialindex.%s' % dso_suffix)                
os.environ['SPATIALINDEX_C_LIBRARY'] = os.path.join(lib, 'libspatialindex_c.%s' % dso_suffix)

setup_py('install', '--prefix=%s' % prefix)

Is this what setup_environment is for?

A second type is where there are more phases than normal, like in py-meep:

self.setup_py('clean', '--all')
self.setup_py('build_ext', include_flags, library_flags)
self.setup_py('install', '--prefix={0}'.format(prefix))
self.setup_py('bdist', include_flags, library_flags)

Right now, the best idea I have is to create <phase> and <phase_args> functions for every possible phase, and then allow packages to override the phase list. But I'm wondering if there isn't an easier way. Could I somehow automate this process so all I need to do is override phase without having to create almost identical functions for every phase? Like dynamically generate the functions at runtime? http://stackoverflow.com/questions/13184281/python-dynamic-function-creation-with-custom-names seems to be a similar problem as mine. But I'm not that familiar with closure in Python.

@alalazo I'm seeing a lot of weird problems with Travis since we updated to pytest. #2711 is one example. Also, check out the last Travis run. Python 2.7 passed even though one of the tests failed!

@adamjstewart @tgamblin I think the problem you are seeing here is due to the script we run. We should change:

coverage run bin/spack test "$@"
coverage combine

to

coverage run bin/spack test "$@" && coverage combine

I'll check if I am right and and submit a PR right away in case.

In combination with the url_for_version, it may be possible to fill in everything needed in a package.py during spack create.

Especially if the package resides on PyPI... which provides "DOAP" records giving a lot of useful info.

Oh boy, I think every file in this PR has a conflict now. This will be fun to rebase.

On rebasing frequently: are you and @citibeth aware of rerere? It makes rebasing things with longer histories pretty painless for me. Not sure why it's not on by default.

@tgamblin I had never heard of that. I'll have to try it out.

While I have your mastery of git on the table, have you ever seen a message like this?

$ git push origin develop
Counting objects: 376, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (41/41), done.
Writing objects: 100% (376/376), 80.32 KiB | 0 bytes/s, done.
Total 376 (delta 188), reused 346 (delta 161)
remote: Resolving deltas: 100% (188/188), completed with 48 local objects.
To github.com:adamjstewart/spack.git
 ! [remote rejected] develop -> develop (pre-receive hook declined)
error: failed to push some refs to '[email protected]:adamjstewart/spack.git'

It's been happening to me very frequently lately. I don't have any webhooks or protected branches on my fork.

I think I can resolve those conflicts. Give me an hour, I'll put up a new version of this branch that will go down more easily.

@citibeth No worries, I'm already working on rebasing. Hopefully git rerere will make things quicker.

@adamjstewart did you rebase and then push? That typically requires a force-push because rebase has rewritten commits, changing the history.

If that is your problem, my advice would be to always work on branches, even within your fork, so that you only ever have to pull from upstream develop. e.g.:

git remote add upstream <main spack repo>
git checkout -b feature
# work work work
git push origin feature
# do PR
# PR is merged
git checkout develop
git pull upstream develop
# now you're synced and feature is merged
git branch -D feature

It looks like you're on a branch in your fork even here, though, so I dunno why you're getting that. Hm.

@citibeth If that's all you have to do then that is probably easier. I tried rebasing but messed some files up, so I'll let you handle the rebase.

@tgamblin I don't think I rebased. My develop should be in sync with your develop.

Replaced by #2778, which is (mostly) a re-based version of this PR.

@adamjstewart: if you create a new fork and force push it there I could take a look...

@tgamblin All of a sudden the problem disappeared. I wonder if my /tmp is full or something weird. I also filed an issue with GitHub support, so I'll see what they have to say.