@adamjstewart @lee218llnl: care to test this out?

@tgamblin I will try to build a bunch of packages and test them out.

@tgamblin, this is breaking py-basemap:

==> '/nfs/tmp2/lee218/delete/spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/python-2.7.13-mdeplzdkoh5635rrmm4vfeeqagqmcljk/bin/python' 'setup.py' '--no-user-cfg' 'install' '--prefix=/nfs/tmp2/lee218/delete/spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/py-basemap-1.0.7-6auw6q62novxuecffpqgmzxiz2onzvtm' '--single-version-externally-managed' '--root=/'
In file included from nad2bin.c:7:0:
src/projects.h:478:1: warning: function declaration isn't a prototype [-Wstrict-prototypes]
 void pj_deallocate_grids();
 ^
nad2bin.c:46:13: warning: function declaration isn't a prototype [-Wstrict-prototypes]
 static void Usage()
             ^
In file included from src/pj_malloc.c:5:0:
src/projects.h:478:1: warning: function declaration isn't a prototype [-Wstrict-prototypes]
 void pj_deallocate_grids();
 ^
/nfs/tmp2/lee218/delete/spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/python-2.7.13-mdeplzdkoh5635rrmm4vfeeqagqmcljk/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'namespace_packages'
  warnings.warn(msg)
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
   or: setup.py --help [cmd1 cmd2 ...]
   or: setup.py --help-commands
   or: setup.py cmd --help

error: option --single-version-externally-managed not recognized

Currently the py-basemap packages lists py-setuptools as a dependence, even though it doesn't directly depend on it. Even if we remove that, though, some of py-basemap's dependences do depend on py-setuptools, so py-setuptools appears in py-basemaps spec. Is there anyway to make it so that this arg is only added when py-setuptools is a direct dependence (and then we can separately fix py-basemap)?

Based on @tgamblin's point (3) in the OP, I'm not surprised it breaks py-basemap, see #1964. Can PythonPackage optionally do eggs for the few packages that need them (such as py-basemap)?

@citibeth: in this case it's actually what @lee218llnl pointed out -- my check in PythonPackage was wrong. I'll update it to only check immediate setuptools dependencies. py-basemap also shouldn't depend on setuptools. There are likely other packages like that, which list setuptools as a dependency but don't need it.

@lee218llnl @adamjstewart: ok, I got py-basemap working per the fix above. Removing the setuptools dependency is the right thing to do, and PythonPackage now only uses setuptools arguments when setuptools is an immediate dependency (or when it's the package being installed).

I felt adventurous and tried installing py-jupyter-notebook, which turned up 15 more packages that don't need setuptools. Most of the IPython stack seems to use distutils unless setup.py is invoked to make a bdist. We should probably try installing all the Python packages that depend on setuptools before merging this, so I am marking it WIP.

@krafczyk: I ended up cleaning up a few dependencies of py-jupyter-notebook to make them build. Can you take a look at those? Specifically:

• fixed py-cffi for Darwin
• Got rid of the problematic css fetch that happens when you install py-nbconvert, and bundled style-min.css with Spack.

@tgamblin, I'll have a look.

I agree with all of the setuptools dependency changes.

@tgamblin looks good for py-basemap, thanks.

@tgamblin, I had to add a couple of commits for missing dependencies for matplotlib and yt. After these changes I'm good with this PR. I've added the commits to features/no-python-eggs on my spack fork, pull from that branch to get my changes.

@tgamblin This PR is great. I have started to install all Python packages, and found some more setuptools dependency problems. The fixes can be found at in my repo and listed below:

Add required setuptools dependency:

• py-bottleneck
• py-csvkit
• py-emcee
• py-periodictable
• py-restview
• py-sncosmo
• py-wcsaxes

Remove unneeded setuptools dependency:

• py-joblib
• py-markdown
• py-terminado

@paulhopkins: thanks! I added you as a collaborator, so you should be able to push to this branch now. Do you mind pushing your updates here?

Thanks @tgamblin, I have uploaded my changes. I have not yet managed to install py-pyqt but I will try and look at it later.

@paulhopkins: is that the only one you haven't tried yet?

@tgamblin I have 'tried' them all, but these are the ones that have failed:

py-pyqt: Fails to find Phonon headers. These should be QT package, but they are not there.
py-guidata, depends on py-pyqt
py-guiqwt, depends on py-pyqt
py-pythonqwt, depends on py-pyqt
py-qtawesome, depends on py-pyqt
py-qtconsole, depends on py-pyqt
py-qtpy, depends on py-pyqt
py-tuiview, depends on py-pyqt

py-yt, Fixed by @krafczyk

py-meep, Meep did not install
py-mysqldb1, MySQLDB did not install
py-pyfftw, Linking error
py-rpy2, Build error

Fixed in new commit:
py-pysocks, Does not depend on setuptools

py-meep has known problems. I wrote the original package. It used to work at one time but no longer does, and the package hasn't changed. It was fairly hacky to begin with. I'll take a look at it someday, but it wasn't broken by this PR.

Would it make sense for this PR to eliminate the write_easy_install_pth function from the python package?

EDIT: I overlooked this part of the original message:

we should still support that for packages that want to override this behavior for some reason.

I'm curious: why would a package want to override this? There aren't currently any python modules in Spack's default repo that do this.

I rebased this and included the changes from @paulhopkins and @krafczyk, so it should be ready to go.

@scheibelp:

I'm curious: why would a package want to override this? There aren't currently any python modules in Spack's default repo that do this.

I don't know, but I don't think it's good practice to break the functionality. I'm almost positive that Python packages wouldn't want to use eggs if building with Spack. But .pth files are one way that Python knows how to do imports, and if there is a package that relies on them, or a site that somehow depends on them, I don't want to make Spack break a Python feature if we don't have to.

@adamjstewart in 8d0758f you added the following function to py-numpy (besides other stuff)

def setup_dependent_package(self, module, dependent_spec):
        python_version = self.spec['python'].version.up_to(2)
        arch = '{0}-{1}'.format(platform.system().lower(), platform.machine())

        self.spec.include = join_path(
            self.prefix.lib,
            'python{0}'.format(python_version),
            'site-packages',
            'numpy-{0}-py{1}-{2}.egg'.format(
                self.spec.version, python_version, arch),
            'numpy/core/include')

Doesn't this conflict with this change here? I think we ran into this problem when one of our packages has to link against numpy on a barebone Debian. I fear that we didn't note this prior because we always worked on systems where a non-spack numpy was available.

Probably? py-meep just needs to know where the py-numpy headers live. If they no longer live in this location due to the removal of eggs, we will need to update py-numpy with the new header location.

Okay, I'll have a PR for that in a couple of days, unfortunately I was unable to build meep but haven't had the time to investigate why yet

Yeah, I noticed that as well. meep and py-meep were working perfectly for me when I wrote the original packages, but the last time I tested them they didn't build. Not sure what changed, but hopefully you'll figure it out.