Skip to content

PythonPackage class#2778

Merged
tgamblin merged 1 commit intodevelopfrom
features/pythonpackage2
Jan 17, 2017
Merged

PythonPackage class#2778
tgamblin merged 1 commit intodevelopfrom
features/pythonpackage2

Conversation

@citibeth
Copy link
Copy Markdown
Member

@citibeth citibeth commented Jan 8, 2017

Replaces #2709

@citibeth
Copy link
Copy Markdown
Member Author

citibeth commented Jan 8, 2017

@adamjstewart This should be the newly rebased PythonPackage PR. Please look in the commit history, I created this PR in three steps:

  1. Got rid of nolink from Python packages (in your branch)

  2. Merge branch '_pythonpackage0' into _develop, taking YOUR branch by default whenever there were conflicts.

  3. Maually reviewed changes from (2), updating when needed. Please pay ESPECIAL attention this commit, these are things I though the original PR got wrong. Of course I might be wrong... so please double-check: d7b9626

@citibeth
Copy link
Copy Markdown
Member Author

citibeth commented Jan 8, 2017

@adamjstewart Can I hand this PR back to you now? spack test succeeded on my local machine. And the failed Travis test seems to do with SVN. Really weird... Anyway, I have rebased as I promised, so I think it's time to hand it back.

depends_on('py-dbf', type=('build', 'run'))
depends_on('py-xlrd', type=('build', 'run'))
depends_on('py-sqlalchemy', type=('build', 'run'))
depends_on('py-SQLAlchemy', type=('build', 'run'))
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change doesn't belong here.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops... that needs to be fixed (thought I fixed it). You are right, it should be py-sqlalchemy.


extends('python')

depends_on('py-ptyprocess')
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This dep shouldn't be removed.

extends('python', ignore=r'bin/pytest')
depends_on('py-six', type=('build', 'run'))
depends_on('py-astroid', type=('build', 'run'))
depends_on('py-logilab-common', type=('build', 'run'))
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These deps shouldn't be removed.

version('2.5.6', 'a36e758b633ce6aec6a5f450bfee980f')

extends('python')
depends_on('py-six', type=('build', 'run'))
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This dep shouldn't be removed.

os.environ['SPATIALINDEX_LIBRARY'] = \
join_path(lib, 'libspatialindex.%s' % dso_suffix)
os.environ['SPATIALINDEX_C_LIBRARY'] = \
join_path(lib, 'libspatialindex_c.%s' % dso_suffix)
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks like a step in the wrong direction.

@citibeth
Copy link
Copy Markdown
Member Author

citibeth commented Jan 8, 2017

@adamjstewart Can you apply the changes that are needed? I intentionally put this PR on the main LLNL repo to facilitate that process.

My only comment would be that I think (but am not 100% sure) that your changes to py-rtree might break it. That's why I changed it back. The env vars need to be set for build, not run. py-rtree then embeds the needed information into its build, RPATH-style.

@adamjstewart
Copy link
Copy Markdown
Member

My only comment would be that I think (but am not 100% sure) that your changes to py-rtree might break it. That's why I changed it back. The env vars need to be set for build, not run. py-rtree then embeds the needed information into its build, RPATH-style.

That's what it does. spack_env is the environment Spack sets when you build it, run_env is the environment that gets set in the resulting module file.

@citibeth
Copy link
Copy Markdown
Member Author

citibeth commented Jan 8, 2017

I see. I knew that setup_environment() set up the env for modules. But apparently it also sets the environment for building. http://spack.readthedocs.io/en/latest/module_file_support.html?highlight=setup_environment

@tgamblin @alalazo What is the best approach here if we want an environment set for building, but not for running?

@adamjstewart adamjstewart changed the title Preliminary PythonPackage class (2) [WIP] Preliminary PythonPackage class Jan 8, 2017
# if pkg.extendees:
# directive = 'extends'
# msg = 'Packages can extend at most one other package.'
# raise DirectiveError(directive, msg)
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this safe to remove? It causes problems for packages that require:

extends('python', ignore='bin/.*')

since there is already:

extends('python')

in the base class.

raise DirectiveError(msg.format(pkg.name, name))
directive = 'variant'
msg = "Invalid variant name in {0}: '{1}'"
raise DirectiveError(directive, msg.format(pkg.name, name))
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These calls to DirectiveError were missing a directive.

@adamjstewart
Copy link
Copy Markdown
Member

Ok, the rebase is complete now. Let me just port all of the packages added in #2320. I also noticed that none of these packages have the right deptypes (the default instead of build/run) so I'll have to fix those up.

@citibeth
Copy link
Copy Markdown
Member Author

citibeth commented Jan 8, 2017 via email

@adamjstewart
Copy link
Copy Markdown
Member

Ok, this is probably all of the changes I want to make in this PR. Now the only remaining thing to do is to test it. Does anyone want to see any specific tests before this is ready to merge? I was going to make sure the "important" packages still build (numpy, scipy, matplotlib) and test all of the "weird" packages (everything with an install method that doesn't look like the following):

def install(self, spec, prefix):
    setup_py('install', '--prefix={0}'.format(prefix))

@citibeth
Copy link
Copy Markdown
Member Author

citibeth commented Jan 9, 2017 via email

@adamjstewart
Copy link
Copy Markdown
Member

@citibeth Will do. Is it just import rtree?

Also, I'll try to activate all of the packages I build too just to make sure that functionality still works.

@citibeth
Copy link
Copy Markdown
Member Author

citibeth commented Jan 9, 2017 via email

@adamjstewart
Copy link
Copy Markdown
Member

I still think the change doesn't feel right. It is akin to putting RPATH in your binary but then overriding it with LD_LIBRARY_PATH anyway. I do not want any extra and unnecessary env vars in the py-rtree module.

I didn't change that. That's how it was to begin with. But I'll test it anyway just to be sure.

@citibeth
Copy link
Copy Markdown
Member Author

citibeth commented Jan 9, 2017 via email

@adamjstewart
Copy link
Copy Markdown
Member

That's exactly right.

I decided to try installing and activating every single Python package in Spack. That ought to catch a slew of conflicting files. Once I can get everything installed, we'll be safe to merge.

@citibeth
Copy link
Copy Markdown
Member Author

citibeth commented Jan 9, 2017 via email

@adamjstewart
Copy link
Copy Markdown
Member

adamjstewart commented Jan 11, 2017

Ok, as of the last commit, I am able to install every py-* package except:

  • py-genders: missing a genders dep
  • py-meep: header file linking broke, not sure why. It was complicated to begin with
  • py-mysqldb1: missing dependency on mysql
  • py-nbconvert: failed to download CSS?
  • py-pygobject: tarball disappeared, can't find another version on PyPi
  • py-pygtk: invalid spec due to cairo+X vs cairo~X
  • py-rpy2: missing dependencies on icuuc, icui18

I added FIXME comments to these packages. Someone can fix them as they need them.

Ok, last step is to try activating all of these packages. We'll see what conflicts we find!

@adamjstewart adamjstewart removed the WIP label Jan 12, 2017
# These dependencies breaks concretization
# See https://github.com/LLNL/spack/issues/2793
# depends_on('py-configparser', when='^python@:3.3.999', type=('build', 'run')) # noqa
# depends_on('py-enum34', when='^python@:3.1.999', type=('build', 'run'))
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe you can use [email protected] here; [email protected] acts like a set that also matches [email protected]. Not sure where else it happen in this PR (and this one's in a comment); but something to look out for.

Are you able to grep through and fix for places where .999 occurs?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't believe that's the case. You need to use a range. 3.3 only matches 3.3, so if you tried to install something that depends on [email protected], it will try to fetch python-3.3.tar.gz, which won't exist.


extends('python')

depends_on('[email protected]:2.7.999,3.3:')
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be depends_on('[email protected],3.3:').

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This won't work. There is no Python version 2.7, so it will never be matched. Try editing it and running spack spec.

@citibeth
Copy link
Copy Markdown
Member Author

@adamjstewart A BIG THANKS! This is the kind of detail work that moves a project from "sort of works" to "really usable."

I agree, secret downloads by pip are a problem, and they should never be happening (or allowed). It would be nice if there's a way to regression-test this property. My thoughts are either we get a regression test in an Internet-free container somewhere; or we find a way to disable PIP to catch these errors. Either way, fixing the existing PIP problems as you have done is a big step forward.

@citibeth
Copy link
Copy Markdown
Member Author

citibeth commented Jan 12, 2017 via email

@citibeth
Copy link
Copy Markdown
Member Author

citibeth commented Jan 12, 2017 via email

@adamjstewart
Copy link
Copy Markdown
Member

depends_on('py-enum34', when='^python@:3.1', type=('build', 'run'))

This is valid, although it doesn't solve the problem reported in #2793.

depends_on('[email protected]:2.7,3.3:')

This does not work, I've tried it before. It complains about a range that starts and stops at the same version.

@citibeth
Copy link
Copy Markdown
Member Author

citibeth commented Jan 12, 2017 via email

@citibeth
Copy link
Copy Markdown
Member Author

GitHub doesn't allow me to approve this PR because I'm its "author." But yes, I do approve it! 👍

@adamjstewart
Copy link
Copy Markdown
Member

Haha, I "approved" it for you

@adamjstewart
Copy link
Copy Markdown
Member

I wish there was an easier way to reassign ownership.

@citibeth
Copy link
Copy Markdown
Member Author

@tgamblin I recommend this gets merged ASAP. It's a BIG change. And without a merge, new Python packages continue to come in that will have to be fixed after-the-fact.

@adamjstewart
Copy link
Copy Markdown
Member

I tried rebasing this to convert the latest new Python packages, figuring it would be easy since there are no conflicts, but it didn't work. @citibeth I'm not sure how you went about rebasing this, but I don't think it can be rebased easily anymore. It still asks me to resolve conflicts for things like the nolink conversion that I thought you had worked out. @tgamblin This needs to be merged or else I'm no longer going to support it. It has become too difficult to maintain. Basically, if there ever becomes any conflicts with existing files, I can't do anything about it anymore.

@citibeth
Copy link
Copy Markdown
Member Author

citibeth commented Jan 16, 2017 via email

@adamjstewart
Copy link
Copy Markdown
Member

No need to take a look. As long as we can get this merged soon, it shouldn't be a problem, I hope.

@tgamblin tgamblin changed the title Preliminary PythonPackage class PythonPackage class Jan 17, 2017
- Add a PythonPackage class with build system support.
  - Support build phases in PythonPackage
  - Add a custom sanity check for PythonPackages
  - Get rid of nolink dependencies in python packages

- Update spack create to use new PythonPackage class

- Port most of Python packages to new PythonPackage class

- Conducted a massive install and activate of Python packages.
  - Fixed bugs introduced by install and activate.

- Update API docs on PythonPackage
@tgamblin tgamblin force-pushed the features/pythonpackage2 branch from fd61ae1 to 25869a0 Compare January 17, 2017 02:22
@tgamblin
Copy link
Copy Markdown
Member

@adamjstewart @citibeth this wasn't so much rebased as merged. It has two bases at different points in the history of develop, which were merged together then added to.

In the future, if you combine two branches like this, it's probably best to rebase one onto develop, then rebase the other onto that if you hope to continue rebasing on develop as things progress. This keeps this history linear. Turning on rerere makes the process much less painful.

OTOH, you can also just keep merging develop into your branch instead of bothering to rebase. Since we squash anyway, continuously merging develop and continuously rebasing on develop will end up looking the same in the end.

It looks like most of this work is @adamjstewart's, but if I squash with GitHub, the commit shows up as authored by the PR submitter. So, I just squashed the result of the merge manually and force-pushed a single commit by @adamjstewart.

@adamjstewart: Can you review these changes and make sure they work and look ok? There was only one minor conflict in spack create that needed to be resolved. Given that you've added a lot of stability with this stuff, I'd like to get this into 0.10.

@tgamblin
Copy link
Copy Markdown
Member

@adamjstewart: note that you should still be able to push commits on top of this if you need to change anything.

@citibeth
Copy link
Copy Markdown
Member Author

citibeth commented Jan 17, 2017 via email

@tgamblin tgamblin merged commit c0aaa8f into develop Jan 17, 2017
@adamjstewart
Copy link
Copy Markdown
Member

Everything looks good to me! Thanks for getting this in @tgamblin. And thanks to @citibeth for helping get this rebased. Even if it was hacky, it certainly saved me a lot of time.

Now I'll have to start thinking about the next steps for the PythonPackage class. I want to add import tests to make sure the module can actually be imported after installation. Any requests aside from that?

@citibeth
Copy link
Copy Markdown
Member Author

citibeth commented Jan 17, 2017 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants