Conversation
|
@adamjstewart This should be the newly rebased PythonPackage PR. Please look in the commit history, I created this PR in three steps:
|
|
@adamjstewart Can I hand this PR back to you now? |
| depends_on('py-dbf', type=('build', 'run')) | ||
| depends_on('py-xlrd', type=('build', 'run')) | ||
| depends_on('py-sqlalchemy', type=('build', 'run')) | ||
| depends_on('py-SQLAlchemy', type=('build', 'run')) |
There was a problem hiding this comment.
This change doesn't belong here.
There was a problem hiding this comment.
Oops... that needs to be fixed (thought I fixed it). You are right, it should be py-sqlalchemy.
|
|
||
| extends('python') | ||
|
|
||
| depends_on('py-ptyprocess') |
There was a problem hiding this comment.
This dep shouldn't be removed.
| extends('python', ignore=r'bin/pytest') | ||
| depends_on('py-six', type=('build', 'run')) | ||
| depends_on('py-astroid', type=('build', 'run')) | ||
| depends_on('py-logilab-common', type=('build', 'run')) |
There was a problem hiding this comment.
These deps shouldn't be removed.
| version('2.5.6', 'a36e758b633ce6aec6a5f450bfee980f') | ||
|
|
||
| extends('python') | ||
| depends_on('py-six', type=('build', 'run')) |
There was a problem hiding this comment.
This dep shouldn't be removed.
| os.environ['SPATIALINDEX_LIBRARY'] = \ | ||
| join_path(lib, 'libspatialindex.%s' % dso_suffix) | ||
| os.environ['SPATIALINDEX_C_LIBRARY'] = \ | ||
| join_path(lib, 'libspatialindex_c.%s' % dso_suffix) |
There was a problem hiding this comment.
This looks like a step in the wrong direction.
|
@adamjstewart Can you apply the changes that are needed? I intentionally put this PR on the main LLNL repo to facilitate that process. My only comment would be that I think (but am not 100% sure) that your changes to |
That's what it does. |
|
I see. I knew that @tgamblin @alalazo What is the best approach here if we want an environment set for building, but not for running? |
| # if pkg.extendees: | ||
| # directive = 'extends' | ||
| # msg = 'Packages can extend at most one other package.' | ||
| # raise DirectiveError(directive, msg) |
There was a problem hiding this comment.
Is this safe to remove? It causes problems for packages that require:
extends('python', ignore='bin/.*')since there is already:
extends('python')in the base class.
| raise DirectiveError(msg.format(pkg.name, name)) | ||
| directive = 'variant' | ||
| msg = "Invalid variant name in {0}: '{1}'" | ||
| raise DirectiveError(directive, msg.format(pkg.name, name)) |
There was a problem hiding this comment.
These calls to DirectiveError were missing a directive.
|
Ok, the rebase is complete now. Let me just port all of the packages added in #2320. I also noticed that none of these packages have the right deptypes (the default instead of build/run) so I'll have to fix those up. |
|
Thanks!
…On Sun, Jan 8, 2017 at 5:31 PM, Adam J. Stewart ***@***.***> wrote:
Ok, the rebase is complete now. Let me just port all of the packages added
in #2320 <#2320>. I also noticed that
none of these packages have the right deptypes (the default instead of
build/run) so I'll have to fix those up.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#2778 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AB1cd5XWbE0YrfwFwE7PNivoJUUo3K3oks5rQWPbgaJpZM4LdxP6>
.
|
|
Ok, this is probably all of the changes I want to make in this PR. Now the only remaining thing to do is to test it. Does anyone want to see any specific tests before this is ready to merge? I was going to make sure the "important" packages still build (numpy, scipy, matplotlib) and test all of the "weird" packages (everything with an install method that doesn't look like the following): def install(self, spec, prefix):
setup_py('install', '--prefix={0}'.format(prefix)) |
|
If you want to keep the change to `py-rtree`, it would be good to do the
"import" test on it too.
…On Sun, Jan 8, 2017 at 6:21 PM, Adam J. Stewart ***@***.***> wrote:
Ok, this is probably all of the changes I want to make in this PR. Now the
only remaining thing to do is to test it. Does anyone want to see any
specific tests before this is ready to merge? I was going to make sure the
"important" packages still build (numpy, scipy, matplotlib) and test all of
the "weird" packages (everything with an install method that doesn't look
like the following):
def install(self, spec, prefix):
setup_py('install', '--prefix={0}'.format(prefix))
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#2778 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AB1cd-wSW1ripjIFzEPaGpfCY7Cx5Fnfks5rQW9sgaJpZM4LdxP6>
.
|
|
@citibeth Will do. Is it just Also, I'll try to activate all of the packages I build too just to make sure that functionality still works. |
|
Try: from rtree import index
I still think the change doesn't feel right. It is akin to putting RPATH
in your binary but then overriding it with LD_LIBRARY_PATH anyway. I do
not want any extra and unnecessary env vars in the py-rtree module.
On Jan 8, 2017 11:18 PM, "Adam J. Stewart" <[email protected]> wrote:
@citibeth <https://github.com/citibeth> Will do. Is it just import rtree?
Also, I'll try to activate all of the packages I build too just to make
sure that functionality still works.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#2778 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AB1cd0J5rOy_70drNGQxEEe1uugLxxIwks5rQbUIgaJpZM4LdxP6>
.
|
I didn't change that. That's how it was to begin with. But I'll test it anyway just to be sure. |
|
On Mon, Jan 9, 2017 at 5:09 PM, Adam J. Stewart ***@***.***> wrote:
I still think the change doesn't feel right. It is akin to putting RPATH
in your binary but then overriding it with LD_LIBRARY_PATH anyway. I do not
want any extra and unnecessary env vars in the py-rtree module.
On third thought, I think my concern is unwarrnated. The new code is:
```
def setup_environment(self, spack_env, run_env):
lib = self.spec['libspatialindex'].prefix.lib
spack_env.set('SPATIALINDEX_LIBRARY',
join_path(lib, 'libspatialindex.%s' % dso_suffix))
spack_env.set('SPATIALINDEX_C_LIBRARY',
join_path(lib, 'libspatialindex_c.%s' % dso_suffix))
```
Am I right that this will set the environment used to build (spack_env) but
not the environment in the module files (run_env)? In that case, there
should be no problem here.
I do appreciate testing, since this is still textually quite different from
what was therebefore:
https://github.com/LLNL/spack/blob/develop/var/spack/repos/builtin/packages/py-rtree/package.py
|
|
That's exactly right. I decided to try installing and activating every single Python package in Spack. That ought to catch a slew of conflicting files. Once I can get everything installed, we'll be safe to merge. |
|
Wow!
…On Jan 9, 2017 5:16 PM, "Adam J. Stewart" ***@***.***> wrote:
That's exactly right.
I decided to try installing and activating every single Python package in
Spack. That ought to catch a slew of conflicting files. Once I can get
everything installed, we'll be safe to merge.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#2778 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AB1cd9Gv0VyzxvixADCPsoS7A0N4uI4Dks5rQrHRgaJpZM4LdxP6>
.
|
|
Ok, as of the last commit, I am able to install every
I added FIXME comments to these packages. Someone can fix them as they need them. Ok, last step is to try activating all of these packages. We'll see what conflicts we find! |
| # These dependencies breaks concretization | ||
| # See https://github.com/LLNL/spack/issues/2793 | ||
| # depends_on('py-configparser', when='^python@:3.3.999', type=('build', 'run')) # noqa | ||
| # depends_on('py-enum34', when='^python@:3.1.999', type=('build', 'run')) |
There was a problem hiding this comment.
I believe you can use [email protected] here; [email protected] acts like a set that also matches [email protected]. Not sure where else it happen in this PR (and this one's in a comment); but something to look out for.
Are you able to grep through and fix for places where .999 occurs?
There was a problem hiding this comment.
I don't believe that's the case. You need to use a range. 3.3 only matches 3.3, so if you tried to install something that depends on [email protected], it will try to fetch python-3.3.tar.gz, which won't exist.
|
|
||
| extends('python') | ||
|
|
||
| depends_on('[email protected]:2.7.999,3.3:') |
There was a problem hiding this comment.
Should be depends_on('[email protected],3.3:').
There was a problem hiding this comment.
This won't work. There is no Python version 2.7, so it will never be matched. Try editing it and running spack spec.
|
@adamjstewart A BIG THANKS! This is the kind of detail work that moves a project from "sort of works" to "really usable." I agree, secret downloads by |
|
Wouldn't this work? I saw the code to do exactly that in `version.py`.
depends_on('py-enum34', when='^python@:3.1', type=('build', 'run'))
…On Thu, Jan 12, 2017 at 12:38 PM, Adam J. Stewart ***@***.***> wrote:
***@***.**** commented on this pull request.
------------------------------
In var/spack/repos/builtin/packages/py-flake8/package.py:
> @@ -57,11 +57,12 @@ class PyFlake8(Package):
# mccabe >= 0.2.1, < 0.5
***@***.***:0.4.0', ***@***.***', type=('build', 'run'))
- depends_on('py-configparser', when='^python@:3.3.999', type=('build', 'run'))
- depends_on('py-enum34', when='^python@:3.1.999', type=('build', 'run'))
+ # These dependencies breaks concretization
+ # See #2793
+ # depends_on('py-configparser', when='^python@:3.3.999', type=('build', 'run')) # noqa
+ # depends_on('py-enum34', when='^python@:3.1.999', type=('build', 'run'))
I don't believe that's the case. You need to use a range. 3.3 only
matches 3.3, so if you tried to install something that depends on
***@***.***, it will try to fetch python-3.3.tar.gz, which won't exist.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#2778>, or mute the thread
<https://github.com/notifications/unsubscribe-auth/AB1cd_bGiAmTUELubYqVBIpEUNPEz2ZSks5rRmT7gaJpZM4LdxP6>
.
|
|
Would this work?
depends_on('[email protected]:2.7,3.3:')
…On Thu, Jan 12, 2017 at 12:41 PM, Adam J. Stewart ***@***.***> wrote:
***@***.**** commented on this pull request.
------------------------------
In var/spack/repos/builtin/packages/py-ipykernel/package.py:
> @@ -42,15 +42,10 @@ class PyIpykernel(Package):
version('4.1.1', '51376850c46fb006e1f8d1cd353507c5')
version('4.1.0', '638a43e4f8a15872f749090c3f0827b6')
- extends('python')
-
***@***.***:2.7.999,3.3:')
This won't work. There is no Python version 2.7, so it will never be
matched. Try editing it and running spack spec.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#2778>, or mute the thread
<https://github.com/notifications/unsubscribe-auth/AB1cd59rU06xWXKyhfkBXAD4aSYYg6V9ks5rRmXFgaJpZM4LdxP6>
.
|
depends_on('py-enum34', when='^python@:3.1', type=('build', 'run'))This is valid, although it doesn't solve the problem reported in #2793. depends_on('[email protected]:2.7,3.3:')This does not work, I've tried it before. It complains about a range that starts and stops at the same version. |
|
Arrgh. It sounds like the version range stuff is still a WIP. Thanks
again for all this detail work.
…On Thu, Jan 12, 2017 at 12:56 PM, Adam J. Stewart ***@***.***> wrote:
depends_on('py-enum34', when='^python@:3.1', type=('build', 'run'))
This is valid, although it doesn't solve the problem reported in #2793
<#2793>.
***@***.***:2.7,3.3:')
This does not work, I've tried it before. It complains about a range that
starts and stops at the same version.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#2778 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AB1cd2gNykJXbx9dUbiqCR8Tv1QnwZj-ks5rRmk4gaJpZM4LdxP6>
.
|
|
GitHub doesn't allow me to approve this PR because I'm its "author." But yes, I do approve it! 👍 |
|
Haha, I "approved" it for you |
|
I wish there was an easier way to reassign ownership. |
|
@tgamblin I recommend this gets merged ASAP. It's a BIG change. And without a merge, new Python packages continue to come in that will have to be fixed after-the-fact. |
|
I tried rebasing this to convert the latest new Python packages, figuring it would be easy since there are no conflicts, but it didn't work. @citibeth I'm not sure how you went about rebasing this, but I don't think it can be rebased easily anymore. It still asks me to resolve conflicts for things like the nolink conversion that I thought you had worked out. @tgamblin This needs to be merged or else I'm no longer going to support it. It has become too difficult to maintain. Basically, if there ever becomes any conflicts with existing files, I can't do anything about it anymore. |
|
I rebased the last one using black magic and ill-defined procedures. I
will take a look at it...
…On Sun, Jan 15, 2017 at 8:39 AM, Adam J. Stewart ***@***.***> wrote:
I tried rebasing this to convert the latest new Python packages, figuring
it would be easy since there are no conflicts, but it didn't work.
@citibeth <https://github.com/citibeth> I'm not sure how you went about
rebasing this, but I don't think it can be rebased easily anymore. It still
asks me to resolve conflicts for things like the nolink conversion that I
thought you had worked out. @tgamblin <https://github.com/tgamblin> This
needs to be merged or else I'm no longer going to support it. It has become
too difficult to maintain. Basically, if there ever becomes any conflicts
with existing files, I can't do anything about it anymore.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#2778 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AB1cd3sn7kEhpdgU8oitPoa28vQ9ta9mks5rSiGegaJpZM4LdxP6>
.
|
|
No need to take a look. As long as we can get this merged soon, it shouldn't be a problem, I hope. |
- Add a PythonPackage class with build system support. - Support build phases in PythonPackage - Add a custom sanity check for PythonPackages - Get rid of nolink dependencies in python packages - Update spack create to use new PythonPackage class - Port most of Python packages to new PythonPackage class - Conducted a massive install and activate of Python packages. - Fixed bugs introduced by install and activate. - Update API docs on PythonPackage
fd61ae1 to
25869a0
Compare
|
@adamjstewart @citibeth this wasn't so much rebased as merged. It has two bases at different points in the history of In the future, if you combine two branches like this, it's probably best to rebase one onto develop, then rebase the other onto that if you hope to continue rebasing on OTOH, you can also just keep merging It looks like most of this work is @adamjstewart's, but if I squash with GitHub, the commit shows up as authored by the PR submitter. So, I just squashed the result of the merge manually and force-pushed a single commit by @adamjstewart. @adamjstewart: Can you review these changes and make sure they work and look ok? There was only one minor conflict in |
|
@adamjstewart: note that you should still be able to push commits on top of this if you need to change anything. |
|
OTOH, you can also just keep merging develop into your branch instead of
bothering to rebase. Since we squash anyway, continuously merging develop
and continuously rebasing on develop will end up looking the same in the
end.
I think that's the best approach here.
It looks like most of this work is @adamjstewart
<https://github.com/adamjstewart>'s, but if I squash with GitHub, the
commit shows up as authored by the PR submitter. So, I just squashed the
result of the merge manually and force-pushed a single commit by
@adamjstewart <https://github.com/adamjstewart>.
I agree, @adamjstewart should get the credit.
|
|
Everything looks good to me! Thanks for getting this in @tgamblin. And thanks to @citibeth for helping get this rebased. Even if it was hacky, it certainly saved me a lot of time. Now I'll have to start thinking about the next steps for the PythonPackage class. I want to add import tests to make sure the module can actually be imported after installation. Any requests aside from that? |
|
Any requests aside from that?
@adamjstewart While #2778 was being prepared, a few packages that SHOULD
use PythonPackage but didn't slipped through. Probably best to chase them
down and convert them to PythonPackage now too (it's only a handful, if I
recall). Then we can set the standard that Python packages use
PythonPackage, with a few exceptions of course.
I want to add import tests to make sure the module can actually be
imported after installation.
Good idea.
…On Tue, Jan 17, 2017 at 8:50 AM, Adam J. Stewart ***@***.***> wrote:
Everything looks good to me! Thanks for getting this in @tgamblin
<https://github.com/tgamblin>. And thanks to @citibeth
<https://github.com/citibeth> for helping get this rebased. Even if it
was hacky, it certainly saved me a lot of time.
Now I'll have to start thinking about the next steps for the PythonPackage
class. I want to add import tests to make sure the module can actually be
imported after installation. Any requests aside from that?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#2778 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AB1cd2a1vQvwqpsmTZ69pGpOIBz7_HB5ks5rTMcwgaJpZM4LdxP6>
.
|
Replaces #2709