Skip to content

various python packages#2320

Merged
tgamblin merged 2 commits intospack:developfrom
krafczyk:fix/when_variants_plus_dependency
Jan 6, 2017
Merged

various python packages#2320
tgamblin merged 2 commits intospack:developfrom
krafczyk:fix/when_variants_plus_dependency

Conversation

@krafczyk
Copy link
Copy Markdown
Contributor

This branch contains a version of the package py-tornado which exposes the problem with the when clauses discussed in #2310.

depends_on('py-certifi')
depends_on('py-backports-abc', when='^python@:2')
depends_on('py-singledispatch', when='^python@:2')
depends_on('py-futures', when='+futures^python@:2')
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i am not sure, but the :2 could be [...,2), i.e. not including 2. whereas @2: is including 2, i.e. [2,...).

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ranges are inclusive, so this is correct.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, did not know that.

# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
##############################################################################
#
# This is a template package file for Spack. We've put "FIXME"
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i guess you forgot to remove these



class PyFutures(Package):
"""FIXME: Put a proper description of your package here."""
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and modify docstring

class PyFutures(Package):
"""FIXME: Put a proper description of your package here."""

# FIXME: Add a proper url for your package's homepage here.
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and add an url

# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
##############################################################################
#
# This is a template package file for Spack. We've put "FIXME"
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same here.



class PyMonotonic(Package):
"""FIXME: Put a proper description of your package here."""
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

here

class PyMonotonic(Package):
"""FIXME: Put a proper description of your package here."""

# FIXME: Add a proper url for your package's homepage here.
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and here

# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
##############################################################################
#
# This is a template package file for Spack. We've put "FIXME"
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and here



class PySingledispatch(Package):
"""FIXME: Put a proper description of your package here."""
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

here

class PySingledispatch(Package):
"""FIXME: Put a proper description of your package here."""

# FIXME: Add a proper url for your package's homepage here.
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

here

@davydden davydden changed the title Fix/when variants plus dependency various python packages Nov 12, 2016
@davydden
Copy link
Copy Markdown
Member

i edited the title, feel free to adjust it if needed.

extends('python')
depends_on('py-setuptools', type='build')
depends_on('py-vcversioner')
depends_on('py-functools32')
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are a couple other optional dependencies that could be added, but I'm not sure if it really matters:

extras_require = {                                                              
    "format" : ["rfc3987", "strict-rfc3339", "webcolors"],                      
    ":python_version=='2.6'": ["argparse", "repoze.lru"],                       
    ":python_version=='2.7'": ["functools32"],                                  
}


extends('python')
depends_on('py-setuptools', type='build')
depends_on('curl')
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you want to be more specific about the dependency versions, they are:

Requirements                                                                    
------------                                                                    

- Python 2.6, 2.7 or 3.1 through 3.5.                                           
- libcurl 7.19.0 or better.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in 1ae60bb.

extends('python')

depends_on('py-setuptools', type='build')
depends_on('py-six')
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

py-ordereddict is also required for Python versions less than 2.7.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in a111742.

depends_on('py-futures', when='+futures^python@:2')
depends_on('py-monotonic', when='+monotonic^python@:2')
depends_on('py-pycurl', when='+curl')
depends_on('py-twisted', when='+twisted')
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The setup.py doesn't mention integrating with py-futures through py-twisted. Where do they come from?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added those since they're listed as optional dependencies in the Arch py-tornado package. I'm looking into it.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you look at this page: http://www.tornadoweb.org/en/stable/index.html They list these packages under prerequisites as 'optional'. I'll leave them off for now since I just want to get things working.

depends_on('py-setuptools', type='build')
depends_on('py-certifi')
depends_on('py-backports-abc', when='^python@:2')
depends_on('py-singledispatch', when='^python@:2')
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[email protected]: is needed for Python < 3.5
py-singledispatch and py-certifi are needed for Python < 3.4
py-ordereddict is needed for Python < 2.7
py-backports is needed for Python < 2.7.9

See setup.py for details.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Corrected this in d5ed437, Also added backports.ssl_match_hostname.


extends('python')
depends_on('py-setuptools', type='build')
depends_on('py-cython')
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requires [email protected]:, py-py, and py-cffi

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now done in 951c82e


# FIXME: Add a proper url for your package's homepage here.
homepage = "http://www.example.com"
url = "https://pypi.python.org/packages/55/db/97c1ca37edab586a1ae03d6892b6633d8eaa23b23ac40c7e5bbc55423c78/futures-3.0.5.tar.gz"
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For url's like this where the PyPi link contains the hash of the package, things like spack versions and spack checksum won't work. Can you try out the pypi.io link format suggested in #1939 (comment)?

Copy link
Copy Markdown
Contributor Author

@krafczyk krafczyk Nov 16, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've corrected all of the packages I've added. However, there are a number of other packages which need this as well, and there's still the issue of #2335.

@krafczyk
Copy link
Copy Markdown
Contributor Author

krafczyk commented Nov 14, 2016

@adamjstewart @davydden I will work on all of these issues, however the purpose of this branch is to show the when clause errors. My branch features/numerous_py_packages will contain the changes suggested here. I'm creating a pull request for that branch as well.

@davydden
Copy link
Copy Markdown
Member

@krafczyk i would say it's better to keep everything in one branch and since @adamjstewart already commented a lot, it would be easier if you just push your changes here. I don't see a reason for another branch with the same content.

@krafczyk krafczyk force-pushed the fix/when_variants_plus_dependency branch from 8ade5b2 to 2d5aa0c Compare November 15, 2016 21:49
@davydden
Copy link
Copy Markdown
Member

@krafczyk : looking at commits, something is wrong here (same commits have different hashes and are duplicated). I suggest you (i) reset your local branch to 7d18593; (ii) rebase on top of current develop in Spack (don't forget ot fetch it or pull it if it's your local copy); (iii) cherry-pick 3f0a6e1 and 3bf1d9b.

This should give you a clean local branch. If all looks good, you can then git push --force to your remote on GitHub. If something went south, you can always git reset --hard to the remote branch on Github.

@krafczyk krafczyk force-pushed the fix/when_variants_plus_dependency branch from 15ffe16 to a111742 Compare November 16, 2016 20:49
@krafczyk
Copy link
Copy Markdown
Contributor Author

@davydden : I've done what you suggested. I'll pay closer attention next time.

@davydden
Copy link
Copy Markdown
Member

@krafczyk thanks, looks nice now 👍

@krafczyk
Copy link
Copy Markdown
Contributor Author

@adamjstewart, @davydden, @citibeth, @jppelteret: I've nearly finished adding everything necessary for the jupyter-notebook to work. Now I'm running into a strange problem having to do with npm.

Since I'm not that familiar with npm, perhaps one of you can help me out here.

When I try to install py-jupyter_notebook from 0d36e3d, I get the output located at this pastebin: http://pastebin.com/rt1Ghv4Z

The log file from the build is at this pastebin: http://pastebin.com/fZR4a8z1

So, while the correct node is used along with the correct npm, I'm getting weird access errors, and npm is trying to write to my home directory. I would think ideally that anything npm installs ought to belong to the py-jupyter_notebook prefix. I'm not sure how to make that happen though.

@citibeth
Copy link
Copy Markdown
Member

Sigh, yet another package manager with its own ideas of how to do things
that breaks spack. You need to figure out how to install jupyter without
npm. Or trick npm by setting a fake home directory.
On Nov 17, 2016 4:47 PM, "Matthew Scott Krafczyk" [email protected]
wrote:

@adamjstewart https://github.com/adamjstewart, @davydden
https://github.com/davydden, @citibeth https://github.com/citibeth,
@jppelteret https://github.com/jppelteret: I've nearly finished adding
everything necessary for the jupyter-notebook to work. Now I'm running into
a strange problem having to do with npm.

Since I'm not that familiar with npm, perhaps one of you can help me out
here.

When I try to install py-jupyter_notebook from 0d36e3d
0d36e3d,
I get the output located at this pastebin: http://pastebin.com/rt1Ghv4Z

The log file from the build is at this pastebin:
http://pastebin.com/fZR4a8z1

So, while the correct node is used along with the correct npm, I'm getting
weird access errors, and npm is trying to write to my home directory. I
would think ideally that anything npm installs ought to belong to the
py-jupyter_notebook prefix. I'm not sure how to make that happen though.


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#2320 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AB1cdwm6NjPgvy9S0g7hRhSaOHPDD0sxks5q_MtOgaJpZM4Kwek5
.

@krafczyk
Copy link
Copy Markdown
Contributor Author

Is there a way to set environment variables when installing or using a specific package?
If there is, is there a way to append like what goes on in PATH?

@tgamblin
Copy link
Copy Markdown
Member

@krafczyk: See the section on setup_dependent_environment. You can customize what goes in the build environment of dependents, and in the run environment of things that use a package.

@davydden
Copy link
Copy Markdown
Member

@krafczyk you can also set environment by os.environ['SLEPC_DIR'] = os.getcwd() or alike.

@krafczyk
Copy link
Copy Markdown
Contributor Author

Great!

@tgamblin
Copy link
Copy Markdown
Member

@krafczyk: note that if you modify os.environ inside setup_dependent_environment, that will succeed for modifying the build environment, because setup_dependent_environment runs for every dependency of a package before it is built. However, it won't get things into a package's module file -- run_env does that. spack_env is provided as a convenience so that the API for setting up the build environment is the same as the one for the run environment.

There is perhaps better documentation on this in the module configuration section.

url = "https://github.com/westes/flex/releases/download/v2.6.2/flex-2.6.2.tar.gz"

version('2.6.0', '5724bcffed4ebe39e9b55a9be80859ec')
version('2.5.39', 'e133e9ead8ec0a58d81166b461244fde')
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why removing 2.5.39 and 2.6.0 ? 2.5.39 is need by foam-extend, for example. Removing versions is generally not a good idea.

btw, the old download works fine for me (i checked 2.5.39). Also if that is not needed for python packages, then please create a separate PR for this change.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry about that, I've fixed up flex good and proper this time.

The reason I did that was because I was running into a segfault when building binutils and moving to the latest version of flex (2.6.2) seemed to solve the problem. In addition, flex development has moved to github so It would be good to reflect that change. The md5s have all been updated.

Copy link
Copy Markdown
Contributor Author

@krafczyk krafczyk Nov 19, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

binutils is part of the dependency chain for yt which was my ultimate goal for using jupyter-notebook.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@davydden, I've created #2385 to deal with this commit. I've also rebased this branch and removed the flex commits.

@krafczyk
Copy link
Copy Markdown
Contributor Author

I'd like to announce that jupyter-notebook is now working as expected for me! It took a lot of work, but now I'm able to use yt in jupyter-notebook!

About the npm problem, it turns out that npm was trying to write to the 'cache' which was set to ~/.npm. This can be overridden with the environment variable npm_config_cache, so I've set this to the npm prefix. This means that all npm downloads will go into npm's prefix. This is a kludgy fix, but it seems to work for now.

There are quite a few commits, so I'd like comments from folks.

@citibeth
Copy link
Copy Markdown
Member

The npm thing means that npm downloads cannot be versioned independently of
npm itself or of each other. I suppose the truly right way to do this
(other than installing npm downloads correctly as with the rest of Spack)
would be to put them all the the Jupyter prefix.

On Fri, Nov 18, 2016 at 7:52 PM, Matthew Scott Krafczyk <
[email protected]> wrote:

I'd like to announce that jupyter-notebook is now working as expected for
me! It took a lot of work, but now I'm able to use yt in jupyter-notebook!

About the npm problem, it turns out that npm was trying to write to the
'cache' which was set to ~/.npm. This can be overridden with the
environment variable npm_config_cache, so I've set this to the npm prefix.
This means that all npm downloads will go into npm's prefix. This is a
kludgy fix, but it seems to work for now.

There are quite a few commits, so I'd like comments from folks.


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#2320 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AB1cd9YZ6gUfHmpuOpeHzyv50BMLlBhSks5q_khmgaJpZM4Kwek5
.

@citibeth
Copy link
Copy Markdown
Member

citibeth commented Jan 5, 2017 via email

@@ -0,0 +1,42 @@
##############################################################################
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This one still needs to be renamed.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed.

stdout=subprocess.PIPE)
result_which = process_pipe.communicate()[0]
process_pipe = subprocess.Popen(["whereis", "libtool"],
stdout=subprocess.PIPE)
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You ought to be able to use which('libtool'), but I'm fine with merging this as is.


depends_on('py-setuptools', type='build')
depends_on('[email protected]:2.7.999,3.3:')
depends_on('[email protected]:')
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Last one, I think.

@davydden
Copy link
Copy Markdown
Member

davydden commented Jan 5, 2017

I would not squash it even during the merge as commits have different authorship

@tgamblin
Copy link
Copy Markdown
Member

tgamblin commented Jan 5, 2017

@alalazo:

Does pulling #2748 solve the issue with the tests?

Yep! Perfect!

@tgamblin
Copy link
Copy Markdown
Member

tgamblin commented Jan 5, 2017

@adamjstewart: approve?

@krafczyk
Copy link
Copy Markdown
Contributor Author

krafczyk commented Jan 5, 2017

@tgamblin, I did not know that you could make a commit on my fork without me pulling it. Nifty!

@tgamblin
Copy link
Copy Markdown
Member

tgamblin commented Jan 5, 2017

@krafczyk: There's a checkbox for this when you commit the PR -- it's the allow edits by maintainers one.

jppelteret and others added 2 commits January 5, 2017 10:37
Adds the following packages:
  node-js  py-backports-abc  py-functools32  py-pycurl   py-vcversione
  npm      py-certifi        py-jsonschema   py-tornado  py-zmq

- Added python packages that are dependencies for the Jupyter suite
- Update new python packaages to use extension package install function.
- Added npm and node-js packages
This adds the following packages:

 py-backports-shutil-get-terminal-size  py-nbformat
 py-backports-ssl-match-hostname        py-pathlib2
 py-entrypoints                         py-pickleshare
 py-futures                             py-pip
 py-ipykernel                           py-prompt-toolkit
 py-ipython-genutils                    py-ptyprocess
 py-ipywidgets                          py-simplegeneric
 py-jupyter-client                      py-singledispatch
 py-jupyter-console                     py-terminado
 py-jupyter-core                        py-traitlets
 py-jupyter-notebook                    py-wcwidth
 py-monotonic                           py-widgetsnbextension
 py-nbconvert
@tgamblin tgamblin force-pushed the fix/when_variants_plus_dependency branch from b6edd93 to 126a115 Compare January 5, 2017 18:38
@tgamblin
Copy link
Copy Markdown
Member

tgamblin commented Jan 5, 2017

I squashed this into two commits by @jppelteret and @krafczyk. They now list the added packages:

$ spack pkg diff develop HEAD^
HEAD^:
    node-js  py-backports-abc  py-functools32  py-pycurl   py-vcversioner
    npm      py-certifi        py-jsonschema   py-tornado  py-zmq

$ spack pkg diff HEAD^ HEAD
HEAD:
    py-backports-shutil-get-terminal-size  py-nbformat
    py-backports-ssl-match-hostname        py-pathlib2
    py-entrypoints                         py-pickleshare
    py-futures                             py-pip
    py-ipykernel                           py-prompt-toolkit
    py-ipython-genutils                    py-ptyprocess
    py-ipywidgets                          py-simplegeneric
    py-jupyter-client                      py-singledispatch
    py-jupyter-console                     py-terminado
    py-jupyter-core                        py-traitlets
    py-jupyter-notebook                    py-wcwidth
    py-monotonic                           py-widgetsnbextension

@krafczyk: if the tests pass and you're ok with this we can merge.

@krafczyk
Copy link
Copy Markdown
Contributor Author

krafczyk commented Jan 5, 2017

@tgamblin I'm testing that things still work for me as I expect. I'll let you know in a bit.

@tgamblin
Copy link
Copy Markdown
Member

tgamblin commented Jan 5, 2017

@krafczyk: Sounds good. If you need to revert, the tip before rebasing was b6edd93. The only diff between that and the two squashed commits above is that the squashed ones are based on develop with @alalazo's test fix.

$ git co fix/when_variants_plus_dependency 
Switched to branch 'fix/when_variants_plus_dependency'
Your branch is up-to-date with 'krafczyk/fix/when_variants_plus_dependency'.
$ git diff b6edd93
diff --git a/lib/spack/spack/test/conftest.py b/lib/spack/spack/test/conftest.py
index 11127d8..79d9018 100644
--- a/lib/spack/spack/test/conftest.py
+++ b/lib/spack/spack/test/conftest.py
@@ -122,7 +122,6 @@ def builtin_mock(repo_path):
 def refresh_builtin_mock(builtin_mock, repo_path):
     """Refreshes the state of spack.repo"""
     # Get back the real repository
-    spack.repo.swap(builtin_mock.real)
     mock_repo = copy.deepcopy(repo_path)
     spack.repo.swap(mock_repo)
     return builtin_mock

@krafczyk
Copy link
Copy Markdown
Contributor Author

krafczyk commented Jan 5, 2017

@tgamblin: Everything looks good to me. You can merge when the tests are finished.

@adamjstewart
Copy link
Copy Markdown
Member

@tgamblin @alalazo Any idea why the macOS tests take so much longer?

@alalazo
Copy link
Copy Markdown
Member

alalazo commented Jan 5, 2017

@adamjstewart It's the set-up phase that seems to take a lot (e.g brew update & Co.).

@krafczyk
Copy link
Copy Markdown
Contributor Author

krafczyk commented Jan 5, 2017

@tgamblin tests are done, ready to merge

@tgamblin tgamblin merged commit 8c65030 into spack:develop Jan 6, 2017
@tgamblin
Copy link
Copy Markdown
Member

tgamblin commented Jan 6, 2017

@krafczyk: merged! Thanks a lot to you and @jppelteret!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants