xsdk: Very preliminary xsdk.py definition for installing xSDK packages by BarrySmith · Pull Request #2035 · spack/spack

BarrySmith · 2016-10-17T04:12:10Z

I don't have a clue what I am doing so all input welcome. @tgamblin @jwillenbring @sarich

Please read the questions/comments in the changes.

Funded-by: IDEAS
Project: IDEAS/xSDK
Time: 4 hours
Thanks-to: Todd Gamblin [email protected]

davydden · 2016-10-17T05:38:38Z

var/spack/repos/builtin/packages/petsc/package.py

+    #   @git:tag_or_commit_hash using maybe
+    #version('*', git='https://bitbucket.org/petsc/petsc.git')
+    #   then users could do petsc@master or [email protected] or petsc@11bc23.. ?
+    version('master', git='https://bitbucket.org/petsc/petsc.git')


if you use master, in comparators for versions the following won't work: master > 3.7.2 and alike. For now you should name it develop to behave correctly. I have a PR #1983 which introduce other aliases for infinity-like versions, i.e. those which are greater than any given release, but there is a lot of discussion around Version and different opinions.

ps. The proper comparision is important if packages write depends_on([email protected]: (open range) or depends_on(petsc@:3.6.4) (again open range).

for a record, here are the unit tests for Version class which reflect improvements i suggest https://github.com/LLNL/spack/pull/1983/files#diff-f3fb0d9f5acb1def55e4b896c30c8acfR98

davydden · 2016-10-17T05:39:34Z

var/spack/repos/builtin/packages/petsc/package.py

+    # could spack be smart enough to know that master is greater than or equal to the 
+    #    to any release and thus not need the line below ? Or is there a better way to handle this?
+    #    if I don't add the line below and build with petsc master, it uses [email protected] which fails
+    depends_on('[email protected]:', when='@master:+superlu-dist+mpi')


you don't need this line if you use develop, because develop will satisfy @3.7:.

could spack be smart enough to know that master is greater than or equal to the

Spack already does this, but for develop only, see my comment above.

btw, you don't need : here, @x.y.z: == [x.y.z, +\infty)

tgamblin · 2016-10-17T05:30:17Z

var/spack/repos/builtin/packages/petsc/package.py

+    #   be a way to describe the git tag/hash in the spack spec such a
+    #   @git:tag_or_commit_hash using maybe
+    #version('*', git='https://bitbucket.org/petsc/petsc.git')
+    #   then users could do petsc@master or [email protected] or petsc@11bc23.. ?


You can provide tag versions like this:

version('1.5', git='<URL>', tag='<tag>')

Same for commit and branch (see fetching from VCS repositories).

Making it automatic would be nice, but keep in mind that Spack is supposed to work behind an air gap -- so the package can't expect a connection to the git server. The versions have to be enumerated here so the Spack can know what to fetch if it's pulling from an https or filesystem mirror.

tgamblin · 2016-10-17T05:39:27Z

var/spack/repos/builtin/packages/petsc/package.py

    depends_on('[email protected]:', when='@3.7:+superlu-dist+mpi')
+    # could spack be smart enough to know that master is greater than or equal to the 
+    #    to any release and thus not need the line below ? Or is there a better way to handle this?
+    #    if I don't add the line below and build with petsc master, it uses [email protected] which fails


There is a huge discussion on this if you want to weigh in. FWIW, we currently do this for develop.

My issue w/this is that develop (or master) is newer now but if we write that into the provenance and compare against it in the future, it's probably wrong (i.e., the installed develop version will one day be older than the most recent version). My idea was to be smart enough to figure out what commit/recently tagged version it is actually at and write that into the provenance, so we can compare later.

huge discussion is here #1975, the other is just another suggestion.

My idea was to be smart enough to figure out what commit/recently tagged version it is actually at and write that into the provenance, so we can compare later.

AFAIK won't always work: #1975 (comment) 😄

tgamblin · 2016-10-17T05:41:49Z

var/spack/repos/builtin/packages/xsdk/package.py

+    variant('trilinos', default=True, description='Enables turning off building Trilinos for testing; since it takes so long to build')
+
+    # we actually want 2.11.1 but that isn't available in the Spack file
+    depends_on('[email protected]~internal-superlu')


You probably eventually want:

depends_on('[email protected]~internal-superlu', when='@0.1.0')

Then you can add another depends_on for Hypre for 0.2.0 when it comes out.

I have ideas about making this kind of thing easier to specify (some kind of version matrix literal) but haven't implemented them yet.

tgamblin · 2016-10-17T05:43:50Z

var/spack/repos/builtin/packages/xsdk/package.py

+
+    # We need a way to pass -DUSE_XSDK_DEFAULTS=YES into Trilinos
+    # How do we propagate debug flag to all depends on packages ?
+    #   If I just do spack install xsdk+debug will that propogate it down?


For this you could just add an xsdk variant to Trilinos that adds the option, and depend on trilinos+xsdk... blahblah.

tgamblin · 2016-10-17T05:45:02Z

var/spack/repos/builtin/packages/xsdk/package.py

+    depends_on('[email protected]~internal-superlu')
+
+    # we actually want git commit 0b5369f but that isn't supported by the Spack file
+    depends_on('[email protected]')


You can add this to superlu:

version('xsdk-0.1.0', git='<url>', commit='0b5369f')

Would it be crazy to ask them to tag that?

Ok, added this but had to commit it out because it looks like commit of Superlu_dist no longer works with spack.

BarrySmith · 2016-10-18T03:10:52Z

Thanks for the comments. Aside from "add the xsdk variant for Trilinos" I think I've done what was suggested.

Also started to use the develop version to other xsdk packages; got it working on linux.

davydden

looks good apart from minor issues.

davydden · 2016-10-18T04:54:38Z

var/spack/repos/builtin/packages/petsc/package.py

    depends_on('hypre~internal-superlu', when='+hypre+mpi~complex')
-    depends_on('superlu-dist@:4.3', when='@:3.6.4+superlu-dist+mpi')
-    depends_on('[email protected]:', when='@3.7:+superlu-dist+mpi')
+    depends_on('superlu-dist@:4.3', when='@3.6.4+superlu-dist+mpi')


this is probably a typo, unless petsc 3.6.3 and below can't be compiled with superlu-dist@:4.3

davydden · 2016-10-18T04:56:00Z

var/spack/repos/builtin/packages/petsc/package.py

-    depends_on('superlu-dist@:4.3', when='@:3.6.4+superlu-dist+mpi')
-    depends_on('[email protected]:', when='@3.7:+superlu-dist+mpi')
+    depends_on('superlu-dist@:4.3', when='@3.6.4+superlu-dist+mpi')
+    depends_on('[email protected]', when='@3.7:+superlu-dist+mpi')


this is probably a typo, unless [email protected]: would not work with [email protected] and 5.1.1

p.s. I am adding those in #2038

davydden · 2016-10-18T04:57:20Z

var/spack/repos/builtin/packages/superlu-dist/package.py

    homepage = "http://crd-legacy.lbl.gov/~xiaoye/SuperLU/"
    url = "http://crd-legacy.lbl.gov/~xiaoye/SuperLU/superlu_dist_4.1.tar.gz"

+    version('xsdk-0.1.0', git='https://github.com/xiaoyeli/superlu_dist', commit='0b5369f')


maybe call it develop for time being, otherwise xsdk-0.1.0 will always compare less-than with any other released version. In other words, it will not satisfy conditions like depends_on([email protected]:).

This might be ok -- xsdk-0.1.0 is unlikely to be requested by packages other than xsdk. Although it would be nice if the xsdk versions of things were also officially tagged releases of their respective projects so we could avoid things like this.

@tgamblin : How about I strip multiple infinity-like versions from #1983 but keep the rest, it would allow you write here develop.xsdk.0.1.0 and it will behave correctly w.r.t comparision with 5.0.0?

point being is that you can't build [email protected]: with [email protected] because petsc depends_on('[email protected]:', when='@3.7:+superlu-dist+mpi')

davydden · 2016-10-18T04:58:26Z

var/spack/repos/builtin/packages/xsdk/package.py

+
+    # we actually want git commit 0b5369f but that isn't supported by the Spack file
+    #depends_on('[email protected]')
+    # the spack build fails on the above so for now just use the previous release


maybe the reasons is exactly because of version comparision. It can't find the graph of packages which would satisfy all the constraints.

tgamblin

LGTM aside from the superlu version @davydden mentioned, and the failing flake8 tests.

@BarrySmith: you can run the style checks locally with share/spack/qa/run-flake8-tests, or just see the travis build details.

adamjstewart · 2016-10-18T16:56:28Z

@BarrySmith See http://spack.readthedocs.io/en/latest/contribution_guide.html#flake8-tests for additional details on getting started with flake8 testing.

BarrySmith · 2016-10-19T21:34:25Z

Replaced with pull request #2058

BarrySmith mentioned this pull request Oct 17, 2016

Extending a preexisting package #2007

Closed

davydden reviewed Oct 17, 2016

View reviewed changes

tgamblin reviewed Oct 17, 2016

View reviewed changes

tgamblin added the xSDK label Oct 17, 2016

davydden suggested changes Oct 18, 2016

View reviewed changes

tgamblin requested changes Oct 18, 2016

View reviewed changes

tgamblin mentioned this pull request Oct 18, 2016

petsc: minor fixes... #2036

Merged

BarrySmith closed this Oct 19, 2016

BarrySmith mentioned this pull request Oct 19, 2016

xsdk: Very preliminary xsdk.py definition for installing xSDK packages #2058

Closed

Conversation

BarrySmith commented Oct 17, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

BarrySmith commented Oct 18, 2016

Uh oh!

davydden left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

davydden Oct 18, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tgamblin left a comment

Choose a reason for hiding this comment

Uh oh!

adamjstewart commented Oct 18, 2016

Uh oh!

BarrySmith commented Oct 19, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

davydden Oct 18, 2016 •

edited

Loading