fix for bluegene-q platform by pramodk · Pull Request #1980 · spack/spack

pramodk · 2016-10-09T16:34:39Z

this will fix #1784 and #1689!

Today I looked into how Cray CNL is implemented and made corresponding changes for bg-q. This is based on my "minimal" understanding of spack internals, so feel free to suggest changes / update the patch.

I built 7-8 packages for bg-q (front-end and back-end) without any issue. I followed below steps:

After this patch you can find compilers. GNU compilers for back-end added manually. My compilers.yaml file looks like:

compilers:
- compiler:
    modules: []
    operating_system: redhat6
    paths:
      cc: /usr/lib64/ccache/gcc
      cxx: /usr/lib64/ccache/g++
      f77: /usr/bin/gfortran
      fc: /usr/bin/gfortran
    spec: [email protected]
- compiler:
    modules: []
    operating_system: redhat6
    paths:
      cc: /opt/ibmcmp/vacpp/bg/12.1/bin/xlc_r
      cxx: /opt/ibmcmp/vacpp/bg/12.1/bin/xlc++
      f77: /opt/ibmcmp/xlf/bg/14.1/bin/xlf_r
      fc: /opt/ibmcmp/xlf/bg/14.1/bin/xlf2008
    spec: [email protected]
- compiler:
    modules: []
    operating_system: CNK
    paths:
      cc: /usr/bin/bgxlc_r
      cxx: /usr/bin/bgxlc++
      f77: /usr/bin/bgxlf_r
      fc: /usr/bin/bgxlf2008
    spec: [email protected]
- compiler:
    modules: []
    operating_system: CNK
    paths:
      cc: /bgsys/drivers/ppcfloor/gnu-linux/bin/powerpc64-bgq-linux-gcc
      cxx: /bgsys/drivers/ppcfloor/gnu-linux/bin/powerpc64-bgq-linux-g++
      f77: /bgsys/drivers/ppcfloor/gnu-linux/bin/powerpc64-bgq-linux-gfortran
      fc: /bgsys/drivers/ppcfloor/gnu-linux/bin/powerpc64-bgq-linux-gfortran
    spec: [email protected]

On BG-Q, MPI is located under:

/bgsys/drivers/ppcfloor/comm/

I added entry into packages.yaml as:

packages:
    mpich:
        paths:
            [email protected]%[email protected] arch=bgq-CNK-powerpc: /bgsys/drivers/ppcfloor/comm/xl
            [email protected]%[email protected] arch=bgq-CNK-powerpc: /bgsys/drivers/ppcfloor/comm/gcc
        buildable: False

(initially I added dummy package for mpi env var setup, but it's not necessary)

To build a package for front-end:

spack install package-xx os=redhat6

To build package for back-end:

spack install package-xx  #OR spack install package-xx os=CNK

Example

parmetis for CNK (by-default), cmake for front-end (i.e. redhat6) :

$ spack spec parmetis ^mpich ^cmake os=redhat6
Input spec
------------------------------
  parmetis
      ^cmake arch=bgq-redhat6-None
      ^mpich

Normalized
------------------------------
  parmetis
      ^[email protected]: arch=bgq-redhat6-None
      ^metis@5:
      ^mpich

Concretized
------------------------------
  [email protected]%[email protected]~debug~gdb+shared arch=bgq-CNK-powerpc
      ^[email protected]%[email protected]~doc+ncurses+openssl~ownlibs~qt arch=bgq-redhat6-powerpc
          ^[email protected]%[email protected] arch=bgq-CNK-powerpc
          ^[email protected]%[email protected] arch=bgq-CNK-powerpc
              ^[email protected]%[email protected] arch=bgq-CNK-powerpc
                  ^[email protected]%[email protected] arch=bgq-CNK-powerpc
          ^[email protected]%[email protected] arch=bgq-CNK-powerpc
          ^[email protected]%[email protected] arch=bgq-CNK-powerpc
              ^[email protected]%[email protected]~python arch=bgq-CNK-powerpc
                  ^[email protected]%[email protected] arch=bgq-CNK-powerpc
              ^lz4@131%[email protected] arch=bgq-CNK-powerpc
              ^[email protected]%[email protected] arch=bgq-CNK-powerpc
              ^[email protected]%[email protected] arch=bgq-CNK-powerpc
              ^[email protected]%[email protected] arch=bgq-CNK-powerpc
                  ^[email protected]%[email protected] arch=bgq-CNK-powerpc
                      ^[email protected]%[email protected]+sigsegv arch=bgq-CNK-powerpc
                          ^[email protected]%[email protected] arch=bgq-CNK-powerpc
          ^[email protected]%[email protected] arch=bgq-CNK-powerpc
      ^[email protected]%[email protected]~debug~gdb~idx64~real64+shared arch=bgq-CNK-powerpc
      ^[email protected]%[email protected]+hydra+pmi~verbs arch=bgq-CNK-powerpc

@tgamblin :

I didn't find CNK version in BG-Q RedBook, so added "version = '1'"
In the parmetis example above, in concretized spec, metis and parmetis are configured for CNK and CMake for redhat6. But why CMake dependencies for CNK? how to force them for redhead6 as well? I have CMake as an external package in packages.yaml but wondering if there is way to force all dependencies of CMake to the same redhat6 (instead of specifying them all on command-line).
If I just build CMake for front-end, I see correct spec:

spack spec cmake os=redhat6
Input spec
------------------------------
  cmake arch=bgq-redhat6-None

Normalized
------------------------------
  cmake arch=bgq-redhat6-None

Concretized
------------------------------
  [email protected]%[email protected]~doc+ncurses+openssl~ownlibs~qt arch=bgq-redhat6-powerpc
      ^[email protected]%[email protected] arch=bgq-redhat6-powerpc
      ^[email protected]%[email protected] arch=bgq-redhat6-powerpc
          ^[email protected]%[email protected] arch=bgq-redhat6-powerpc
              ^[email protected]%[email protected] arch=bgq-redhat6-powerpc
      ^[email protected]%[email protected] arch=bgq-redhat6-powerpc
      ^[email protected]%[email protected] arch=bgq-redhat6-powerpc
          ^[email protected]%[email protected]~python arch=bgq-redhat6-powerpc
              ^[email protected]%[email protected] arch=bgq-redhat6-powerpc
          ^lz4@131%[email protected] arch=bgq-redhat6-powerpc
          ^[email protected]%[email protected] arch=bgq-redhat6-powerpc
          ^[email protected]%[email protected] arch=bgq-redhat6-powerpc
          ^[email protected]%[email protected] arch=bgq-redhat6-powerpc
              ^[email protected]%[email protected] arch=bgq-redhat6-powerpc
                  ^[email protected]%[email protected]+sigsegv arch=bgq-redhat6-powerpc
                      ^[email protected]%[email protected] arch=bgq-redhat6-powerpc
      ^[email protected]%[email protected] arch=bgq-redhat6-powerpc

pramodk · 2016-10-09T16:43:40Z

lib/spack/spack/platforms/bgq.py

@@ -9,9 +11,22 @@ class Bgq(Platform):
    default     = 'powerpc'


@tgamblin : should be powerpc64 instead of just powerpc?

This should be ppc64 to match the output of platform.machine()

On the newer little-endian Power machines it's ppc64le

lee218llnl · 2016-10-10T23:24:50Z

@pramodk I was able to build STAT for the BGQ frontend with this PR, thanks! I haven't tried to build anything for the compute nodes.

tgamblin · 2016-10-11T08:02:48Z

@pramodk: This is awesome. Sorry for not getting back to you sooner and thanks for putting it together. I don't see anything wrong with the approach -- The platform setup looks similar to how Cray is set up. I had been looking to port what our BG/Q users were doing, but they're still on an old version of Spack and I think starting with just this makes more sense. So, I'm trying to get 0.10 stuff done but would love to put this in there.

The concretization issue is because architecture is not handled the same way as compilers in concretize.py. That is an oversight in the platform support implementation that we should probably fix. Look at the way compilers are inherited from ancestors by the concretizer -- that is probably what you also want for the various architecture components.

One place where you may want to adjust that behavior is for build dependencies. I think build dependencies should default to have the architecture of the default frontend for the platform, and likewise they should pick a valid compiler for the frontend. So if you see something like cmake that is only a build dependency, I think you can skip the inheritance and revert to the default frontend.

I think this exposes a potential issue with the dependency model that I didn't think of. If something is a build/link dependency but also needs to be run at build time, then we may need another deptype for that, which would cause Spack to build both a front-end and a back-end version of the package (one to run at build time and one to link with). I think we can hold off on that for now, but it's something to keep in mind.

tgamblin · 2016-10-11T08:07:54Z

@pramodk: one last thing -- I had been meaning to add additional configuration scoping to Spack so that in any config directory you could have a subdirectory with defaults for a specific platform. With that you would be able to add your various compiler settings (which AFAIK are pretty much the same across different BG/Q machines) to etc/spack/defaults/bgq. Then I think this would just work out of the box on BG/Q.

pramodk · 2016-10-11T22:18:42Z

@tgamblin : I agree, build dependencies should default to architecture of front-end. I already see this issue while building some packages (in deep dependency hierarchy, if some package is specified as build dependency then in the current implementation there is no easy way to specify build for front-end architetcure)

Would it be useful to also have "arch" or "os" option in packages.yaml for package? Currently I see only variants.

I will try to look into concretize.py and see how it works. But if time permits and if you can update this PR, it will be great!

pramodk · 2016-10-15T14:30:06Z

I was looking through the concretize_architecture() in concretize.py. I understand how concretize_architecture is called for every spec, how to traverse dependents/dependencies etc. But still not able to put some something concrete to handle build or other dependencies.

It would be great if someone with more understanding of concretizer could update this PR.

tgamblin · 2016-10-15T23:49:32Z

@pramodk: can we make CNK lowercase? Personally I would like the same thing for CNL on the Cray

pramodk · 2016-10-16T00:22:09Z

yeah, I would also prefer lowercase. Initially I made that consistent with Cray CNK. I am changing that.

tgamblin · 2016-11-05T00:25:02Z

@robertdfrench @mpbelhorn: FYI this will change the Cray OS name to lowercase. If that is something that will cause you headaches, let me know.

mpbelhorn · 2016-11-07T13:34:49Z

@tgamblin not a problem for us and I too prefer lowercase.

pramodk · 2016-11-07T13:55:18Z

FYI this will change the Cray OS name to lowercase

@tgamblin : you mean bg-q? because we are just changing cnk and not CNL ?

scheibelp · 2016-11-12T03:41:23Z

build dependencies should default to architecture of front-end

FYI, I am working on #2292 to provide this functionality. It is expected to merge soon: potentially tomorrow, although likely after 2261.

tgamblin · 2016-12-08T09:00:55Z

@pramodk: now that #2292 is merged, I think we're good on this PR. Can you submit a separate PR with the default BG/Q compilers in `etc/spack/defaults/compilers.yaml? That should allow most people to have a decent back-end setup for BG/Q out of the box.

I think it should include these compilers:

/bgsys/drivers/ppcfloor/gnu-linux/bin/powerpc64-bgq-linux-gcc (and friends)
/opt/ibmcmp/vacpp/bg/12.1/bin/xlc_r (and friends)

Although I notice that in your original compilers.yaml, the /opt/ibmcmp/vacpp/bg/12.1/bin/xlc_r is marked as a front-end compiler. Is that XL compiler not the one for the backend (b/c it's bg/12.1)?

pramodk · 2016-12-08T21:38:23Z

Is that XL compiler not the one for the backend (b/c it's bg/12.1)?

yes, that was by mistake!

(i am currently travelling, will see if I can look into default compiler PR over the weekend)

tgamblin · 2016-12-08T21:39:43Z

Thanks @pramodk! Hope you have a good trip

* fix for bluegene-q platform * change powerpc to ppc64 * change CNK to cnk

pramodk commented Oct 9, 2016

View reviewed changes

pramodk mentioned this pull request Oct 9, 2016

spack install hanging on BG/Q #1689

Closed

tgamblin added feature A feature is missing in Spack platform-support labels Oct 11, 2016

tgamblin mentioned this pull request Oct 15, 2016

Platform-specific config scopes #2030

Merged

pramodk added 2 commits October 16, 2016 02:10

fix for bluegene-q platform

35590c4

change powerpc to ppc64

89b7532

pramodk force-pushed the bgq_fix branch from 4858a64 to 89b7532 Compare October 16, 2016 00:13

change CNK to cnk

208263e

sarich added the xSDK label Oct 27, 2016

sarich mentioned this pull request Nov 1, 2016

spec.py _replace_with dictionary error #2162

Closed

pramodk mentioned this pull request Nov 2, 2016

bg-q : spack compiler find adds front-end compilers for cnk #2220

Closed

tgamblin added the hash-change label Nov 5, 2016

pramodk mentioned this pull request Nov 10, 2016

Features/frontend default build #2292

Merged

tgamblin merged commit 29e4214 into spack:develop Dec 8, 2016

kserradell pushed a commit to kserradell/spack that referenced this pull request Dec 9, 2016

fix for bluegene-q platform (spack#1980)

aaa767c

* fix for bluegene-q platform * change powerpc to ppc64 * change CNK to cnk

pramodk mentioned this pull request Dec 11, 2016

Adding default bg-q compilers configuration #2557

Closed

haampie added the hash-change things that will change a lot of hashes label Nov 7, 2024

Conversation

pramodk commented Oct 9, 2016

Example

Uh oh!

pramodk Oct 9, 2016

Choose a reason for hiding this comment

Uh oh!

tgamblin Oct 11, 2016

Choose a reason for hiding this comment

Uh oh!

tgamblin Oct 11, 2016

Choose a reason for hiding this comment

Uh oh!

lee218llnl commented Oct 10, 2016

Uh oh!

tgamblin commented Oct 11, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tgamblin commented Oct 11, 2016

Uh oh!

pramodk commented Oct 11, 2016

Uh oh!

pramodk commented Oct 15, 2016

Uh oh!

tgamblin commented Oct 15, 2016

Uh oh!

pramodk commented Oct 16, 2016

Uh oh!

tgamblin commented Nov 5, 2016

Uh oh!

mpbelhorn commented Nov 7, 2016

Uh oh!

pramodk commented Nov 7, 2016

Uh oh!

scheibelp commented Nov 12, 2016

Uh oh!

tgamblin commented Dec 8, 2016

Uh oh!

pramodk commented Dec 8, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tgamblin commented Dec 8, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

tgamblin commented Oct 11, 2016 •

edited

Loading

pramodk commented Dec 8, 2016 •

edited

Loading