Intel prefixes by mgsternberg · Pull Request #7469 · spack/spack

mgsternberg · 2018-03-12T23:24:10Z

I reworked class IntelPackage to overcome what was for me a major barrier to entry in adopting Spack that showed up when attempting to integrate an existing Intel tool chain on my HPC system.

There was a serious conceptual conflict for prefix between versions of Intel packages that were installed under Spack and those brought in via packages.yaml. I am not the only one to have stumbled over this, see e.g. https://groups.google.com/d/msg/spack/x28qlmqPAys/Ewx6220uAgAJ

This is an initial attempt at tackling the problem, having targeted MKL and Intel-MPI as proof-of-concept. I was able to build ABINIT and CP2K with those, the latter having a downright gluttonous list of dependencies. Other Intel packages should be possible to integrate.

The major affected files so far are:

.../build_systems/intel.py
.../intel-mkl/package.py
.../intel-mpi/package.py
.../intel-parallel-studio/package.py

A major goal was elimination of duplicate code. To this end, I pulled up foo_libs() and related methods into the IntelPackage class definition. The following methods should give a flavor of my approach:

def product_dir(self, product_dir_name, version_glob='_2???.*.*[0-9]',
                postfix_dir=''):
    '''Returns the version-specific directory of an Intel product release,
    holding the main product and possibly auxiliary files from other
    products.
    '''
…
def component_dir(self, component=None):
    '''Returns the directory of a product component, appropriate for
    presenting to users in environment variables like MKLROOT and
    I_MPI_ROOT, or the product dir itself (when the component not evident
    from the package name and wasn't specified).
    '''
…
def component_bin_dir(self, component=None, relative=False):
…
def component_lib_dir(self, component=None, relative=False):

The reorganization dramatically shortens .../intel-mkl/package.py, and allows to write, e.g.

def file_to_source(self):
    return join_path(self.component_bin_dir(), 'mklvars.sh')

Other changes are minor and showed up/were prompted by testing:

lib/spack/spack/environment.py
var/spack/repos/builtin/packages/libxc/package.py

Comments and suggestions on my approach are needed and welcome.

…m spack-native install and explain what happens for spack-external installs

…aa76a

…ation.

mgsternberg · 2018-08-14T23:53:43Z

@davydden @adamjstewart @tgamblin
I updated the doc in the present form and addressed the majority of @adamjstewart's review comments from last month. We touched on factoring out the docs, but this would leave this PR without any doc at all which I would think would be worse. Factoring out and re-distributing the docs elsewhere into the doc tree may well be tackled later.

I also left in the perhaps unexpecterd instructions for declaring a "stub" compilers.yaml entry prior to installing intel-parallel-studio. This is related to the otherwise still unresolved issue of unexpected package re-installs, which would be particularly troubling and annoying for that package and for new users, as discussed here before.

mgsternberg · 2018-08-16T02:40:17Z

I added a doc section on how to properly link the MKL using the utilities from class llnl.util.filesystem. This overlaps with, extends, and links to the relevant Packaging Guide section.
Parts might in fact be migrated into the Packaging Guide at some point.

While large parts are not specific to the MKL, the issues came to the fore when MKL is used.

mgsternberg · 2018-08-16T20:59:38Z

@mamelara
As mentioned in today's Telcon, here's the doc section on MKL linkage that I recently added. – Does that help?

mamelara · 2018-08-16T21:06:39Z

So just to clarify, the intel mkl advisor recommends a link line looking like this:

 ${MKLROOT}/lib/intel64/libmkl_scalapack_ilp64.a -Wl,--start-group ${MKLROOT}/lib/intel64/libmkl_intel_ilp64.a ${MKLROOT}/lib/intel64/libmkl_sequential.a ${MKLROOT}/lib/intel64/libmkl_core.a ${MKLROOT}/lib/intel64/libmkl_blacs_intelmpi_ilp64.a -Wl,--end-group -lpthread -lm -ldl

With the key part being this:

-Wl,--start-group ${MKLROOT}/lib/intel64/libmkl_intel_ilp64.a ${MKLROOT}/lib/intel64/libmkl_sequential.a ${MKLROOT}/lib/intel64/libmkl_core.a ${MKLROOT}/lib/intel64/libmkl_blacs_intelmpi_ilp64.a -Wl,--end-group

So I was thinking that something like spec['blas'].ld_flags would return something like that if it is trying to link statically.

We've been running into the issue of getting undefined references since for whatever reason those libraries have circular dependencies on each other.

Obviously, I don't want to hold this PR back any longer and could add the change later if need be but I was wondering if this PR addressed that.

mgsternberg · 2018-08-16T22:11:43Z

@mamelara Can you offer more specifics on the failure? Is it just undefined symbols or inability to link statically in general?

Some thoughts:

static linkage did not come up as issue in this PR, except peripherally for TBB where Intel has published a note explaining the absence of a static TBB.
The Intel packages that provide the MKL offer a +shared variant which is coded along the lines of find_libraries(..., shared=('+shared' in self.spec)). From that, spec['blas'].libs will duly return a list of .a files, though without the -Wl bracketing, and the spack codebase uses --start-group only in a test context, under lib/spack/spack/test/cc.py.

I just realized that spec['blas'].libs.ld_flags as in the doc would in fact mask the distinction between .so and .a because it outputs -L<dir> -l<libname> ... strings.
Spack-built executables do include the MKL paths in their -rpath settings. Assuming you have the Intel libs available on target nodes, doesn't that give you most of the same effect as -static without having to worry about LD_LIBRARY_PATH?
All said, I recall a hackish ancient setup where one of the libs (scalapack or blacs) needed to be given again near the end of the link line. Spack at the moment doesn't do that.

scheibelp · 2018-08-29T18:30:05Z

I would be happy to merge now (after merge conflicts are resolved): @davydden mentioned that this worked for him. And the documentation is great. @adamjstewart had some comments about refactoring but IMO those could be handled in another issue/PR. I also think @mamelara's concerns could be handled in a new issue/PR.

Were you looking for any additional user testing or feedback?

If so, if you resolve the conflicts in the next couple days. I could merge #7540 at which point I assume it would be easy to resolve conflicts which arise from that (since you mention your edits are a superset of that).

adamjstewart · 2018-08-29T18:56:44Z

@adamjstewart had some comments about refactoring but IMO those could be handled in another issue/PR.

Agreed. The semester just started for me so I probably won't have time to review very many Spack PRs. If everyone can build their favorite packages with this branch them I'm fine with the merge.

mgsternberg · 2018-08-29T18:56:52Z

@scheibelp
Good to hear! Yes, I'd be glad to hear from @mamelara, but you're right in handling that later.

Following up, I just saw the conflicts, and will review.

@davydden had a note on #9032, obviating an SPACK_COMPILER_EXTRA_RPATHS modification for MKL here. That's not merged yet, though, so I'll have to leave that RPATH part in.

davydden · 2018-08-29T20:12:13Z

@davydden had a note on #9032, obviating an SPACK_COMPILER_EXTRA_RPATHS modification for MKL here. That's not merged yet, though, so I'll have to leave that RPATH part in.

I agree. I can always adopt #9032 after this PR is merged.

mgsternberg · 2018-08-29T20:31:49Z

@scheibelp
Given the high-profile and thus potentially showstopping role of licensing for the Intel packages, merging #8991 alongside here should be helpful for new users.

Note that there was a strange Travis failure for Python-3.6. The raw log does not mention "intel" so I don't think the failure has to do with that PR itself.

scheibelp · 2018-08-30T04:10:27Z

Thanks for this!

Note that there was a strange Travis failure for Python-3.6. The raw log does not mention "intel" so I don't think the failure has to do with that PR itself.

Agreed - the final message indicates that the build was stalled.

Consolidate prefix calculation logic for intel packages into the IntelPackage class. Add documentation on installing Intel packages with Spack an (alternatively) adding them as external packages in Spack.

mgsternberg added 30 commits February 20, 2018 18:47

reviewed and modularized self.prefix handling, accommodate prefix fro…

657d029

…m spack-native install and explain what happens for spack-external installs

simplify _mkl_libdir calls, rethink mklroot() approach

61b0000

simplify _want_shared calls, reduce diff noise

eec0fd4

FIX: UnboundLocalError for "search"

d7a3926

$prefix/ file analysis to explain rationale; reorder libs to match 82…

153618a

…aa76a

merged upstream/develop

ca1feaa

WIP: rename _mkl_int_suffix to intel64_int_suffix, prep for generaliz…

e368fc6

…ation.

WIP: generalize more - component_bindir()

4174108

WIP: simplify intel64_int_suffix()

edc2bb8

FIX: args to from_sourcing_file()

7dda576

Merge remote branch 'upstream/develop' into intel-prefixes

6c42ca8

Merge remote branch 'upstream/develop' into intel-prefixes

846769a

promote functions and method up into class

d17935f

Merge remote branch 'upstream/develop' into intel-prefixes

2414f02

WIP: cover intel-mkl in parent and child pkg.

ae91426

Merge remote branch 'upstream/develop' into intel-prefixes

9d201cc

WIP; renames to product_dir, component_foo_dir (readability)

a6f9fed

WIP integrate intel-parallel-studio

d66edbb

merge 98949bd and d3ed46e

bf20614

Merge remote branch 'upstream/develop' into intel-prefixes

0f0be9d

re-int. product_dir; setupt_env back to mpi client

699c81a

replace _compiler_{bin,lib}_dir with more flex. methods in parent

2a726f7

review for compat.

a3b5181

Merge remote branch 'upstream/develop' into intel-prefixes

b5ee4e1

Merge remote branch 'upstream/develop' into intel-prefixes

3b331c4

fix presence of setup_dependent_environment()

b714c8f

Merge remote branch 'upstream/develop' into intel-prefixes

b4355f0

review/fix determine_license_type()

ee29e74

flake8

0ab5235

readability

3dd097c

doc test: remove an rST comment

2d3953a

mgsternberg added 3 commits August 15, 2018 11:12

doc: review intro and licensing

906df78

Merge remote branch 'upstream/develop' into intel-prefixes

a5b28a0

doc: new section with tips on using MKL

db0577e

mgsternberg mentioned this pull request Aug 15, 2018

Update instructions to initialize Spack-global license files #8991

Merged

mgsternberg added 2 commits August 15, 2018 20:10

doc: review and better link MKL tips section

b6d2615

Merge remote branch 'upstream/develop' into intel-prefixes

bb7bd64

doc: add section on MKLROOT, bulletize virtual package instructions

5cf0b8e

merged upstream/develop

4870626

doc (minor)

c71260e

scheibelp merged commit a86f22d into spack:develop Aug 30, 2018

davydden mentioned this pull request Sep 10, 2018

petsc: fix compilation with intel #9198

Merged

scheibelp mentioned this pull request Sep 14, 2018

use Package.libs in RPATH with a fallback to prefix.lib/lib64 #9032

Closed

jrood-nrel mentioned this pull request Oct 25, 2018

Intel Parallel Studio writes user environment to module #9639

Closed

alalazo mentioned this pull request Apr 5, 2019

Error building elpa: AttributeError: 'Spec' object has no attribute 'mpicc' #6143

Closed

marcfehling mentioned this pull request May 15, 2022

Quicktests fail with Intel oneAPI suite. dealii/dealii#13477

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Intel prefixes#7469

Intel prefixes#7469
scheibelp merged 221 commits intospack:developfrom
mgsternberg:intel-prefixes

mgsternberg commented Mar 12, 2018

Uh oh!

mgsternberg commented Aug 14, 2018

Uh oh!

mgsternberg commented Aug 16, 2018

Uh oh!

mgsternberg commented Aug 16, 2018 •

edited

Loading

Uh oh!

mamelara commented Aug 16, 2018 •

edited

Loading

Uh oh!

mgsternberg commented Aug 16, 2018 •

edited

Loading

Uh oh!

scheibelp commented Aug 29, 2018

Uh oh!

adamjstewart commented Aug 29, 2018

Uh oh!

mgsternberg commented Aug 29, 2018

Uh oh!

davydden commented Aug 29, 2018

Uh oh!

mgsternberg commented Aug 29, 2018

Uh oh!

scheibelp commented Aug 30, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

12 participants

Conversation

mgsternberg commented Mar 12, 2018

Uh oh!

mgsternberg commented Aug 14, 2018

Uh oh!

mgsternberg commented Aug 16, 2018

Uh oh!

mgsternberg commented Aug 16, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mamelara commented Aug 16, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mgsternberg commented Aug 16, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

scheibelp commented Aug 29, 2018

Uh oh!

adamjstewart commented Aug 29, 2018

Uh oh!

mgsternberg commented Aug 29, 2018

Uh oh!

davydden commented Aug 29, 2018

Uh oh!

mgsternberg commented Aug 29, 2018

Uh oh!

scheibelp commented Aug 30, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

12 participants

mgsternberg commented Aug 16, 2018 •

edited

Loading

mamelara commented Aug 16, 2018 •

edited

Loading

mgsternberg commented Aug 16, 2018 •

edited

Loading