Skip to content

Bootstrap environment modules optimizing for generic architectures#13105

Merged
tgamblin merged 2 commits intospack:developfrom
alalazo:fixes/bootstrap_environment_modules
Oct 21, 2019
Merged

Bootstrap environment modules optimizing for generic architectures#13105
tgamblin merged 2 commits intospack:developfrom
alalazo:fixes/bootstrap_environment_modules

Conversation

@alalazo
Copy link
Copy Markdown
Member

@alalazo alalazo commented Oct 9, 2019

fixes #13073

Since #3206 was merged bootstrapping environment-modules was using the architecture of the current host or the best match supported by the default compiler. The former case is an issue since shell integration was looking for a spec targeted at the host microarchitecture.

This commit fixes the problem by:

  1. Bootstrapping an en modules targeted at generic architectures
  2. Looking for generic targets in shell integration scripts

fixes spack#13073

Since spack#3206 was merged bootstrapping environment-modules was using
the architecture of the current host or the best match supported
by the default compiler. The former case is an issue since shell
integration was looking for a spec targeted at the host
microarchitecture.

This commit fixes the problem by:

1. Bootstrapping an en modules targeted at generic architectures
2. Looking for generic targets in shell integration scripts
@alalazo alalazo added platform-support modules shell-support bugfix Something wasn't working, here's a fix labels Oct 9, 2019
@alalazo
Copy link
Copy Markdown
Member Author

alalazo commented Oct 9, 2019

@bassenj Can you confirm this fixes the issue for you?

@bassenj
Copy link
Copy Markdown

bassenj commented Oct 9, 2019

$ spack find
==> 3 installed packages
-- linux-ubuntu19.04-x86_64 / [email protected] -------------------------
[email protected]  [email protected]  [email protected]
$ spack --print-shell-vars modules
_sp_sys_type='linux-ubuntu19.04-zen2'
_sp_compatible_sys_types='linux-ubuntu19.04-zen2:linux-ubuntu19.04-zen:linux-ubuntu19.04-x86_64'
_sp_tcl_roots='/tests/spack/share/spack/modules'
_sp_lmod_roots='/tests/spack/share/spack/lmod'
_sp_module_prefix='/tests/spack/opt/spack/linux-ubuntu19.04-x86_64/gcc-8.3.0/environment-modules-4.3.1-6fxrie7v4tfkvtbqt24jt6ytyofpmm36'

spack load works as expected. However, my architecture is recognized as generic x86_64 again. Is that the expected behavior now?

@alalazo
Copy link
Copy Markdown
Member Author

alalazo commented Oct 9, 2019

@bassenj Your architecture is recognized as zen2 but this PR bootstraps environment-modules for generic architectures and uses that. The rationale is not to chase compiler support for every machine for a software that doesn't need the latest instruction sets to be useful.

User docs contains more details on specific microarchitecture support if you're interested.

@alalazo
Copy link
Copy Markdown
Member Author

alalazo commented Oct 9, 2019

To reviewers: I added an additional job in Travis and put it into the first stage to make it fail fast in case it was not working correctly. If you think it belongs somewhere else this can be moved around and / or the corresponding stage can be renamed.

@bassenj
Copy link
Copy Markdown

bassenj commented Oct 9, 2019

Ok, I see, it is only falling back for bootstrapping and the bash functions and zen vs. zen2 depends on my compiler.

$ spack install python
...
$ spack find
==> 22 installed packages
-- linux-ubuntu19.04-x86_64 / [email protected] -------------------------
[email protected]  [email protected]  [email protected]

-- linux-ubuntu19.04-zen / [email protected] ----------------------------
[email protected]    [email protected]  [email protected]  [email protected]   [email protected]  [email protected]  [email protected]  [email protected]   [email protected]  [email protected]
[email protected]  [email protected]  [email protected]      [email protected]  [email protected]    [email protected]     [email protected]   [email protected]  [email protected]

$ spack install zlib target=zen2
==> Error: cannot produce optimized binary for micro-architecture 'zen2' with [email protected] [supported compiler versions are 9:]

There is a duplicate zlib for both generic and specific architecture. Out of curiosity, I installed environment-modules again and started a new terminal which sources setup-env.sh. Now, module uses the most specific architecture:

$ type module
module is a function
module ()
{
    eval `/tests/spack/opt/spack/linux-ubuntu19.04-zen/gcc-8.3.0/environment-modules-4.3.1-lbrd2b7vesqpvgaxx4fkdqgya557fgpk/bin/modulecmd bash $*`
}

Would it be possible to have another stage during bootstrapping that installs environment-modules for the specific architecture and drops the generic version again?

@tgamblin tgamblin merged commit 3d77ecd into spack:develop Oct 21, 2019
@alalazo alalazo deleted the fixes/bootstrap_environment_modules branch October 21, 2019 18:20
jrmadsen pushed a commit to jrmadsen/spack that referenced this pull request Oct 30, 2019
…pack#13105)

fixes spack#13073

Since spack#3206 was merged bootstrapping environment-modules was using the architecture of the current host or the best match supported by the default compiler. The former case is an issue since shell integration was looking for a spec targeted at the host microarchitecture.

1. Bootstrap an env modules targeted at generic architectures
2. Look for generic targets in shell integration scripts
3. Add a new entry in Travis to test shell integration
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bugfix Something wasn't working, here's a fix modules shell-support

Projects

None yet

Development

Successfully merging this pull request may close these issues.

zen vs zen2 inconsitency breaks modules() in spack load

4 participants