Skip to content

Modules: Deduplicate suffixes but don't sort them.#18351

Merged
sethrj merged 3 commits intospack:developfrom
RemiLacroix-IDRIS:modules_suffixes_order
Sep 8, 2020
Merged

Modules: Deduplicate suffixes but don't sort them.#18351
sethrj merged 3 commits intospack:developfrom
RemiLacroix-IDRIS:modules_suffixes_order

Conversation

@RemiLacroix-IDRIS
Copy link
Copy Markdown
Contributor

@RemiLacroix-IDRIS RemiLacroix-IDRIS commented Aug 28, 2020

PR #14920 changed the behavior to sorting the suffixes. That change is causing some issues with our modules.

foo/X.Y.Z-cuda-mpi becomes foo/X.Y.Z-cuda-mpi for example, which we aren't really happy about because it is causing duplicated modules if we don't pay attention and we cannot have consistent module naming without breaking existing submission scripts.

The documentation (https://spack.readthedocs.io/en/latest/module_file_support.html#selection-by-anonymous-specs) currently states:

Order does matter
The modifications associated with the all keyword are always evaluated first, no matter where they appear in the configuration file. All the other spec constraints are instead evaluated top to bottom.

I think that it makes sense that "modifications" also apply to suffixes. As far as I can tell, the suffixes' ordering is indeed guaranteed to follow the ordering of configuration files (except for all which is considered first, as already noted in the documentation).

Copy link
Copy Markdown
Member

@alalazo alalazo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change in behavior was part of the 0.14.1 Changelog and this PR reverts it. I would advocate for not bouncing back and forth with behavior, especially considering that this was posted at the time. Can you be more specific on the issues that this causes for you? Wondering if it is possible to find an alternative solution.

@RemiLacroix-IDRIS
Copy link
Copy Markdown
Contributor Author

RemiLacroix-IDRIS commented Aug 28, 2020

I am sorry I missed that change earlier but it is hard to keep track, especially when you let some time go by without updating your working Spack install...

Can you be more specific on the issues that this causes for you? Wondering if it is possible to find an alternative solution.

If we refresh an existing module foo/X.Y.Z-mpi-cuda, it will create a duplicated module foo/X.Y.Z-cuda-mpi.

If we create a module for a new product, it will create a module foo/X.Y.Z-cuda-mpi while we may already have foo/I.J.K-mpi-cuda, leading to inconsistent module names.

I cannot think of any solution that would allow forcing the order to -mpi-cuda. Renaming existing module files is not something we would be happy to do as users HATE when that sort of things change as it breaks their submission scripts.

@alalazo alalazo requested a review from sethrj August 28, 2020 11:42
@alalazo
Copy link
Copy Markdown
Member

alalazo commented Aug 28, 2020

Renaming existing module files is not something we would be happy to do as users HATE when that sort of things change as it breaks their submission scripts.

Are you updating in place the Spack that is used for site-wide deployment? Would it be a solution to generate a new tree of modules at another location and provide users with a command to swap between the two?

@RemiLacroix-IDRIS
Copy link
Copy Markdown
Contributor Author

RemiLacroix-IDRIS commented Aug 28, 2020

Are you updating in place the Spack that is used for site-wide deployment?

Yeah, we already did in fact. Unfortunately I missed that issue when updating and I am only noticing it now, a few weeks after the upgrade. :/

Would it be a solution to generate a new tree of modules at another location and provide users with a command to swap between the two?

I don't really think we want to go down that road. To be honest, I think that if this PR is rejected, we will just keep the patch for our install, it will be easier for us.

Additionally I do think it is nice to be able to control the ordering of the suffixes.

@sethrj
Copy link
Copy Markdown
Contributor

sethrj commented Aug 28, 2020

@RemiLacroix-IDRIS I think you might also have missed this comment: at least at the time of that pull request, the ordering of the suffixes was unspecified: python 3.6+ kept them in the order of creation, but other versions of python shuffled them around. So the changeset you've submitted might not even get you the behavior you want.

Indeed, given that the new test I added (https://github.com/spack/spack/pull/14920/files) doesn't change behavior in your PR, I suspect this isn't doing what you want.

@sethrj
Copy link
Copy Markdown
Contributor

sethrj commented Aug 28, 2020

P.S. Is the double-hyphen in your PR description foo/X.Y.Z--cuda-mpi a typo or actually a generated module name? Because that does look like a bug that should be fixed.

@RemiLacroix-IDRIS
Copy link
Copy Markdown
Contributor Author

P.S. Is the double-hyphen in your PR description foo/X.Y.Z--cuda-mpi a typo or actually a generated module name? Because that does look like a bug that should be fixed.

That's a typo (now fixed).

Copy link
Copy Markdown
Contributor

@sethrj sethrj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR needs a test demonstrating the change in behavior. See lib/spack/spack/test/modules/tcl.py . Because the old test for sorted behavior is passing, I believe your change isn't doing what you think it's doing.

@RemiLacroix-IDRIS
Copy link
Copy Markdown
Contributor Author

@RemiLacroix-IDRIS I think you might also have missed this comment: at least at the time of that pull request, the ordering of the suffixes was unspecified: python 3.6+ kept them in the order of creation, but other versions of python shuffled them around. So the changeset you've submitted might not even get you the behavior you want.

You are correct, it seems that there are some remaining dict in the config, although Spack seems to use ordereddict mostly everywhere.

Turns out that in Python 3.6+ dict implementation is ordered so everything works as expected.

I am looking into this to understand whether it is expected or not that dict are used.

Indeed, given that the new test I added (https://github.com/spack/spack/pull/14920/files) doesn't change behavior in your PR, I suspect this isn't doing what you want.

That might not have been what was intended back then but as far as I can tell bar-foo is what is expected both when using the config file order and when sorting.

If I modify the test so that the order is different, I can see that I don't get the expected behavior with Python 2.7 after my change.

@sethrj
Copy link
Copy Markdown
Contributor

sethrj commented Aug 28, 2020

@RemiLacroix-IDRIS Before you go any further, I'd recommend double-checking that retaining the original order actually updates the module names to what you expect from your original module name generation. It's possible that if the extensions are arbitrarily ordered there's nothing that can be done.

@RemiLacroix-IDRIS
Copy link
Copy Markdown
Contributor Author

Before you go any further, I'd recommend double-checking that retaining the original order actually updates the module names to what you expect from your original module name generation. It's possible that if the extensions are arbitrarily ordered there's nothing that can be done.

It does match the original module name generation when using the config file order.

I am looking into this to understand whether it is expected or not that dict are used.

Turns out the problem was in the tests themselves! The configuration file was loaded differently in Spack and in the modules tests. Spack does use ordered dict everywhere, even with Python 2.7!

Copy link
Copy Markdown
Contributor

@sethrj sethrj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me with the added tests (good catch with the unit tests being different from the main code!). @alalazo we may want to post another Spack survey/announcement making sure no one else will be taken by surprise by this change.

@alalazo
Copy link
Copy Markdown
Member

alalazo commented Aug 31, 2020

@sethrj I agree with posting a comment in #modules and merge this PR if no objection arise in 1-2 weeks. Can you post that message?

@RemiLacroix-IDRIS Do you have time to add a few more tests to check that the order of suffixes is the one we expect?

@RemiLacroix-IDRIS
Copy link
Copy Markdown
Contributor Author

Do you have time to add a few more tests to check that the order of suffixes is the one we expect?

@alalazo What do you have in mind exactly? It's what I tried to do with 3362838.

@alalazo
Copy link
Copy Markdown
Member

alalazo commented Aug 31, 2020

@RemiLacroix-IDRIS For instance, if I understand this changeset correctly, maintaining the original configuration for the test we should have bar-foo as a suffix, right? I think we should check both and maybe other edge cases to verify the semantics are correct.

The suffixes' order is defined by the order in which they appear in the configuration file.
spack_yaml.load_config ensures that the configuration is stored in an ordered manner. Without this change, the behavior of the tests did not match Spack's.
@RemiLacroix-IDRIS
Copy link
Copy Markdown
Contributor Author

@alalazo I have extended the tests a bit.

@sethrj
Copy link
Copy Markdown
Contributor

sethrj commented Sep 8, 2020

OK, it's been a week with one positive and no negative feedback. The old (sorted) behavior is easy to do (just sort the keywords in your module file), so it should be easy for anyone to work around if they're surprised by the updated behavior. Merging!

@sethrj sethrj merged commit 92bf949 into spack:develop Sep 8, 2020
@RemiLacroix-IDRIS RemiLacroix-IDRIS deleted the modules_suffixes_order branch September 8, 2020 14:45
@RemiLacroix-IDRIS RemiLacroix-IDRIS restored the modules_suffixes_order branch September 8, 2020 15:19
@RemiLacroix-IDRIS RemiLacroix-IDRIS deleted the modules_suffixes_order branch September 8, 2020 15:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants