Modules: Deduplicate suffixes but don't sort them.#18351
Modules: Deduplicate suffixes but don't sort them.#18351sethrj merged 3 commits intospack:developfrom
Conversation
alalazo
left a comment
There was a problem hiding this comment.
This change in behavior was part of the 0.14.1 Changelog and this PR reverts it. I would advocate for not bouncing back and forth with behavior, especially considering that this was posted at the time. Can you be more specific on the issues that this causes for you? Wondering if it is possible to find an alternative solution.
|
I am sorry I missed that change earlier but it is hard to keep track, especially when you let some time go by without updating your working Spack install...
If we refresh an existing module If we create a module for a new product, it will create a module I cannot think of any solution that would allow forcing the order to |
Are you updating in place the Spack that is used for site-wide deployment? Would it be a solution to generate a new tree of modules at another location and provide users with a command to swap between the two? |
Yeah, we already did in fact. Unfortunately I missed that issue when updating and I am only noticing it now, a few weeks after the upgrade. :/
I don't really think we want to go down that road. To be honest, I think that if this PR is rejected, we will just keep the patch for our install, it will be easier for us. Additionally I do think it is nice to be able to control the ordering of the suffixes. |
|
@RemiLacroix-IDRIS I think you might also have missed this comment: at least at the time of that pull request, the ordering of the suffixes was unspecified: python 3.6+ kept them in the order of creation, but other versions of python shuffled them around. So the changeset you've submitted might not even get you the behavior you want. Indeed, given that the new test I added (https://github.com/spack/spack/pull/14920/files) doesn't change behavior in your PR, I suspect this isn't doing what you want. |
|
P.S. Is the double-hyphen in your PR description |
That's a typo (now fixed). |
sethrj
left a comment
There was a problem hiding this comment.
This PR needs a test demonstrating the change in behavior. See lib/spack/spack/test/modules/tcl.py . Because the old test for sorted behavior is passing, I believe your change isn't doing what you think it's doing.
You are correct, it seems that there are some remaining Turns out that in Python 3.6+ I am looking into this to understand whether it is expected or not that
That might not have been what was intended back then but as far as I can tell If I modify the test so that the order is different, I can see that I don't get the expected behavior with Python 2.7 after my change. |
|
@RemiLacroix-IDRIS Before you go any further, I'd recommend double-checking that retaining the original order actually updates the module names to what you expect from your original module name generation. It's possible that if the extensions are arbitrarily ordered there's nothing that can be done. |
It does match the original module name generation when using the config file order.
Turns out the problem was in the tests themselves! The configuration file was loaded differently in Spack and in the modules tests. Spack does use ordered dict everywhere, even with Python 2.7! |
7429a9b to
3362838
Compare
sethrj
left a comment
There was a problem hiding this comment.
This looks good to me with the added tests (good catch with the unit tests being different from the main code!). @alalazo we may want to post another Spack survey/announcement making sure no one else will be taken by surprise by this change.
|
@sethrj I agree with posting a comment in #modules and merge this PR if no objection arise in 1-2 weeks. Can you post that message? @RemiLacroix-IDRIS Do you have time to add a few more tests to check that the order of suffixes is the one we expect? |
|
@RemiLacroix-IDRIS For instance, if I understand this changeset correctly, maintaining the original configuration for the test we should have |
The suffixes' order is defined by the order in which they appear in the configuration file.
spack_yaml.load_config ensures that the configuration is stored in an ordered manner. Without this change, the behavior of the tests did not match Spack's.
3362838 to
f6ccf31
Compare
|
@alalazo I have extended the tests a bit. |
|
OK, it's been a week with one positive and no negative feedback. The old (sorted) behavior is easy to do (just sort the keywords in your module file), so it should be easy for anyone to work around if they're surprised by the updated behavior. Merging! |
PR #14920 changed the behavior to sorting the suffixes. That change is causing some issues with our modules.
foo/X.Y.Z-cuda-mpibecomesfoo/X.Y.Z-cuda-mpifor example, which we aren't really happy about because it is causing duplicated modules if we don't pay attention and we cannot have consistent module naming without breaking existing submission scripts.The documentation (https://spack.readthedocs.io/en/latest/module_file_support.html#selection-by-anonymous-specs) currently states:
I think that it makes sense that "modifications" also apply to
suffixes. As far as I can tell, the suffixes' ordering is indeed guaranteed to follow the ordering of configuration files (except forallwhich is considered first, as already noted in the documentation).