Skip to content

Don't change properties on already-installed packages#5580

Merged
tgamblin merged 6 commits intospack:developfrom
scheibelp:bugfix/dont-change-installed
Oct 5, 2017
Merged

Don't change properties on already-installed packages#5580
tgamblin merged 6 commits intospack:developfrom
scheibelp:bugfix/dont-change-installed

Conversation

@scheibelp
Copy link
Copy Markdown
Member

@scheibelp scheibelp commented Oct 3, 2017

@tgamblin @becker33

Fixes: #5565

This includes a number of edits to address issues where spack concretization attempts to set properties on already-installed specs. This is a WIP at the moment since I haven't thought all the changes through, but I can replicate #5565 and this appears to resolve it.

  • Don't set patches on packages if they have been concretized
  • Don't invoke _concretize_helper on already-concretized packages (which can add variants for example) - this may not strictly be an issue in py-regex SHA-1 Collision #5565 but came up in my debugging. EDIT: this likely shares some logic with Flags and hashes #4939
  • Don't add new dependencies to already-concretized specs: this was the trickiest part since the concretization deconstructs the DAG on each pass, I want to think over my approach here

Copy link
Copy Markdown
Member

@tgamblin tgamblin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@scheibelp: this looks good, though I think it should avoid modifying concrete specs even if they are not installed. Is there a reason that's needed?

if self.name in visited:
return False

if self.concrete and self.package.installed:
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it is sufficient and more correct in all of these cases to just check self.concrete. If the spec is concrete, we shouldn't change it, even if it's not installed. This should also be safe to do, since we only mark things concrete at the end of concretize now (since #5332 was merged)

…nt also need to check if the package is installed
@scheibelp
Copy link
Copy Markdown
Member Author

@scheibelp: this looks good, though I think it should avoid modifying concrete specs even if they are not installed. Is there a reason that's needed?

Most of my checks didn't need this. But the normalize call in the concretization loop sets force=True, so I kept the check for package.installed in _mark_concrete.

@becker33
Copy link
Copy Markdown
Member

becker33 commented Oct 3, 2017

@scheibelp Have you seen whether this interacts nicely with #4939? I can try to merge the two and test if you don't have time to.

@scheibelp
Copy link
Copy Markdown
Member Author

scheibelp commented Oct 3, 2017

@scheibelp Have you seen whether this interacts nicely with #4939? I can try to merge the two and test if you don't have time to.

@becker33 it shouldn't conflict but there may be some redundancy between the two: in particular this PR doing a check in _concretize_helper may be effectively the same as the checks you add to the beginning of each function in 4939 (EDIT) in concretize.py in #4939

@scheibelp
Copy link
Copy Markdown
Member Author

I think it is sufficient and more correct in all of these cases to just check self.concrete. If the spec is concrete, we shouldn't change it, even if it's not installed. This should also be safe to do, since we only mark things concrete at the end of concretize now (since #5332 was merged)

@tgamblin I changed it everywhere except in Spec._mark_concrete

@tgamblin
Copy link
Copy Markdown
Member

tgamblin commented Oct 3, 2017

@scheibelp: looks good to me. I think it would be good if we could get rid of normalize(force=True) but that can wait.

  1. Can you add a test for this? I am not sure of the best way to do it, but it seems like you could:

    1. first make a MockPackage that has no patch
    2. Install it
    3. Update it to require a patch
    4. Try to install another package that depends on the original install of the mock package
  2. Will this properly cause a conflicts when used with incompatible dependencies? i.e., if I don't descend into a concrete dependency and try to merge constraints with it, how do I know it satisfies all the required constraints? Two scenarios:

    1. Suppose that A depends_on B, with a patch, but the installed B has no patch. Will concretization fail?
    2. Suppose that A depends_on B plus a variant, but the installed B doesn't have the variant. Will concretization fail?

@scheibelp
Copy link
Copy Markdown
Member Author

Can you add a test for this?

I can do that - I'll aim for having that ready around 5pm.

That being said MockPackages cannot be installed. This is likely a good candidate for adding a definition to builtin.mock.

Will this properly cause a conflicts when used with incompatible dependencies? i.e., if I don't descend into a concrete dependency and try to merge constraints with it, how do I know it satisfies all the required constraints? Two scenarios:

This should still report conflicts for incompatible dependencies. It still descends everywhere and e.g. constrains deps. The only case where it doesn't descend is if the dependency to merge doesn't exist in spec_deps and the given spec is concrete. That only happens when the dependency is a build dependency that was stripped of the spec when it was read from the install directory.

@tgamblin
Copy link
Copy Markdown
Member

tgamblin commented Oct 3, 2017

This should still report conflicts for incompatible dependencies. It still descends everywhere and e.g. constrains deps. The only case where it doesn't descend is if the dependency to merge doesn't exist in spec_deps and the given spec is concrete. That only happens when the dependency is a build dependency that was stripped of the spec when it was read from the install directory.

Perfect. Can you ensure that the test checks the cases above? I think we should be good then.

@scheibelp
Copy link
Copy Markdown
Member Author

I can do that - I'll aim for having that ready around 5pm.

I'm going to be later than I hoped but still hopefully have tests for this tonight

@scheibelp
Copy link
Copy Markdown
Member Author

@tgamblin

So far I've got a test to make sure patches aren't added to installed specs at ee5e702

If I understand correctly you also want a test to check that if a new spec requests an installed dependency with a conflicting dependency that is still detected.

@tgamblin
Copy link
Copy Markdown
Member

tgamblin commented Oct 4, 2017

If I understand correctly you also want a test to check that if a new spec requests an installed dependency with a conflicting dependency that is still detected.

... with a conflicting constraint (version, variant, etc.).

…s a dependency constraint which conflicts with a requested installed dependency
@scheibelp
Copy link
Copy Markdown
Member Author

If I understand correctly you also want a test to check that if a new spec requests an installed dependency with a conflicting dependency that is still detected.

... with a conflicting constraint (version, variant, etc.).

@tgamblin alright that is added in e3587b4

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

py-regex SHA-1 Collision

3 participants