bugfix: ensure proper dependency handling for package-only installs by tldahlgren · Pull Request #15197 · spack/spack

tldahlgren · 2020-02-25T02:15:36Z

The recent distributed build PR -- #13100 -- did not check the install status of dependencies when using the --only package option so would refuse to install a package with the claim that it had uninstalled dependencies whether that was the case or not.

This PR adds install status checks for the --only package case.

tldahlgren · 2020-02-25T02:56:37Z

@scottwittenburg Could you try this fix and let me know if it resolves your issue?

tgamblin

@tldahlgren: this looks great, but we need to get the coverage up -- there are some key pieces of the code still not covered by tests.

tldahlgren · 2020-02-25T17:26:10Z

@tldahlgren: this looks great,

Thanks for the confirmation that you're fine with this approach.

but we need to get the coverage up -- there are some key pieces of the code still not covered by tests.

I agree. I was half expecting the patch coverage numbers but must've missed them when I selected reviewers.

As for increasing coverage of some key pieces, it would be easier with refactoring (e.g., build_process). I was hoping to do that in separate issues/PRs as there are a couple of other requests you made for #13100 that still need to be addressed.

eugeneswalker · 2020-02-25T18:27:56Z

This PR restores functionality for our pipeline @tldahlgren @tgamblin @scottwittenburg

eugeneswalker · 2020-02-25T18:42:50Z

BUT, I am still seeing the same issue using spack install --only dependencies <pkg> . For example, try this in Docker image spack/ubuntu-bionic:

spack mirror add e4s https://instinct.nic.uoregon.edu:8083/e4s
spack spec -y py-ptyprocess > py-ptyprocess.spec.yaml
spack install --cache-only --only dependencies ./py-ptyprocess.spec.yaml

Output (truncated to only show final error):

==> Error: Detected uninstalled dependencies for diffutils: {'libiconv'}
==> Error: Cannot proceed with diffutils: 1 uninstalled dependency: libiconv

The KEY is that libiconv IS INSTALLED, despite the above error message:

$> spack find libiconv
--- linux-ubuntu18.04-x86_64 / [email protected] --------------
[email protected]

@tldahlgren @scottwittenburg

tldahlgren · 2020-02-25T18:44:58Z

BUT, I am still seeing the same issue using spack install --only dependencies <pkg> . For example, try this in Docker image spack/ubuntu-bionic:
spack mirror add e4s https://instinct.nic.uoregon.edu:8083/e4s
spack spec -y py-ptyprocess > py-ptyprocess.spec.yaml
spack install --cache-only --only dependencies ./py-ptyprocess.spec.yaml
Output (truncated to only show final error):
==> Error: Detected uninstalled dependencies for diffutils: {'libiconv'}
==> Error: Cannot proceed with diffutils: 1 uninstalled dependency: libiconv
The KEY is that libiconv IS INSTALLED, despite the above error message:
$> spack find libiconv
--- linux-ubuntu18.04-x86_64 / [email protected] --------------
[email protected]
@tldahlgren @scottwittenburg

Thanks for clearing that up. I plan to work on this after my morning meetings.

tldahlgren · 2020-02-25T20:17:57Z

BUT, I am still seeing the same issue using spack install --only dependencies <pkg> . For example, try this in Docker image spack/ubuntu-bionic:
spack mirror add e4s https://instinct.nic.uoregon.edu:8083/e4s
spack spec -y py-ptyprocess > py-ptyprocess.spec.yaml
spack install --cache-only --only dependencies ./py-ptyprocess.spec.yaml
Output (truncated to only show final error):
==> Error: Detected uninstalled dependencies for diffutils: {'libiconv'}
==> Error: Cannot proceed with diffutils: 1 uninstalled dependency: libiconv
The KEY is that libiconv IS INSTALLED, despite the above error message:
$> spack find libiconv
--- linux-ubuntu18.04-x86_64 / [email protected] --------------
[email protected]
@tldahlgren @scottwittenburg
Thanks for clearing that up. I plan to work on this after my morning meetings.

@eugeneswalker Not sure what is going on. I followed the directions above but get the same failure, which appears to be tied to curl refusing connections (code 7). This happens whether I use the -k option or not.

spack find libiconv returns the following for me:

==> No package matches the query: libiconv

Using curl -- since no wget -- to get e4s.pub (per prior slack messages) fails with a "Connection refused" error.

tldahlgren · 2020-02-26T00:31:28Z

@tgamblin May I create separate issues to address #13100 coverage not included in this PR (or the equivalent)?

tldahlgren · 2020-02-26T02:15:10Z

BUT, I am still seeing the same issue using spack install --only dependencies <pkg> . For example, try this in Docker image spack/ubuntu-bionic:
spack mirror add e4s https://instinct.nic.uoregon.edu:8083/e4s
spack spec -y py-ptyprocess > py-ptyprocess.spec.yaml
spack install --cache-only --only dependencies ./py-ptyprocess.spec.yaml
Output (truncated to only show final error):
==> Error: Detected uninstalled dependencies for diffutils: {'libiconv'}
==> Error: Cannot proceed with diffutils: 1 uninstalled dependency: libiconv
The KEY is that libiconv IS INSTALLED, despite the above error message:
$> spack find libiconv
--- linux-ubuntu18.04-x86_64 / [email protected] --------------
[email protected]
@tldahlgren @scottwittenburg
Thanks for clearing that up. I plan to work on this after my morning meetings.
BUT, I am still seeing the same issue using spack install --only dependencies <pkg> . For example, try this in Docker image spack/ubuntu-bionic:
spack mirror add e4s https://instinct.nic.uoregon.edu:8083/e4s
spack spec -y py-ptyprocess > py-ptyprocess.spec.yaml
spack install --cache-only --only dependencies ./py-ptyprocess.spec.yaml
Output (truncated to only show final error):
==> Error: Detected uninstalled dependencies for diffutils: {'libiconv'}
==> Error: Cannot proceed with diffutils: 1 uninstalled dependency: libiconv
The KEY is that libiconv IS INSTALLED, despite the above error message:
$> spack find libiconv
--- linux-ubuntu18.04-x86_64 / [email protected] --------------
[email protected]
@tldahlgren @scottwittenburg
Thanks for clearing that up. I plan to work on this after my morning meetings.
@eugeneswalker Not sure what is going on. I followed the directions above but get the same failure, which appears to be tied to curl refusing connections (code 7). This happens whether I use the -k option or not.

spack find libiconv returns the following for me:
==> No package matches the query: libiconv
Using curl -- since no wget -- to get e4s.pub (per prior slack messages) fails with a "Connection refused" error.

Thanks for your help setting up the environment to reproduce the errors you were seeing. I created a separate issue, #15219 , for it. I was able to determine that the dependents were not getting populated for any of the specs loaded from yaml. It appears the main change in #15220 resolved the problem, at least for me.

eugeneswalker · 2020-02-27T20:06:41Z

In conjunction with PR #15220, the pipeline seems to be working now! Thank you!

tldahlgren · 2020-02-27T21:48:38Z

In conjunction with PR #15220, the pipeline seems to be working now! Thank you!

@tgamblin Can this be merged now? @eugeneswalker mentioned in slack that this was also needed to resolve his issues with the distributed build

tldahlgren · 2020-02-27T21:51:12Z

@tldahlgren: this looks great,

Thanks for the confirmation that you're fine with this approach.

but we need to get the coverage up -- there are some key pieces of the code still not covered by tests.

I agree. I was half expecting the patch coverage numbers but must've missed them when I selected reviewers.

As for increasing coverage of some key pieces, it would be easier with refactoring (e.g., build_process). I was hoping to do that in separate issues/PRs as there are a couple of other requests you made for #13100 that still need to be addressed.

See #15237 for work on additional installer.py coverage. My intention is to review other package/install-related packages and increase their coverage in separate PRs.

…15197) The distributed build PR (#13100) -- did not check the install status of dependencies when using the `--only package` option so would refuse to install a package with the claim that it had uninstalled dependencies whether that was the case or not. - [x] add install status checks for the `--only package` case. - [x] add initial set of tests

…upstream_master * commit 'e2b1737a42c9c0c796671f9dd0c39f623e4c91c0': (1343 commits) update CHANGELOG.md for 0.14.1 version bump: 0.14.1 multiprocessing: allow Spack to run uninterrupted in background (spack#14682) Cray bugfix: TERM missing while reading default target (spack#15381) Upstreams: don't write metadata directory to upstream DB (spack#15526) Creating versions from urls doesn't modify class attributes (spack#15452) bugfix: fix install_missing_compilers option bug from v0.14.0 (spack#15416) bugfix: installer.py shouldn't be executable (spack#15386) Add function replace_prefix_nullterm for use on mach-o rpaths. (spack#15347) ArchSpec: fix semantics of satisfies when not concrete and strict is true (spack#15319) suite-sparse: fix installation for v5.X (spack#15326) testing: increase installer coverage (spack#15237) bugfix: resolve undefined source_pkg_dir failure (spack#15339) Bugfix: resolve StopIteration message attribute failure (spack#15341) Recover coverage from subprocesses during unit tests (spack#15354) Correct pytest.raises matches to match (spack#15346) bugfix: Add dependents when initializing spec from yaml (spack#15220) Uniquify suffixes added to module names (spack#14920) bugfix: ensure proper dependency handling for package-only installs (spack#15197) Fix for being able to 'spack load' packages that have been renamed. (spack#14348) ... # Conflicts: # .travis.yml # lib/spack/spack/modules/common.py # var/spack/repos/builtin/packages/mofem-cephas/package.py # var/spack/repos/builtin/packages/mofem-fracture-module/package.py # var/spack/repos/builtin/packages/mofem-users-modules/package.py # var/spack/repos/builtin/packages/python/package.py

bugfix: ensure proper dependency handling for package-only installs

6728a40

tldahlgren self-assigned this Feb 25, 2020

tldahlgren added bugfix Something wasn't working, here's a fix build labels Feb 25, 2020

tldahlgren requested review from alalazo, becker33 and tgamblin February 25, 2020 02:55

tgamblin requested changes Feb 25, 2020

View reviewed changes

Added test coverage for _check_deps_status

deeff7e

tldahlgren mentioned this pull request Feb 26, 2020

bug: spack install from yaml fails #15219

Closed

tgamblin approved these changes Mar 2, 2020

View reviewed changes

tgamblin merged commit 73f7301 into spack:develop Mar 2, 2020

tgamblin mentioned this pull request Jul 23, 2020

ci: fixes for compiler bootstrapping #17563

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bugfix: ensure proper dependency handling for package-only installs#15197

bugfix: ensure proper dependency handling for package-only installs#15197
tgamblin merged 2 commits intospack:developfrom
tldahlgren:bugfix/fix-install-package-only

tldahlgren commented Feb 25, 2020 •

edited

Loading

Uh oh!

tldahlgren commented Feb 25, 2020

Uh oh!

tgamblin left a comment

Uh oh!

tldahlgren commented Feb 25, 2020 •

edited

Loading

Uh oh!

eugeneswalker commented Feb 25, 2020 •

edited

Loading

Uh oh!

eugeneswalker commented Feb 25, 2020 •

edited

Loading

Uh oh!

tldahlgren commented Feb 25, 2020

Uh oh!

tldahlgren commented Feb 25, 2020

Uh oh!

tldahlgren commented Feb 26, 2020 •

edited

Loading

Uh oh!

tldahlgren commented Feb 26, 2020

Uh oh!

eugeneswalker commented Feb 27, 2020

Uh oh!

tldahlgren commented Feb 27, 2020 •

edited

Loading

Uh oh!

tldahlgren commented Feb 27, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

tldahlgren commented Feb 25, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tldahlgren commented Feb 25, 2020

Uh oh!

tgamblin left a comment

Choose a reason for hiding this comment

Uh oh!

tldahlgren commented Feb 25, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eugeneswalker commented Feb 25, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eugeneswalker commented Feb 25, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tldahlgren commented Feb 25, 2020

Uh oh!

tldahlgren commented Feb 25, 2020

Uh oh!

tldahlgren commented Feb 26, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tldahlgren commented Feb 26, 2020

Uh oh!

eugeneswalker commented Feb 27, 2020

Uh oh!

tldahlgren commented Feb 27, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tldahlgren commented Feb 27, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

tldahlgren commented Feb 25, 2020 •

edited

Loading

tldahlgren commented Feb 25, 2020 •

edited

Loading

eugeneswalker commented Feb 25, 2020 •

edited

Loading

eugeneswalker commented Feb 25, 2020 •

edited

Loading

tldahlgren commented Feb 26, 2020 •

edited

Loading

tldahlgren commented Feb 27, 2020 •

edited

Loading