Skip to content

Do not load view into environment in generated spack ci jobs#17440

Merged
becker33 merged 1 commit intospack:developfrom
haampie:fix-view-in-ci-build
Jul 21, 2020
Merged

Do not load view into environment in generated spack ci jobs#17440
becker33 merged 1 commit intospack:developfrom
haampie:fix-view-in-ci-build

Conversation

@haampie
Copy link
Copy Markdown
Member

@haampie haampie commented Jul 9, 2020

Solves a problem (#17432) where spack seems to be using the wrong python version when python was a build dependency. It seems enabling the environment without loading a view is sufficient to work around that.


job_scripts = [
'spack env activate .',
'spack env activate --without-view .',
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@becker33 It seems like this was encountered by @haampie in attempts to run compiler bootstrapping pipelines, though I have not encountered any need for it in normal pipelines. Can you shed any light on why it might be necessary?

Copy link
Copy Markdown
Member Author

@haampie haampie Jul 15, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If one of the dependencies of a package is python, spack seems to be confusing the system python with the python version in the view during spack commands themselves.

I encountered it when trying to build [email protected] in one of the lasts steps (well, that's where python gets installed...). The main issue is that it cannot find botocore anymore which is a system dependency, and it fails to upload the binaries to the build cache. Probably the search paths for python modules are simplify confused by loading the other python into a view.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So you needed this independent of compiler bootstrapping? I'm just surprised because I have run lots of pipelines with packages that have a dependency on python.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, I'm running it in a non-standard container runtime, so it could be that that somehow influences the environment a bit. Let me try and see tomorrow if it's reproducible when dropping this commit ^ but keeping all the other current patches that seem necessary.

Copy link
Copy Markdown
Member

@becker33 becker33 Jul 15, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@haampie are you running this on Spack v0.14.2? We had a bug up until v0.15.0 that caused Spack to fail if an environment including python was partially installed and the view was activated. If you're seeing this behavior in v0.15.0 that is a much bigger concern to me.

@scottwittenburg On the other hand, there's not really any reason to add the view to the user environment as part of the CI job, so eliminating it should have a (probably negligible) performance benefit.

@becker33 becker33 changed the title Do not create a view in generated spack ci jobs Do not load view into environment in generated spack ci jobs Jul 16, 2020
@becker33
Copy link
Copy Markdown
Member

@haampie updated the title for accuracy.

The view is created on the basis of the spack.yaml file (or the spack env create command). The spack env activate command just controls whether we use that view to load things into the user environment.

I don't think this clarification affects the purpose, effect, or beneficial nature of the PR.

@haampie
Copy link
Copy Markdown
Member Author

haampie commented Jul 17, 2020

It's still reproducible by the way, with all other patches included. See https://gitlab.com/cscs-ci/ci-testing/spack-ci/-/pipelines/167717054

==> [2020-07-17-10:03:27.042200, 2151] Error: ModuleNotFoundError: No module named 'botocore'

only happens when --without-view is missing.

@becker33 becker33 merged commit 6c69b8a into spack:develop Jul 21, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants