Skip to content

Buildcache: attempt to build and install buildcaches in test environment#15577

Merged
gartung merged 5 commits intodevelopfrom
gartung-bindist-test
Mar 21, 2020
Merged

Buildcache: attempt to build and install buildcaches in test environment#15577
gartung merged 5 commits intodevelopfrom
gartung-bindist-test

Conversation

@gartung
Copy link
Copy Markdown
Member

@gartung gartung commented Mar 20, 2020

@tgamblin @alalazo This was working in the test environment back in December but then changes were made that broke it.

@alalazo
Copy link
Copy Markdown
Member

alalazo commented Mar 20, 2020

@gartung I'll have a look at this asap. Also, while reading the code in spack.relocate I'm trying to get some basic tests for that. I'll ping you for a review when ready to submit a PR.

@gartung
Copy link
Copy Markdown
Member Author

gartung commented Mar 20, 2020

The tests that can be done without binaries are already in packaging.py. I might be able to add a test for get_normalized_elf_rpaths to packaging.py which would have caught the bug with $ORIGIN not getting replaced correctly.

@gartung
Copy link
Copy Markdown
Member Author

gartung commented Mar 20, 2020

I hope this is fixable because I spent way too much time getting it to work and then had to drop it from the other PR because I couldn't figure what broke.

@alalazo alalazo self-assigned this Mar 20, 2020
@gartung
Copy link
Copy Markdown
Member Author

gartung commented Mar 20, 2020

It looks like the error occurs because no build_stage is configured.

@gartung
Copy link
Copy Markdown
Member Author

gartung commented Mar 20, 2020

I used the mock_stage fixture with the nomockstage keyword.

@gartung
Copy link
Copy Markdown
Member Author

gartung commented Mar 21, 2020

I needed to add build_stage and connect_timeout config settings to get this to work,

@gartung gartung merged commit 25893f1 into develop Mar 21, 2020
scheibelp added a commit to scheibelp/spack that referenced this pull request Mar 24, 2020
@scheibelp
Copy link
Copy Markdown
Member

Unit tests should not install real packages. I'm going to have to revert this. I have added #15646 to do so and will be merging it ASAP. We can talk about how to achieve the same result without installing real packages later.

@scheibelp
Copy link
Copy Markdown
Member

Note that PR also explains why unit tests shouldn't install real packages and provides an example of how that can cause our unit tests to fail for unrelated PRs

@gartung
Copy link
Copy Markdown
Member Author

gartung commented Mar 24, 2020

Would it be possible to store small tar files of an already built package and unpack that into an install prefix instead? I am thinking of the corge, quux and garply dummy cmake packages that Jim Amundson made to demostrate spack-dev functionality. The buildcaches for these packages are really tiny.

@scheibelp
Copy link
Copy Markdown
Member

That could work: I'm assuming this means that all the code for the packages is present and doesn't need to be downloaded by Spack.

Fetching is probably the main issue. The amount of time it takes to compile the packages is not a big deal. Actually running the compiler may arguably introduce fragility, but in Spacks case it might be useful to run it where available.

One issue: if these packages actually need CMake to run, then installing CMake as a prereq of doing the unit tests may be problematic (unless we can get the cache to share across all unit test runs).

@gartung
Copy link
Copy Markdown
Member Author

gartung commented Mar 24, 2020

The other time consumer is running file on every file in the package to determine its type.

I propose building them and creating tarballs from the install prefix to be stored in the test/data directory. The install would be mocked up to unpack the tarballs into the correct directory and maybe using patchelf to set the initial rpaths in ELF files. The mock package definitions would not need a dependency on cmake.

@gartung
Copy link
Copy Markdown
Member Author

gartung commented Mar 24, 2020

Would it be possible to run a script in Github Actions and capture the coverage from running spack buildcache create commands?

@gartung gartung deleted the gartung-bindist-test branch March 26, 2020 04:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

buildcache tests General test capability(ies)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants