Allow the relocation of build cache when directory layout is different. by gartung · Pull Request #27 · FNALssi/spack

gartung · 2019-11-13T22:28:58Z

No description provided.

lib/spack/spack/binary_distribution.py

lib/spack/spack/relocate.py

var/spack/repos/builtin/packages/mesa/package.py

greenc-FNAL · 2019-11-14T17:03:52Z

Question: what is special about allow_root, how is it set, and why does that make things work?

gartung · 2019-11-14T17:06:49Z

Allow_root is a flag passed to spack buildcache create and install. Its meaning is allow install root string in binaries. This was added because some people still wanted to create buildcaches even with the strings.

greenc-FNAL · 2019-11-14T17:09:13Z

Is there a reason the exceptions inherit from Exception rather than SpackError, or BuildCacheRelocationError (say)? I'd also prefer if msg were something short and pithy (e.g. "New path longer than old path: binary relocation not possible," and the details (what the user can do to mitigate, for example) in long_msg).

gartung · 2019-11-14T17:12:06Z

I will switch to SpackError. Does SpackError allow for short and long error messages?

Right now I am trying to debug why spack suddenly thinks that larsoft loaded from the builcache is a virtual or external package.

gartung · 2019-11-14T20:49:56Z

If you look at /scratch/gartung/unrelocated_strings_gcc730_sorted.txt on scisoftbuild01 you will see that many references are to the stage directory paths and files. After removing the stage paths I get /scratch/gartung/unrelocated_strings_gcc730_nostage.txt which appears at first glance to innocuous.

greenc-FNAL · 2019-11-14T22:47:22Z

I believe SpackError offers parameters msg and long_msg.

From the point of view of the quality of the relocation operation overall, it looks innocuous, but the question is always going to be, "does it work?" The problem with most of these schemes is that we're not going to know until we know.

gartung · 2019-11-15T02:47:59Z

Right now I am trying to debug why spack suddenly thinks that larsoft loaded from the builcache is a virtual or external package.

The cause of this was a missing package definition for larsoft. I had not added the fnal_art package repo to the spack directory I was working with.

gartung · 2019-11-15T15:44:09Z

Thanks for the comments. With the correct layout on the install side this work is not needed for the MVP1a bootstrap.

@omsai

1. support version 3.1.3, which now depends on sundials@6 2. support version 3.1.2:, which broke the two patch files and therefore the two patch files have been replaced by more flexible filter_file() commands inside a patch() function. 3. rename the variant for python extension from using the package name "+pyuqtk" to the more standard "+python" 4. add maintainers @omsai and the upstream developer @bjdebus who offered to help with the spack packaging. 5. swig should only be a build-time dependency. swig is only necessary until @:3.1.0 6. confirmed python dependencies are correct by inspecting imports, subset python dependencies type to build, run, and confirmed all 31 build-time tests pass including the 9 python tests: ```console $ spack env create uqtk-dev $ spack add [email protected] $ spack install --test root && cat $(spack location -i uqtk)/.spack/install-time-test-log.txt ==> Testing package uqtk-3.1.3-nok6fut ==> [2023-04-19-14:56:25.005361] Running build-time tests ==> [2023-04-19-14:56:25.005536] RUN-TESTS: build-time tests [check] ==> [2023-04-19-14:56:25.009543] '/home/omsai/src/spack/opt/spack/linux-pureos10-skylake/gcc-10.2.1/gmake-4.4.1-b6g4apmfvxz3bn4eabh37dehcrg65fj7/bin/make' '-j4' '-n' 'test' ==> [2023-04-19-14:56:25.014903] '/home/omsai/src/spack/opt/spack/linux-pureos10-skylake/gcc-10.2.1/gmake-4.4.1-b6g4apmfvxz3bn4eabh37dehcrg65fj7/bin/make' '-j4' 'test' Running tests... /home/omsai/src/spack/opt/spack/linux-pureos10-skylake/gcc-10.2.1/cmake-3.26.3-zjmsfz23j5l4ytniz26uzvxonlu5qebr/bin/ctest --force-new-ctest-process Test project /tmp/omsai/spack-stage/spack-stage-uqtk-3.1.3-nok6fut47h42cnaau7wkoohgqy5f2qqa/spack-build-nok6fut Start 1: ArrayReadAndWrite Start 2: ArrayDelColumn Start 3: Array1DMiscTest Start 4: Array2DMiscTest 1/31 Test #1: ArrayReadAndWrite ................ Passed 0.01 sec Start 5: ArraySortTest 2/31 Test #2: ArrayDelColumn ................... Passed 0.01 sec Start 6: MultiIndexTest 3/31 Test #3: Array1DMiscTest .................. Passed 0.01 sec Start 7: CorrTest 4/31 Test #4: Array2DMiscTest .................. Passed 0.01 sec Start 8: QuadLUTest 5/31 Test #5: ArraySortTest .................... Passed 0.02 sec Start 9: MCMC2dTest 6/31 Test #6: MultiIndexTest ................... Passed 0.01 sec Start 10: MCMCRandomTest 7/31 Test #8: QuadLUTest ....................... Passed 0.02 sec Start 11: MCMCNestedTest 8/31 Test #10: MCMCRandomTest ................... Passed 0.02 sec Start 12: Deriv1dTest 9/31 Test #12: Deriv1dTest ...................... Passed 0.01 sec Start 13: SecondDeriv1dTest 10/31 Test #13: SecondDeriv1dTest ................ Passed 0.01 sec Start 14: GradHessianTest 11/31 Test #11: MCMCNestedTest ................... Passed 0.03 sec Start 15: GradientPCETest 12/31 Test #14: GradHessianTest .................. Passed 0.01 sec Start 16: PCE1dTest 13/31 Test #15: GradientPCETest .................. Passed 0.01 sec Start 17: PCEImplTest 14/31 Test #16: PCE1dTest ........................ Passed 0.01 sec Start 18: PCELogTest 15/31 Test #18: PCELogTest ....................... Passed 0.01 sec Start 19: Hessian2dTest 16/31 Test #19: Hessian2dTest .................... Passed 0.01 sec Start 20: BCS1dTest 17/31 Test #20: BCS1dTest ........................ Passed 0.01 sec Start 21: BCS2dTest 18/31 Test #21: BCS2dTest ........................ Passed 0.01 sec Start 22: LowRankRegrTest 19/31 Test #22: LowRankRegrTest .................. Passed 0.01 sec Start 23: PyModTest 20/31 Test #17: PCEImplTest ...................... Passed 0.07 sec Start 24: PyArrayTest 21/31 Test #23: PyModTest ........................ Passed 0.08 sec Start 25: PyArrayTest2 22/31 Test #25: PyArrayTest2 ..................... Passed 0.30 sec Start 26: PyQuadTest 23/31 Test #24: PyArrayTest ...................... Passed 1.44 sec Start 27: PyBCSTest1D 24/31 Test #26: PyQuadTest ....................... Passed 1.68 sec Start 28: PyBCSTest2D 25/31 Test #27: PyBCSTest1D ...................... Passed 1.66 sec Start 29: PyBADPTest 26/31 Test #7: CorrTest ......................... Passed 3.43 sec Start 30: PyRegressionTest 27/31 Test #28: PyBCSTest2D ...................... Passed 1.50 sec Start 31: PyGalerkinTest 28/31 Test #9: MCMC2dTest ....................... Passed 3.90 sec 29/31 Test #29: PyBADPTest ....................... Passed 1.66 sec 30/31 Test #30: PyRegressionTest ................. Passed 1.72 sec 31/31 Test #31: PyGalerkinTest ................... Passed 1.63 sec 100% tests passed, 0 tests failed out of 31 Total Test time (real) = 5.35 sec ==> [2023-04-19-14:56:30.382797] '/home/omsai/src/spack/opt/spack/linux-pureos10-skylake/gcc-10.2.1/gmake-4.4.1-b6g4apmfvxz3bn4eabh37dehcrg65fj7/bin/make' '-j4' '-n' 'check' ==> [2023-04-19-14:56:30.385983] Target 'check' not found in Makefile ```

greenc-FNAL requested changes Nov 14, 2019

View reviewed changes

gartung force-pushed the MVP1a-branch branch from 686079a to 032e55f Compare November 14, 2019 16:48

gartung force-pushed the MVP1a-branch branch 5 times, most recently from ae1f3ad to 22d23ec Compare November 14, 2019 19:36

Allow relocation of buildcache when directory layout is different

ebbe993

gartung force-pushed the MVP1a-branch branch from 22d23ec to ebbe993 Compare November 14, 2019 19:47

gartung closed this Nov 15, 2019

gartung mentioned this pull request Nov 15, 2019

Feature: buildcache relocation with different layout and compiler location spack/spack#10522

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow the relocation of build cache when directory layout is different.#27

Allow the relocation of build cache when directory layout is different.#27
gartung wants to merge 1 commit intoFNALssi:MVP1a-branchfrom
gartung:MVP1a-branch

gartung commented Nov 13, 2019

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

greenc-FNAL commented Nov 14, 2019

Uh oh!

gartung commented Nov 14, 2019 via email •

edited

Loading

Uh oh!

greenc-FNAL commented Nov 14, 2019

Uh oh!

gartung commented Nov 14, 2019

Uh oh!

gartung commented Nov 14, 2019

Uh oh!

greenc-FNAL commented Nov 14, 2019

Uh oh!

gartung commented Nov 15, 2019

Uh oh!

gartung commented Nov 15, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

gartung commented Nov 13, 2019

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

greenc-FNAL commented Nov 14, 2019

Uh oh!

gartung commented Nov 14, 2019 via email • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

greenc-FNAL commented Nov 14, 2019

Uh oh!

gartung commented Nov 14, 2019

Uh oh!

gartung commented Nov 14, 2019

Uh oh!

greenc-FNAL commented Nov 14, 2019

Uh oh!

gartung commented Nov 15, 2019

Uh oh!

gartung commented Nov 15, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

gartung commented Nov 14, 2019 via email •

edited

Loading