Changes needed to support installing on cvmfs by gartung · Pull Request #22 · FNALssi/spack

gartung · 2019-09-18T14:37:08Z

Changes to buildcache creation and installation to allow the installation of macOS binaries using a linux host.
Change the location of the shebang script from the spack checkout directory to the install root. The shebang script can then be referenced from a directory that is visible on the cvmfs server and the cvmfs client. This is important so that the install root can be set to /cvmfs/fermilab.opensciencegrid.org while the spack directory could be elsewhere.

Some hackery is involved if the install host is not the same OS as the buildcache. A compiler entry is needed for the OS the buildcache was created on. I added an entry for macOS to spack/etc/spack/compilers. An entry will be needed for SL7 if the cvmfs host is SL6 as well.

…eplacing long sbangs.

…location using py-machotools on linux or macos. Update py-machotools dependencies and versions.

gartung · 2019-09-18T14:37:42Z

@chissg I need to redo this. I should have done a cherry-pick instead of a merge.

gartung · 2019-09-18T14:37:57Z

spack/etc/compilers.yaml

compilers:
- compiler:
    environment: {}
    extra_rpaths: []
    flags: {}
    modules: []
    operating_system: scientific7
    paths:
      cc: /usr/bin/gcc
      cxx: /usr/bin/g++
      f77: /usr/bin/gfortran
      fc: /usr/bin/gfortran
    spec: [email protected]
    target: x86_64
- compiler:
    environment: {}
    extra_rpaths: []
    flags: {}
    modules: []
    operating_system: mojave
    paths:
      cc: /usr/bin/gcc
      cxx: /usr/bin/g++
      f77:
      fc:
    spec: [email protected]
    target: x86_64

gartung · 2019-09-18T14:40:09Z

Nevermind. Changing the base from develop to fnal-devel fixed the extra commit showing up.

gartung · 2019-09-18T14:40:39Z

@marcmengel These are the changes that I think will allow installation on cvmfs to work.

gartung · 2019-09-18T15:32:20Z

The PR's are
spack#11598
and
spack#12858

The later was merged by me last night. I pinged Todd about the former this morning.

gartung · 2019-09-18T15:33:16Z

Checking out this PR would allow testing the cvmfs installation.

commit cfcbda86b393a561d5de4100e7464ebb2f9c3673 Author: Patrick Gartung <[email protected]> Date: Wed Sep 18 19:05:39 2019 -0500 get_patchelf should return None for test platform as well because create_buildinfo invokes patchelf to get rpaths. commit 173c976 Author: Patrick Gartung <[email protected]> Date: Wed Sep 18 17:08:06 2019 -0500 Add missing arg to relocate_elf_binaries commit c34df55 Author: Patrick Gartung <[email protected]> Date: Wed Sep 18 16:57:33 2019 -0500 flake8 commit d0ba53f Author: Patrick Gartung <[email protected]> Date: Wed Sep 18 16:40:22 2019 -0500 Save current RPATHS in buildinfo dictionary. Remove RPATHS from binaries for later addition commit ff89390 Author: Patrick Gartung <[email protected]> Date: Wed Sep 18 13:32:38 2019 -0500 Put back the use of otool and install_name_tool when running on macOS. Only use machotools on linux.

@omsai

1. support version 3.1.3, which now depends on sundials@6 2. support version 3.1.2:, which broke the two patch files and therefore the two patch files have been replaced by more flexible filter_file() commands inside a patch() function. 3. rename the variant for python extension from using the package name "+pyuqtk" to the more standard "+python" 4. add maintainers @omsai and the upstream developer @bjdebus who offered to help with the spack packaging. 5. swig should only be a build-time dependency. swig is only necessary until @:3.1.0 6. confirmed python dependencies are correct by inspecting imports, subset python dependencies type to build, run, and confirmed all 31 build-time tests pass including the 9 python tests: ```console $ spack env create uqtk-dev $ spack add [email protected] $ spack install --test root && cat $(spack location -i uqtk)/.spack/install-time-test-log.txt ==> Testing package uqtk-3.1.3-nok6fut ==> [2023-04-19-14:56:25.005361] Running build-time tests ==> [2023-04-19-14:56:25.005536] RUN-TESTS: build-time tests [check] ==> [2023-04-19-14:56:25.009543] '/home/omsai/src/spack/opt/spack/linux-pureos10-skylake/gcc-10.2.1/gmake-4.4.1-b6g4apmfvxz3bn4eabh37dehcrg65fj7/bin/make' '-j4' '-n' 'test' ==> [2023-04-19-14:56:25.014903] '/home/omsai/src/spack/opt/spack/linux-pureos10-skylake/gcc-10.2.1/gmake-4.4.1-b6g4apmfvxz3bn4eabh37dehcrg65fj7/bin/make' '-j4' 'test' Running tests... /home/omsai/src/spack/opt/spack/linux-pureos10-skylake/gcc-10.2.1/cmake-3.26.3-zjmsfz23j5l4ytniz26uzvxonlu5qebr/bin/ctest --force-new-ctest-process Test project /tmp/omsai/spack-stage/spack-stage-uqtk-3.1.3-nok6fut47h42cnaau7wkoohgqy5f2qqa/spack-build-nok6fut Start 1: ArrayReadAndWrite Start 2: ArrayDelColumn Start 3: Array1DMiscTest Start 4: Array2DMiscTest 1/31 Test #1: ArrayReadAndWrite ................ Passed 0.01 sec Start 5: ArraySortTest 2/31 Test #2: ArrayDelColumn ................... Passed 0.01 sec Start 6: MultiIndexTest 3/31 Test #3: Array1DMiscTest .................. Passed 0.01 sec Start 7: CorrTest 4/31 Test #4: Array2DMiscTest .................. Passed 0.01 sec Start 8: QuadLUTest 5/31 Test #5: ArraySortTest .................... Passed 0.02 sec Start 9: MCMC2dTest 6/31 Test #6: MultiIndexTest ................... Passed 0.01 sec Start 10: MCMCRandomTest 7/31 Test #8: QuadLUTest ....................... Passed 0.02 sec Start 11: MCMCNestedTest 8/31 Test #10: MCMCRandomTest ................... Passed 0.02 sec Start 12: Deriv1dTest 9/31 Test #12: Deriv1dTest ...................... Passed 0.01 sec Start 13: SecondDeriv1dTest 10/31 Test #13: SecondDeriv1dTest ................ Passed 0.01 sec Start 14: GradHessianTest 11/31 Test #11: MCMCNestedTest ................... Passed 0.03 sec Start 15: GradientPCETest 12/31 Test #14: GradHessianTest .................. Passed 0.01 sec Start 16: PCE1dTest 13/31 Test #15: GradientPCETest .................. Passed 0.01 sec Start 17: PCEImplTest 14/31 Test #16: PCE1dTest ........................ Passed 0.01 sec Start 18: PCELogTest 15/31 Test #18: PCELogTest ....................... Passed 0.01 sec Start 19: Hessian2dTest 16/31 Test #19: Hessian2dTest .................... Passed 0.01 sec Start 20: BCS1dTest 17/31 Test #20: BCS1dTest ........................ Passed 0.01 sec Start 21: BCS2dTest 18/31 Test #21: BCS2dTest ........................ Passed 0.01 sec Start 22: LowRankRegrTest 19/31 Test #22: LowRankRegrTest .................. Passed 0.01 sec Start 23: PyModTest 20/31 Test #17: PCEImplTest ...................... Passed 0.07 sec Start 24: PyArrayTest 21/31 Test #23: PyModTest ........................ Passed 0.08 sec Start 25: PyArrayTest2 22/31 Test #25: PyArrayTest2 ..................... Passed 0.30 sec Start 26: PyQuadTest 23/31 Test #24: PyArrayTest ...................... Passed 1.44 sec Start 27: PyBCSTest1D 24/31 Test #26: PyQuadTest ....................... Passed 1.68 sec Start 28: PyBCSTest2D 25/31 Test #27: PyBCSTest1D ...................... Passed 1.66 sec Start 29: PyBADPTest 26/31 Test #7: CorrTest ......................... Passed 3.43 sec Start 30: PyRegressionTest 27/31 Test #28: PyBCSTest2D ...................... Passed 1.50 sec Start 31: PyGalerkinTest 28/31 Test #9: MCMC2dTest ....................... Passed 3.90 sec 29/31 Test #29: PyBADPTest ....................... Passed 1.66 sec 30/31 Test #30: PyRegressionTest ................. Passed 1.72 sec 31/31 Test #31: PyGalerkinTest ................... Passed 1.63 sec 100% tests passed, 0 tests failed out of 31 Total Test time (real) = 5.35 sec ==> [2023-04-19-14:56:30.382797] '/home/omsai/src/spack/opt/spack/linux-pureos10-skylake/gcc-10.2.1/gmake-4.4.1-b6g4apmfvxz3bn4eabh37dehcrg65fj7/bin/make' '-j4' '-n' 'check' ==> [2023-04-19-14:56:30.385983] Target 'check' not found in Makefile ```

gartung added 4 commits May 31, 2019 11:30

Copy sbang script to spack.store.layout.root and use that path when r…

7e8877d

…eplacing long sbangs.

Fix sbang test

2af7838

Merge branch 'pr11598' into fnal-devel

26473e5

Update buildcache creation and installation to allow mach-o binary re…

bda19d3

…location using py-machotools on linux or macos. Update py-machotools dependencies and versions.

gartung changed the base branch from develop to fnal-develop September 18, 2019 14:39

gartung force-pushed the fnal-devel-cvmfs-prs branch from c59652c to 0ef9fa0 Compare September 19, 2019 00:11

gartung closed this Sep 19, 2019

gartung deleted the fnal-devel-cvmfs-prs branch September 23, 2019 14:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Changes needed to support installing on cvmfs#22

Changes needed to support installing on cvmfs#22
gartung wants to merge 5 commits intoFNALssi:fnal-developfrom
gartung:fnal-devel-cvmfs-prs

gartung commented Sep 18, 2019 •

edited

Loading

Uh oh!

gartung commented Sep 18, 2019

Uh oh!

gartung commented Sep 18, 2019 •

edited

Loading

Uh oh!

gartung commented Sep 18, 2019

Uh oh!

gartung commented Sep 18, 2019

Uh oh!

gartung commented Sep 18, 2019

Uh oh!

gartung commented Sep 18, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

gartung commented Sep 18, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gartung commented Sep 18, 2019

Uh oh!

gartung commented Sep 18, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gartung commented Sep 18, 2019

Uh oh!

gartung commented Sep 18, 2019

Uh oh!

gartung commented Sep 18, 2019

Uh oh!

gartung commented Sep 18, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

gartung commented Sep 18, 2019 •

edited

Loading

gartung commented Sep 18, 2019 •

edited

Loading