Skip to content

Pin git sha to git ref based versions#44319

Closed
psakievich wants to merge 43 commits intospack:developfrom
psakievich:psakiev/f/pin
Closed

Pin git sha to git ref based versions#44319
psakievich wants to merge 43 commits intospack:developfrom
psakievich:psakiev/f/pin

Conversation

@psakievich
Copy link
Copy Markdown
Contributor

@psakievich psakievich commented May 22, 2024

This PR will make it so the concretizer assigns the latest commit sha for branch based versions of packages. Right now this is done after the solve. If we try to move it inside the solve I believe we'd have to run git ls-remote on every possible branch-version that could appear in the graph. Seems wasteful makes the solve dependent on the speed of that operation.

- [x] Add a config option, and CLI option to the common solve arguments

  • Add tests for re-use of an exact hash match
  • Add functionality for air-gapped systems
  • Extend branch implementation to all git ref checkouts
  • Add docs

Depends on #43859
Fixes #4674

This PR will make it so the concretizer assigns the latest commit
sha for branch based versions of packages.
@spackbot-app spackbot-app bot added core PR affects Spack core functionality new-version tests General test capability(ies) labels May 22, 2024
@psakievich
Copy link
Copy Markdown
Contributor Author

Effect of this PR:

$ spack solve exawind@master ^nalu-wind@master+tioga ^amr-wind@main
==> Best of 13 considered solutions.
==> Optimization Criteria:
  Priority  Criterion                                            Installed  ToBuild
  1         number of packages to build (vs. reuse)                      -       21
  2         number of nodes from the same package                        -        0
  3         requirement weight                                           0        0
  4         deprecated versions used                                     0        0
  5         version badness (roots)                                      0        0
  6         number of non-default variants (roots)                       0        0
  7         preferred providers for roots                                0        0
  8         default values of variants not being used (roots)            0        0
  9         number of non-default variants (non-roots)                  13        0
  10        preferred providers (non-roots)                              0        0
  11        compiler mismatches that are not from CLI                    0        0
  12        compiler mismatches that are not from CLI                    0        0
  13        non-preferred OS's                                          36        0
  14        version badness (non roots)                                500       32
  15        default values of variants not being used (non-roots)        2       19
  16        non-preferred compilers                                    108        0
  17        target mismatches                                           27        0
  18        non-preferred targets                                      180        0
  19        compiler mismatches (runtimes)                               0        0
  20        version badness (runtimes)                                   0        0
  21        non-preferred targets (runtimes)                             0        0
  22        edge wiring                                               2699        0

 -   [email protected]=master%[email protected]~amr_wind_gpu~cuda~gpu-aware-mpi~ipo~nalu_wind_gpu~rocm~sycl build_system=cmake build_type=Release generator=make arch=darwin-sonoma-m1
 -       ^[email protected]=main%[email protected]~ascent~cuda~gpu-aware-mpi~hdf5~helics~hypre~ipo~masa+mpi+netcdf~openfast~openmp~rocm+shared~sycl+tests+tiny_profile~umpire~waves2amr build_system=cmake build_type=Release generator=make arch=darwin-sonoma-m1
 -           ^[email protected]%[email protected]+blosc~byterange~dap~fsync~hdf4~jna+mpi~nczarr_zip+optimize+parallel-netcdf+pic+shared+szip+zstd build_system=autotools patches=0161eb8,4e10547 arch=darwin-sonoma-m1
 -               ^[email protected]%[email protected] build_system=autotools arch=darwin-ventura-aarch64
 -               ^[email protected]%[email protected] build_system=autotools arch=darwin-ventura-aarch64
 -               ^[email protected]%[email protected]~debug~pic+shared build_system=generic arch=darwin-ventura-aarch64
 -                   ^[email protected]%[email protected] build_system=autotools arch=darwin-ventura-aarch64
 -               ^[email protected]%[email protected]+avx2~ipo build_system=cmake build_type=Release generator=make arch=darwin-sonoma-m1
 -                   ^[email protected]%[email protected]+pic build_system=makefile libs=shared,static arch=darwin-sonoma-m1
 -                   ^[email protected]%[email protected]~ipo+pic+shared build_system=cmake build_type=Release generator=make arch=darwin-sonoma-m1
 -               ^[email protected]%[email protected]~ipo+shared build_system=cmake build_type=Release generator=make arch=darwin-sonoma-m1
 -               ^[email protected]%[email protected]+sigsegv build_system=autotools patches=9dc5fbd,bfdffa7 arch=darwin-ventura-aarch64
 -                   ^[email protected]%[email protected] build_system=autotools arch=darwin-ventura-aarch64
 -               ^[email protected]%[email protected]+programs build_system=makefile compression=none libs=shared,static arch=darwin-ventura-aarch64
 -       ^[email protected]%[email protected]~doc+ncurses+ownlibs build_system=generic build_type=Release arch=darwin-ventura-aarch64
 -           ^[email protected]%[email protected]~gssapi~ldap~libidn2~librtmp~libssh~libssh2+nghttp2 build_system=autotools libs=shared,static tls=secure_transport arch=darwin-ventura-aarch64
 -               ^[email protected]%[email protected] build_system=autotools arch=darwin-ventura-aarch64
 -               ^[email protected]%[email protected]+cpanm+opcode+open+shared+threads build_system=generic patches=714e4d1 arch=darwin-ventura-aarch64
 -           ^[email protected]%[email protected]~symlinks+termlib abi=none build_system=autotools patches=7a351bc arch=darwin-ventura-aarch64
 -           ^[email protected]%[email protected]+compat+new_strategies+opt+pic+shared build_system=autotools arch=darwin-ventura-aarch64
 -       ^[email protected]%[email protected]~guile build_system=generic arch=darwin-ventura-aarch64
 -       ^[email protected]=master%[email protected]~boost~catalyst~cuda~fftw+fsi~gpu-aware-mpi+hypre~ipo+openfast+pic~rocm+shared+tioga+trilinos-solvers~umpire~wind-utils abs_tol=1e-15 build_system=cmake build_type=Release generator=make rel_tol=1e-12 arch=darwin-sonoma-m1
 -           ^[email protected]%[email protected]~caliper~complex~cublas~cuda~debug+fortran~gptune~gpu-aware-mpi~int64~internal-superlu~magma~mixedint+mpi~openmp~rocblas~rocm~shared~superlu-dist~sycl~umpire~unified-memory build_system=autotools arch=darwin-sonoma-m1
 -           ^[email protected]%[email protected]~ipo build_system=cmake build_type=Release generator=make arch=darwin-sonoma-m1
 -       ^[email protected]=develop%[email protected]+cxx+dll-interface+double-precision~ipo+netcdf~openmp+pic~rosco+shared build_system=cmake build_type=Release generator=make arch=darwin-sonoma-m1
 -           ^[email protected]%[email protected]+cxx~fortran+hl~ipo~java~map+mpi+shared~subfiling~szip~threadsafe+tools api=default build_system=cmake build_type=Release generator=make patches=82088c8 arch=darwin-sonoma-m1
 -           ^[email protected]%[email protected]+pic~python+shared build_system=autotools arch=darwin-ventura-aarch64
 -               ^[email protected]%[email protected] build_system=autotools libs=shared,static arch=darwin-ventura-aarch64
 -               ^[email protected]%[email protected]~pic build_system=autotools libs=shared,static arch=darwin-ventura-aarch64
 -           ^[email protected]%[email protected]~bignuma~consistent_fpcsr+dynamic_dispatch~fortran~ilp64+locking+pic+shared build_system=makefile symbol_suffix=none threads=none arch=darwin-ventura-aarch64
 -       ^[email protected]%[email protected]~atomics~cuda~gpfs~internal-hwloc~internal-libevent~internal-pmix~java~legacylaunchers~lustre~memchecker~openshmem~orterunprefix~romio+rsh~static+vt+wrapper-rpath build_system=autotools fabrics=none romio-filesystem=none schedulers=none arch=darwin-ventura-aarch64
 -           ^[email protected]%[email protected] build_system=autotools arch=darwin-ventura-aarch64
 -           ^[email protected]%[email protected] build_system=autotools arch=darwin-ventura-aarch64
 -           ^gnuconfig@2022-09-17%[email protected] build_system=generic arch=darwin-ventura-aarch64
 -           ^[email protected]%[email protected]~cairo~cuda~gl~libudev+libxml2~netloc~nvml~oneapi-level-zero~opencl~pci~rocm build_system=autotools libs=shared,static arch=darwin-ventura-aarch64
 -           ^[email protected]%[email protected]+openssl build_system=autotools arch=darwin-ventura-aarch64
 -               ^[email protected]%[email protected]~docs+shared build_system=generic certs=mozilla arch=darwin-ventura-aarch64
 -                   ^ca-certificates-mozilla@2023-05-30%[email protected] build_system=generic arch=darwin-ventura-aarch64
 -           ^[email protected]%[email protected] build_system=autotools arch=darwin-ventura-aarch64
 -               ^[email protected]%[email protected] build_system=autotools patches=440b954 arch=darwin-ventura-aarch64
 -           ^[email protected]%[email protected]+gssapi build_system=autotools patches=3505c58,d886b98 arch=darwin-ventura-aarch64
 -               ^[email protected]%[email protected]+shared build_system=autotools arch=darwin-ventura-aarch64
 -                   ^[email protected]%[email protected]~color build_system=autotools arch=darwin-ventura-aarch64
 -                   ^[email protected]%[email protected]+bzip2+curses+git~libunistring+libxml2+pic+shared+tar+xz build_system=autotools arch=darwin-ventura-aarch64
 -                       ^[email protected]%[email protected] build_system=autotools zip=pigz arch=darwin-ventura-aarch64
 -                           ^[email protected]%[email protected] build_system=makefile arch=darwin-ventura-aarch64
 -               ^[email protected]%[email protected] build_system=autotools arch=darwin-ventura-aarch64
 -               ^[email protected]%[email protected]~obsolete_api build_system=autotools patches=4885da3 arch=darwin-ventura-aarch64
 -           ^[email protected]%[email protected]+cpanm+opcode+open+shared+threads build_system=generic patches=714e4d1 arch=darwin-ventura-aarch64
 -               ^[email protected]%[email protected]+cxx~docs+stl build_system=autotools patches=26090f4,b231fcc arch=darwin-ventura-aarch64
 -               ^[email protected]%[email protected] build_system=autotools arch=darwin-ventura-aarch64
 -                   ^[email protected]%[email protected] build_system=autotools patches=bbf97f1 arch=darwin-ventura-aarch64
 -           ^[email protected]%[email protected] build_system=autotools arch=darwin-ventura-aarch64
 -           ^[email protected]%[email protected]~docs+pmi_backwards_compatibility~python~restful build_system=autotools arch=darwin-ventura-aarch64
 -       ^[email protected]=develop%[email protected]~ipo~nodegid+pic~shared~stats~timers build_system=cmake build_type=Release generator=make arch=darwin-sonoma-m1
 -       ^[email protected]%[email protected]~adelus~adios2+amesos+amesos2+anasazi+aztec~basker+belos~boost~chaco~complex~cuda~cuda_rdc~debug~dtk+epetra+epetraext~epetraextbtf~epetraextexperimental~epetraextgraphreorderings+exodus+explicit_template_instantiation~float+fortran+gtest+hdf5~hypre+ifpack+ifpack2~intrepid~intrepid2~ipo~isorropia+kokkos~mesquite~minitensor+ml+mpi+muelu~mumps~nox~openmp~pamgen~panzer~phalanx~piro~python~rocm~rocm_rdc~rol~rythmos+sacado~scorec+shards~shared~shylu+stk~stokhos~stratimikos~strumpack~suite-sparse~superlu~superlu-dist~teko~tempus~test~thyra+tpetra~trilinoscouplings~wrapper~x11+zoltan~zoltan2 build_system=cmake build_type=Release cxxstd=17 generator=make gotype=long arch=darwin-sonoma-m1
 -           ^[email protected]%[email protected]~base_scope~fortran+hdf5~int64~ipo~legacy~mem_debug+mpi~pic+scoping+shared~static~testing~tools build_system=cmake build_type=Release generator=make arch=darwin-sonoma-m1
 -           ^[email protected]%[email protected]~aggressive_vectorization~compiler_warnings~cuda~debug~debug_bounds_check~debug_dualview_modify_check~deprecated_code~examples~hpx~hpx_async_dispatch~hwloc~ipo~memkind~numactl~openmp~openmptarget~pic~rocm+serial+shared~sycl~tests~threads~tuning~wrapper build_system=cmake build_type=Release cxxstd=17 generator=make intel_gpu_arch=none arch=darwin-sonoma-m1
 -           ^[email protected]%[email protected]+hdf5+shared+zlib build_system=autotools arch=darwin-sonoma-m1
 -           ^[email protected]%[email protected]~gdb~int64~ipo~real64+shared build_system=cmake build_type=Release generator=make patches=4991da9,93a7903 arch=darwin-sonoma-m1
 -           ^[email protected]%[email protected]~burstbuffer+cxx+fortran+pic+shared build_system=autotools arch=darwin-sonoma-m1
 -           ^[email protected]%[email protected]~gdb~int64~ipo+shared build_system=cmake build_type=Release generator=make patches=4f89253,50ed208,704b84f arch=darwin-sonoma-m1
 -       ^[email protected]%[email protected]~ipo+pic+shared~tests build_system=cmake build_type=Release generator=make arch=darwin-sonoma-m1

@psakievich psakievich changed the title Pin git sha to branch based versions Pin git sha to git ref based versions Jul 3, 2024
@psakievich psakievich added the snl-core-team Issue for SNL Spack developers label Dec 3, 2024
@becker33
Copy link
Copy Markdown
Member

becker33 commented Dec 5, 2024

I think this fixes #4674, @psakievich can you edit the description to say this closes/fixes that issue?

@psakievich psakievich closed this by deleting the head repository May 14, 2025
@tgamblin tgamblin removed this from the v1.0.0 stretch goals milestone Jun 11, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core PR affects Spack core functionality defaults fetching new-version shell-support snl-core-team Issue for SNL Spack developers tests General test capability(ies) update-package versions

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Store commit hash along with tag or branch

3 participants