I thought I'd add an example of how you edit config.yaml to include upstream instances but that interferes with doc generation and testing so I'll add an example here:

UPDATED 12/12: upstream configuration has been moved to its own config file called upstreams.yaml; each upstream Spack instance includes a name specification; spack-install-prefix has been replaced with install_tree to match the variable as specified in config.yaml for the local Spack install. The new upstreams.yaml config file looks like:

upstreams:
  test-a:
    install_tree: /g/g17/scheibel/repos/test-upstream-spack/spack/opt/spack
    modules:
      tcl: /g/g17/scheibel/repos/test-upstream-spack/share/spack/modules

(OLD) UPDATED 11/21: spack-binary has been removed - it was not necessary. Specification of module roots is now combined with specification of upstream installations to ensure that order matches. For a given upstream Spack instance, a given module type (e.g. tcl) can now have at most one associated module root.

config:
  upstream_spack_installations:
    - spack-install-prefix: /g/g17/scheibel/repos/test-upstream-spack/spack/opt/spack
      modules:
        tcl: /g/g17/scheibel/repos/test-upstream-spack/share/spack/modules

(OLD) UPDATED 8/23: specification of upstream_spack_installations has been updated in the config.yaml. It is now a list of dictionaries; each dictionary has the properties spack-binary and spack-install-prefix. The group of spack installations is now a list to ensure that they are searched in the same order each time.

config:

  upstream_spack_installations:
    - spack-binary: /g/g17/scheibel/repos/test-upstream-spack/bin/spack
      spack-install-prefix: /g/g17/scheibel/repos/test-upstream-spack/spack/opt/spack

  # THIS IS THE SAME AS BEFORE - FROM 8/10
  upstream_module_roots:
    tcl:
      - /g/g17/scheibel/repos/test-upstream-spack/share/spack/modules

(OLD) UPDATED 8/10: originally the upstream installations config variable was list of Spack install directories. Now it is a dictionary of Spack binary to installation directory. The binary is needed to compute module file locations for upstream Spack instances. It is not strictly required to specify the install directory along with the Spack binary for current versions of Spack, since you could have Spack print out the install location. That being said I don't know how far back this support goes, and in general I don't want to invoke several instances of Spack each time I construct the Spack database.

config:
  upstream_spack_installations:
    /path/to/spack/bin/spack: /path/to/spack/opt/spack/

  # This is required if you want the upstream Spack modules files to be in MODULE_PATH
  upstream_module_roots:
    tcl:
      - /path/to/spack/share/spack/modules/

OLD:

config:
  # For now, you specify the path up to where the DB is stored (everything
  # before .spack-db)
  upstream_spack_installations:
    - /path/to/spack/opt/spack

EDIT: it also appears that if there is no upstream_spack_installations entry in config.yaml, that Spack will generally fail - definitely a bug with this PR! I'll be able to tackle that on 7/30. (Now handled)

Hello, I've tested this feature with two installations of spack and two packages, installation 1 with package A and installation 2 with package B -> A and I can confirm that in the second installation spack will only build package B. However I have a case where I point spack to a directory with many pre-built packages and then try to build package C which depends on all of those. Spack will proceed to install every dependency of C again. I am not sure why I get this in the second scenario and I wanted to point it out and that there may be something that is still missing from this feature.

Hi @paul-chelarescu, as a sanity check: did you confirm that C depended on the exact same instance of B and A (i.e. that the dependency hashes were the same)? I think spack spec gives you an option to output the DAG hash (apologies but I'm not in a spot to run spack spec --help to make sure of that right now).

If the dependency hashes do match (i.e. they exist in the upstream DB and aren't being used) would it be possible to attach the package.py files for A/B/C? I've had luck for at least one case of A->B->C, namely sqlite, where all of it's dependencies were installed upstream (so perhaps it is clear how your case differs from sqlite and you can just mention that vs. including the package.py files).

I'll have a chance to experiment with this on 7/30

How hard is it to set up a real test suite implementation of something like this?

I anticipate it will be somewhat challenging but I should be able to sort it out by the end of this week. I think either mock packages provided for Spack testing (which makes it easy to create dependency DAGs) or the mock testing repository (which includes packages which actually install) will be useful. More details on that soon.

@scheibelp I think I have a better clue why C was installing every dependency again, the .spack-db in the upstream was created using an older version of spack (tags/v0.11.0). There are None objects
passed into cmake because of these differences.

  >> 20    CMake Error at /cvmfs/sft.cern.ch/lcg/views/LCG_93/x86_64-slc6-gcc62-opt/cmake/DD4hepConfig.cmake:32 (include):
     21      include could not find load file:
     22    
     23        None/cmake/DD4hep.cmake

After your last commit which looks at the spec hash, spack will correctly link against a few of the packages there, but there are some which will fail because of the differences in the database versions. I will try to make another upstream installation with a newer version of spack, but this points out at least that this method is not backwards compatible with older versions of spack installations.

I've now added tests to check the following:

• Spec.concretize appropriately sets _installed_upstream on packages (when they are in an upstream DB)
• Packages that are registered in an upstream database are not installed locally
• Upstream packages have the expected prefix (e.g. vs. using the directory layout of the local Spack install)
• Upstream packages are not added to the local DB

I'm going to update the PR description with remaining issues.

this points out at least that this method is not backwards compatible with older versions of spack installations.

Thanks for testing that! I have not checked what happens when using a different DB version. Minimally I should add an explicit check to assert that the upstream DB is the same version (vs. failing silently).

@scheibelp I have generated another spack installation in my upstream using a newer version of spack (commit c79cd5f) and ran this feature locally pointing at it. Every package is correctly flagged as being installed upstream, except for one of them:

==> Installing gaudi
==> Error: RuntimeError: Unable to locate python command in None/bin

/build/paul/spack/var/spack/repos/builtin/packages/python/package.py:419, in command:
        416            command = 'python'
        417        else:
        418            msg = 'Unable to locate {0} command in {1}'
  >>    419            raise RuntimeError(msg.format(self.name, self.prefix.bin))
        420
        421        # The python command may be a symlink if it was installed
        422        # with Homebrew. Since some packages try to determine the

I am trying to investigate the differences between the local installation of this package, which I have been able to link against, and this one above installed upstream which is failing. There are differences between the two index.json databases but it is not clear to me why and what difference exactly is causing this instance to pick up a None object.

Edit:
It appears that gaudi is the only package which, in my case, when installed upstream, has a different hash than when installed locally. Spack will therefore try to install it again. That should cause no problem if self.prefix.bin weren't None/bin, so there is clearly something wrong happening.

Edit2:
The reason spack cannot find the installation directory of python is that python is externally installed in this case, and the prefix path appears as None. When spack sees the spec for gaudi has a different hash in upstream, it attempts to build it and fails like above. I have replicated this locally by creating a spack installation in one directory A with package gaudi depending on externally installed python, manually changing the hash of this package inside .spack-db/index.json and then using a spack instance in directory B pointing to A as its upstream. This latter instance will recognize the difference in hashes, attempt to build the package and fail because it can't resolve the external dependency python. Alternatively, the same can be achieved it I remove this package from directory A. It appears that packages which are externally installed inside an upstream instance of spack will not be picked up by this feature.

#8772 (comment) and #8772 (comment) are now addressed. Also, note that 5c4330b is addressed by beb9a31.

I decided as part of this that YamlDirectoryLayout should defer entirely to Spec.prefix when constructing the metadata path, which meant a test in directory_layout had to be updated to update spack.store.layout (see the layout_and_dir fixture update in 2e39fdf).

I'm really excited to see this merged and I am ready to try it again. I know the yaml files changed format several times during the development process. Is there an official documentation location yet? I didn't see anything about spack chain at https://spack.readthedocs.io.

@chissg @drbenmorgan

@scheibelp I want to try this out, is any documentation available?

@sknigh

I want to try this out, is any documentation available?

not in the actual docs, it is all here for now (the PR description and #8772 (comment)). Although docs are definitely worth adding and I can add a PR for something official in the next few days.