@amundson @JavierCVilla I have the test working and I made a few improvements. I will submit a PR to Jim's branch once the tests complete.

@chissg @lgarren you might be interested in this PR.

You mentioned:

I see the problem. The system-like db is not being linked into the user-like db. When I try linking them I end up with infinite recursions that I am trying to debug.

This PR is broken since commit f202198 was merged. Since then I am unable to find a way to add the parent database without infinite recursion.

I could look at why this might be occurring this afternoon if that is of interest, but more context would be useful: where is the infinite recursion occurring?

I think I have the infinite recursion problem fixed. Now I can build Jim's example with the expected linking of libraries from the system-like spack install.

Great! I'll be looking through this more later today and expect to have some suggestions/comments by tomorrow afternoon (6/28)

This is the example of infinite recursion I was talking about.

spack -d find
==> In /build/spack/opt/spack:
==> READ LOCK: /build/spack/opt/spack/.spack-db/lock[0:0] [Acquiring]
==> READ LOCK: /build/spack/opt/spack/.spack-db/lock[0:0] [Acquired]
==> READ LOCK: /build/spack/opt/spack/.spack-db/lock[0:0] [Released]
==> 6 installed packages.
-- linux-rhel7-x86_64 / [email protected] -------------------------------
[email protected] [email protected] [email protected] [email protected] [email protected] [email protected]
Traceback (most recent call last):
File "/home/gartung/spack/bin/spack", line 85, in
sys.exit(spack.main.main())
File "/home/gartung/spack/lib/spack/spack/main.py", line 607, in main
return _main(command, parser, args, unknown)
File "/home/gartung/spack/lib/spack/spack/main.py", line 505, in _main
return _invoke_spack_command(command, parser, args, unknown_args)
File "/home/gartung/spack/lib/spack/spack/main.py", line 417, in _invoke_spack_command
return_val = command(parser, args)
File "/home/gartung/spack/lib/spack/spack/cmd/find.py", line 147, in find
tty.msg("In {0}:".format(db.root))
File "/home/gartung/spack/lib/spack/llnl/util/lang.py", line 575, in getattr
return getattr(self.ref_function(), name)
File "/home/gartung/spack/lib/spack/spack/store.py", line 117, in
db = llnl.util.lang.LazyReference(lambda: store.db)
File "/home/gartung/spack/lib/spack/llnl/util/lang.py", line 556, in getattr
return getattr(self.instance, name)
File "/home/gartung/spack/lib/spack/llnl/util/lang.py", line 556, in getattr
return getattr(self.instance, name)
File "/home/gartung/spack/lib/spack/llnl/util/lang.py", line 556, in getattr
return getattr(self.instance, name)
File "/home/gartung/spack/lib/spack/llnl/util/lang.py", line 556, in getattr
return getattr(self.instance, name)
File "/home/gartung/spack/lib/spack/llnl/util/lang.py", line 556, in getattr
return getattr(self.instance, name)
File "/home/gartung/spack/lib/spack/llnl/util/lang.py", line 556, in getattr
return getattr(self.instance, name)
File "/home/gartung/spack/lib/spack/llnl/util/lang.py", line 556, in getattr
return getattr(self.instance, name)
...
File "/home/gartung/spack/lib/spack/llnl/util/lang.py", line 556, in getattr
return getattr(self.instance, name)
File "/home/gartung/spack/lib/spack/llnl/util/lang.py", line 552, in instance
self._instance = self.factory()
File "/home/gartung/spack/lib/spack/spack/store.py", line 109, in _store
spack.config.get('config:install_hash_length'))
File "/home/gartung/spack/lib/spack/spack/store.py", line 83, in init
parent_db=spack.parents.parent_dbs[-1])
File "/home/gartung/spack/lib/spack/spack/database.py", line 207, in init
self.lock = Lock(self._lock_path)
File "/home/gartung/spack/lib/spack/spack/util/lock.py", line 46, in init
self._enable = spack.config.get('config:locks', True)
File "/home/gartung/spack/lib/spack/spack/config.py", line 512, in get
return config.get(path, default, scope)
File "/home/gartung/spack/lib/spack/spack/config.py", line 409, in get
value = self.get_config(section, scope=scope)
File "/home/gartung/spack/lib/spack/spack/config.py", line 383, in get_config
merged_section = _merge_yaml(merged_section, data)
File "/home/gartung/spack/lib/spack/spack/config.py", line 671, in _merge_yaml
dest[sk] = copy.copy(sv)
File "/usr/lib64/python2.7/copy.py", line 88, in copy
rv = reductor(2)
File "/usr/lib64/python2.7/collections.py", line 181, in reduce
for k in vars(OrderedDict()):
File "/usr/lib64/python2.7/collections.py", line 52, in init
self.__update(*args, **kwds)
File "/usr/lib64/python2.7/_abcoll.py", line 540, in update
if isinstance(other, Mapping):
File "/usr/lib64/python2.7/abc.py", line 141, in instancecheck
subtype in cls._abc_negative_cache):
File "/usr/lib64/python2.7/_weakrefset.py", line 73, in contains
return wr in self.data
RuntimeError: maximum recursion depth exceeded in cmp

I'm still looking at this but a couple of complications I see currently are:

• Dependents should set up RPATHs to point to the true install location of dependencies (I think that will not happen atm because Spec.prefix refers to the local spack)
• There are remaining functions which query one of the db's when they would want to query upstream db's as well, e.g. Spec._install_status (for reporting install status when printing out the Spec DAG)

Other notes:

• It may not be necessary to track separate DirectoryLayouts in addition to databases since the install records of databases store the install path
• I'd prefer to avoid a recursive implementation and instead add logic to query the set of DB's together. Potentially analogous to RepoPath and Repo but an aggregation object may not be required (I'm still thinking that through)

As a side note: this appears to contain unrelated changes to setup-env.sh

Functionality wise this feature seems to work properly. @gartung besides fixing the unit tests what is there left to develop on this PR? We would like to have this feature merged into develop and start using it.

There is one quirk: If a package is installed in multiple prefixes, spack find -p shows the same path in all prefixes.

@gartung what changes would be needed to prevent this from happening? I would like to help you out with this feature in order to make it ready and would benefit greatly from some starting points.

This is where the paths get printed
https://github.com/gartung/spack/blob/dd646d409c1cb5949e8883b5934e6290300e34ef/lib/spack/spack/cmd/__init__.py#L303

Regarding #8545 (EDIT: I was trying to reference a recent comment in this thread but the link is currently not rendering properly and just directing to the initial description)

Hi @paul-chelarescu, I am also working on this. My plan is to get a PR out by the end of tomorrow which (a) will help me (and hopefully you) understand the ramifications of edits made here and (b) will serve as a reference point for how this could be improved. Overall I'm looking for an approach which interacts a bit less with Spack internals and I think that PR will either help guide this there or build on this to achieve that.

See #8772