ping @alalazo who prefers when the database is the single source of truth instead of config files

@haampie as I was thinking about this more last night, I realized this would require additional functionality to add the externals from the cray manifest file to the packages configuration in addition to the database. This might prove tricky, since the cray externals from the manifest can have dependency information, and we don't currently allow dependency information from externals in the packages config.

Allowing dependencies on externals looked challenging last time I looked at it, but I can take another stab at it now. The other option would be some more complex mapping between the externals in the database and the externals in the config. For example, externals in the db could store the yaml block in which they were defined, which would make identifying the correct external possible even if there are multiple satisfying externals

I didn't think of the cray manifest at all, that's a bit of a blocker.

@becker33 can't we keep track of how externals were added and use that to determine reuse or not? That could also help with undoing an import from a cray config file.

I agree that migrating entries from db to config works, but doesn't really sound great.

I don't understand what you meant with that foo ^bar/hash example, that doesn't trigger reuse, it's just a constraint?

I'll add my thoughts on this:

1. This is in line with reuse concretization: allow externals from remote when locally configured #35975 which was merged. Can't we merge this as is as an improvement on Linux?
2. On Cray, afaik, externals can be added from a JSON file that is on the system (and is structured in a way that is very similar to a DB) and from config files. We can monitor the JSON file AND the config files for the time being.
3. Migrating stuff from DB to packages.yaml is a very bad idea as a) we'll end up with a less powerful DB b) we'll get back entries that are the "concretization" of something that was in the DB already c) packages.yaml will be cluttered¹

As for a more long term way forward, I still advocate for an intermediate DB to store externals. That will:

1. Solve the one to many mapping we have now on externals (right now the same external on disk has potentially many different hashes in the DB)
2. Allow the same semantic we use here and in reuse concretization: allow externals from remote when locally configured #35975 (instead of satisfiability of an external abstract spec, we need to check if the external we want to reuse is in the intermediate DB - and don't consider it if it isn't)
3. Allow for automatic updates / cleaning procedures for externals (the intermediate DB is under Spack control, so can be updated whenever a change in configuration etc. is detected. Based on the intermediate DB we can have commands that update, with prompt, the user DB too).
4. (This is for us, not for users) Remove the weird concretization logic and ad-hoc rules we have for externals during concretization

We can monitor the JSON file AND the config files for the time being.

we don't keep track what file was used, so maybe easier to mark externals from config as such so we can apply different reuse conditions.

Idk how to deal with reuse from build cache with imported cray manifest entries, that doesn't work now 🤔 currently I think that is fine; shouldn't affect many users, and if we allow reuse there, it can cause the same issue fixed by #35975

users don't think of externals being "installed in the database" at all,
their externals are "installed on the system"

☝🏻 this times 💯.

Maybe this is crazy talk, but I wish externals could be implemented as an in-memory artifact rather than registering them in a database. I'm sure in practice this is hard to do, but in principle that seems closer to reality. An external is not a spack artifact, just a spack registration. When system externals shift under us they usually affect low level specs that require a lot of rebuilding across the DAG. Often times the best thing to do is just delete everything and start from scratch. (Especially if uninstalling by hash is the other option 😝 ).

If the system changed then spack's database is not valid anymore wrt to the external even if nothing changed in spack. AND the only way spack will be able to detect these changes ATM is if the users tell it.

So, what do we do here @becker33 / @alalazo?

I noticed that cray entries set spec.origin = "external-db" which ends up as part of an install record, so we do keep track after all.

Keep the PR as is but allow reuse of records marked external-db w/o corresponding config sounds sensible to me then.

Pushed that change