One alternative to #20068 (possibly complementary?) is to attempt to hack away at the build system for clingo (CMake) so that it will avoid introducing a requirement on a C++14 compiler and we can reliably bootstrap it from source in arcane environments.

Rationale

Removing the futuristic and unnecessary C++14 requirement doesn't break anything (yet)! It currently doesn't require more than a small patch:
cosmicexplorer/clingo@a1db68e

Description

I found the above clingo commit built and was successfully importable with python 2.6.7, and editing the CMakeLists.txt as in the HEAD commit avoids producing any command lines requiring a C++14 compiler!

So only two changes are necessary to keep this clingo patchset up to date and buildable on e.g. RHEL/CentOS 7 as per #20069! If we hooked that up into spack, it would likely be a lot more stable / a lot less tech debt than a fork (which I was previously worried about for some version).

Additional information

I think #20068 seems 100% orthogonal to this approach -- rather, developing a reliable (reproducible?) method of bootstrapping from sources seems like the key to both of these issues. If we can trust our robust script, we can easily re-ship updated binaries for multiple architectures as in #20068, while ensuring via CI that our bootstrap script avoids correctness (or performance?) regressions as we try to self-bootstrap.

I think from any self-bootstrapping process for a tool like spack that runs itself from source, I as a user might (eventually) have these requirements:

1. Users will clearly understand when a bootstrap occurs (can possibly cancel it if performed as a transaction?).
2. Users will be able to point spack to their own internal toolchain binary distribution server (servers?) without any configuration.
   • Spack could (like pants does) assume such a server's endpoints are laid out corresponding to a hierarchical filesystem tree.
   • The docs clearly and unambiguously state how to do this.
3. There is a tag created specifically to track issues with binary resolution.

General information

It looks like spack already exposes the entry point for cron jobs/etc to clean up space with spack gc, and if we can make the bootstrap "look like a normal spack install", we may be able to get control of that like this for free.

Relatedly, here's a really interesting discussion thread from the pants repo on invalidating external binaries: pantsbuild/pants#10768 (comment)

Aaaaand ... this is still broken in realistic ways. Namely, upgrading or downgrading a binary found here previously will result in the same output from this rule and thus not trigger a re-run of rules depending on this data, which is wrong. The only cases currently covered by this fix from the list below are 1 and 2:

An applicable binary is added to the search path.
An applicable binary is removed from the search path.
An applicable binary on the search path is modified (upgraded or downgraded most likely).

To handle 3 the contents of the discovered binaries need to be hashed (And even that is not enough! Dynamically linked libraries could change the binaries output ... but we'll need to whistle past the graveyard - that level of detail can really only be addressed by always running in an image / using a local docker / podman / runc / crun type solution). This could be done by storing it in the CAS as a hack, but we won't / can't generally use the binary stored in the CAS so that just adds more (see make_process_volatile comment) storage bloat. Ideally we'd ask an engine intrinsic to get us a fingerprint without storing the object. It could also be done by just hashing the resulting absolute path in python rule code at the expense of potentially heavy IO inhibiting parallelism. On my machine, hashing my biggest binary (/usr/bin/packer @ 154M) takes ~400ms in pure python code and hashing the currently more typical python binary (@ 19M) takes ~50ms.

I'd like to defer handling case 3 to a follow-up if folks are amenable.