Thanks for the detailed reply! I apologize in advance for the even more wordy response.

I'm currently creating mirrors and patching as needed (which works great 95% of the time --- even if a bit of fiddling is required for the git-based packages. The extreme cases often get messy and start to look more like dark arts than controlled builds. If there's a need for help on this, I'm interested in improving this for usability, reproducibility, and to make lives inside the airgap better.

For the third item, there are a few categories of packages I've come across, each that may have varying levels of difficulty to address:

A) Packages that run curl/wget to some discrete set of additional resources or patches on their own for convenience and make the installation easier for users. These are usually pretty easy to handle. As long as you can figure out what is being downloaded where, you can add a resource and patch out the download. I ran into this while packaging HAIL and I believe RDKit will attempt this under certain conditions.

B) Packages that are written in languages that have first-class package managers that retrieve external resources on l-demand as part of the build process. Both Rust and Go do this, making building those packages from mirrors difficult. I ran into this with the SingularityCE package and some rust packages I can't recall right now. NodeJS/NPM too, but that's not as much of an issue I suppose.

C) Packages that run some sort of explicit large recursive dependency installation through a package manager or script, such as calling pip as part of the install. I believe this was what I was running into in PyTorch. In some ways, this is similar to (B), but it seems different to me as it's not a corr feature of the language (one could argue).

For (A), it's pretty straightforward as part of packaging to handle.

For (B), I suspect it could be handled by ensuring all of the specific dependencies are available in Spack. It's hard to figure out what they all are given the "quietness" of it working outside an airgap, but the tooling is standard and could probably be audited statically. We may be able to have some sort of a Cargo.toml checker for rust, for example.

For (C), these are often the hardest since they're just occurring (possibly inconsistently) as part of the build process. It would be nice to know when this is happening, but it'd be even nicer to have a way to capture what's requested and either mirror, review, or white list it.

I was considering creating an extension that would turn on a slim HTTP(S) proxy that would be configured so that (at least well-behaved) tool chains would pasd through it. It would be more of an admin tool than anything else. It could be activated within spack to monitor installations and see if they retrieve external resources. It could also be used to create a list of the URLs and checksums to retain some of Spack's integrity checking. Lastly (and most interesting to me), it likely could also cache everything downloaded during an installation and create a resource bundle that could be re-served through the HTTP(S) proxy within the airgapped environment. That would at least provide a path forward for mirroring those extra-tricky packages.