Tests are failing on matching str subclasses, particularly syaml_str—allowing for those with issubclass() should not take away from the performance gains.

shaved off half the runtime

of what exactly? I see a 3.5% speedup in setup time, and a 0.4% speedup overall when comparing spack solve --timers --fresh cmake before/after this PR. That's good, but curious where you get the "half as a long" from.

That's confusing. I had been getting some pretty reliable results from switching over to develop and then running the profile script which concretized icu4c cxxstd=17. This change was turning 48 seconds => 24 seconds, and 30 seconds => 15 seconds. I'm now not getting that at all from develop, on spack spec or the profile script. I had been getting 50-second spack runtimes very reliably and it was part of why I thought this was so important to follow up on.

Like, I have two snakeviz windows open right now showing 30s/48s runtimes for spack spec from yesterday. I truly have no clue what changed.

Kind of upset about this. I was at first still getting about an 11% improvement (after running several times to alleviate e.g. page cache issues) but that's not happening anymore either and I'm getting about the performance you describe as well. I had been getting 50-second runtimes extremely reliably and I don't know where those were coming from now.

Solver performance is deterministic for fixed input, but can easily be 2x slower if some new fact is added or the same facts are presented in a different order.

For benchmarking of the Python side you can best look at "setup" reported by spack solve --timers, which is deterministic enough.

For end-to-end performance testing it's best to use a statistical test, see this repo: https://github.com/spack/spack-benchmark. It's very time consuming to run a full benchmark with multiple samples per spec (i.e. multiple shuffles of input data), so in your case I would just look at "setup" time.