Would testing COSX results against the implementation in Orca 5 make sense? Of course the grid dependence makes such things tricky.

Would testing COSX results against the implementation in Orca 5 make sense? Of course the grid dependence makes such things tricky.

Practically speaking, this should be doable, but one would need to ensure that as many factors between the two implementations as possible are standardized and made the same. Some of these factors (e.g., screening tolerances) shouldn't be that hard, but others (like grids, as you mentioned) would be quite a bit harder and could be quite a thorn in our side for comparing the two implementations.

This question opens up the discussion to another approach I was thinking for this PR - using separate reference energies for COSX and LINK in the scf5 test, rather than comparing against canonical or DF energies. I am rather neutral on that matter, and am definitely open to doing so; but there are some concerns that should be considered with the separate reference energies approach. These are concerns that might not matter as much now; but the scf5 test will undoubtedly be expanded as CompositeJK and new separate J and K algorithms are added to Psi4, and these concerns might pop up then:

1. Using separate reference energies for each CompositeJK JK build combination, for each O2 multiplicity and SCF reference, could quickly make scf5 quite unwieldy and messy.
2. Against what do we compare reference energies? In some cases, it won't be too big of a problem (as with here, where we can compare against ORCA), but I'm willing to bet that there are certain combinations of J and K algorithms that CompositeJK will eventually be able to do, that might not actually be present in other codes to compare against.

Of course, these concerns are more specific to CompositeJK testing, so maybe they're an issue that can punted down the road for when scf5 gets updated again.

Is the methodology by which COSX and LinK tested acceptable? Unfortunately, the reference energies used in scf5 are either canonical (i.e., non-DF) or DF, neither of which fully describe the COSX and LINK methods. The approach I took is to compare each COSX and LINK energy to the corresponding canonical energy with an atol of 1E-4. Any feedback on this particular approach to testing COSX and LINK would be appreciated.

I think it'd be most useful to continue scf5 as a regression test. That is, record new refs for the in-between-conv-and-df algorithms collected at tight convergences, then check them against usual 6 decimal places. Then we'll know in future if code changes affect the algorithms.

How are you generating the samples/ files? If it's any other way than autogenerated via the docs build, please don't go to the trouble. We can catch the changes en masse before release.

I strongly agree with Lori about having this as a regression test rather than an accuracy test. Put the new, approximate energies with all the others. That should be a good indicator to others working with the SCF system of what accuracy to expect.

Is the methodology by which COSX and LinK tested acceptable? Unfortunately, the reference energies used in scf5 are either canonical (i.e., non-DF) or DF, neither of which fully describe the COSX and LINK methods. The approach I took is to compare each COSX and LINK energy to the corresponding canonical energy with an atol of 1E-4. Any feedback on this particular approach to testing COSX and LINK would be appreciated.

I think it'd be most useful to continue scf5 as a regression test. That is, record new refs for the in-between-conv-and-df algorithms collected at tight convergences, then check them against usual 6 decimal places. Then we'll know in future if code changes affect the algorithms.

How are you generating the samples/ files? If it's any other way than autogenerated via the docs build, please don't go to the trouble. We can catch the changes en masse before release.

I strongly agree with Lori about having this as a regression test rather than an accuracy test. Put the new, approximate energies with all the others. That should be a good indicator to others working with the SCF system of what accuracy to expect.

All right, sounds like a plan! I will do this, then!

Also, Lori, to answer your question about samples, I have been editing them manually. Thank you for letting me know about their autogeneration!

I did a couple more things here:

1. At the suggestion of the reviewers, I added individual reference energies for the current composite methods (which were acquired at tight tolerances - 1e-10 for D and E convergence, and 1e-14 for ERI screening). The COSX and LinK methods now test against these corresponding reference energies at the normal 1e-6 atol.

2. With 1) in mind, I made more sweeping changes as well. First, the reference energies are now kept in a dictionary rather than spread out across different variables. This improves readability and should facilitate the addition of new methods, Composite or otherwise, to the scf5 testing suite. Additionally, composite methods are tested in a loop per test case. This will facilitate the testing of new Composite methods added to Psi4, as will happen in the future.