Have you considered requiring an installed version of OCaml in the testsuite and using a local install (in a _install sub directory for instance)? This would avoid the various if in Makefiles which will sometimes be tedious to maintain. All we would need to do is set OCAMLLIB to point to the local directory; I believe that's the only place we hard-code the installation directory.

BTW, I welcome the various Printexc.record_backtrace false to ensure better reproducibility, I always forget to unset OCAMLRUN from my environment. However, I think it would be even more robust to clear this variable, at least by default and explicitly do Printexc.record_backtrace true in the few tests that are testing backtraces.

@diml - I hadn't, but I'm not convinced by that strategy, given the reasons for doing this. I actually think the number of ifeqs required isn't too bad (I was expecting more Makefiles to need patching, to be honest) - but it does seem to be important to ensure that the testsuite is definitely not run in the build tree, because it allows a mis-ported testsuite Makefile to pick up the build tree by accident. I'd also worry that given that we're wanting to test the install targets by doing this that we'd be moving the logic into make install (it's effectively overriding PREFIX, BINDIR, LIBDIR, etc.) and risk failing to test the very logic that we want to check?

I debated clearing OCAMLRUNPARAM completely too, but decided to leave that for the future. I also like that those tests - which care about exceptions in their output - are explicitly turning it off (I think that the ones which test backtraces either do this or manually add b to OCAMLRUNPARAM already?). The only reason I fixed the backtrace ones is because the expect tests already do this and because I happened to have OCAMLRUNPARAM=b leftover in my environment! I wasn't certain that there mightn't be a use out there (e.g. GC parameters?) why someone might invoke the testsuite and expect OCAMLRUNPARAM to be respected? I also altered the flambda tests to run with OCAMLRUNPARAM=b, so we should notice if future tests accidentally do the same thing!

OK, the CIs need to finish catching up, but this should finally be ready to go.

If a particularly keen person with access to an OSX environment were to verify that the unwind test works, I think everything would have been tested.

The first thing to note is that the revised CI would have caught the error in #785.

There are implications to some of these changes, which want further consideration:

• The first nine commits are just tidying/refactoring with the addition of a guard against backtraces appearing in result files in future (so the flambda Travis test is executed with OCAMLRUNPARAM=b,v=0 to try to force this issue)
• Switching Travis so that one test is done using the installed testsuite and the other "in-place" means that both branches will be constantly exerised, which should mitigate the chances of bitrot
• Has anyone run the Fortran interop test in a while? I was surprised at the patches needed to build it with gfortran. It seems sensible to keep this test permanently on, although that slows the CI down slightly because gfortran is not installed by default.
• Windows only tests one configuration (at the moment) and it seems sensible for that to be the installed version. However, at the moment a big implication of that is that the installation test now happens in a directory with no space in it. This requires a big overhaul of the testsuite logic, which really should use relative directory names, not absolute ones so readily and it seems worth leaving that until after the ocamltest stuff has taken over everything. Installing OCaml (and being able to build it in a directory with spaces in the name) is a high priority for OPAM infrastructure because Windows 10 user names (and hence OPAM roots) have spaces in them in a default installation. @Chris00 - what're your thoughts on this?

why someone might invoke the testsuite and expect OCAMLRUNPARAM to be respected?

I do, because I sometimes run the testsuite with the debug runtime, which displays GC messages on stderr (which break some tests) unless I also add OCAMLRUNPARAM=v=0 to the environment. So please don't break that.

Two years later, do we want to pursue this PR?

I haven't got time to look up the reference now, but the motivation for this PR was to ensure that we're installing things correctly, and it was a follow-up to a real bug, even if I've somewhat stupidly not referenced it in the original summary.

A required artefact was not being installed, so the theory was to be able to ensure all the tests pass when OCaml is installed as well as from its build tree as a way of checking for that in future. IIRC the intention was never that a user would install the testsuite, it allowed CI to install it. The alternative would be to clean the tree completely and instruct the testsuite to run on the installed compiler - that may now be much easier to implement thanks to ocamltest than it was with the old Makefile infrastructure.

Offering a way to run the testsuite on an installed compiler may also help some of our friends packaging OCaml when we move things around - Arch has/had a broken 4.09 ocaml-compiler-libs package because we reorganised the middle-end sources (see ocaml/opam#4013). The self-contained-toplevel would have highlighted that packaging error.