My mental model of how the Go runtime handles panics is a bit incomplete, but my understanding was that an un-recovered panic normally unwinds right to the top of the goroutine's call stack and only exits if it gets to the top without being recovered.

If I'm correct about that, is it true to say that the only correct way to use this PanicHandler is as a defer PanicHandler() as the very first statement in a new goroutine, so that we can avoid this os.Exit inhibiting any other pending defers? If so, I think it would help to be explicit about that in the docstring above.

I'm also curious as to whether exiting here sufficiently mimics what the Go runtime would do with an unrecovered panic if there are still other non-panicing goroutines running at the time of panic. Does the runtime normally just interrupt those wherever they are and exit immediately, or does it first force them to unwind (via something like runtime.Goexit)? Perhaps this is an immaterial question anyway given how we're using this, but my general worry here is that subtle changes to how we react to panics might upset assumptions we've made elsewhere; convincing ourselves that there are no such assumptions would also be fine. 😀

Finally, it occurs to me that if we're now taking control of the panic exit in most codepaths this could be an opportunity to address our long-standing wart that we happened to select the same exit code for terraform plan -detailed-exitcode to signal that there are no changes to make, and so currently a panic there reads to automation as a success with no changes. Although there might still be places where normal panic handling escapes and thus we still exit with code 2 anyway, maybe we'd like to select a different code for the common case where we handle the panic through here, so that automation can still treat it as an error then.

Yes, a panic unwinds to the very top of the call stack within the goroutine, so the handler does need to be the first defer in the stack, so I can add a note about that specifically.

The runtime exits with exit(2) once the call stack has been unwound to the top of the goroutine (which is why I used 2 here too), so any other running goroutines are aborted immediately as the process exits, and should not different significantly from out behavior here. Perhaps a different exit code would work here for the sake of -detailed-exitcode, though differing from the standard runtime code may make this less consistent than we hope.

Given that our goal here is for most panics to be caught by our handler, and for missed ones to be exceptional, I'd personally vote for using a different exit code so that most of the time our behavior isn't ambiguous if we panic during planning, and the ambiguity can therefore be exceptional. The fact that Terraform is written in Go is an implementation detail rather than a contract, so I don't think we get any strong value from being consistent with the Go runtime behavior here.

With that said, it's an opinion loosely held so I'm happy to go ahead without that if you prefer. We can always change the exit status we use here later anyway, if we change our minds, since we don't make any contractural promises about how we report different kinds of errors via exit codes.

I'm not opposed to changing the exit code here, my next idea would be to use the slightly system dependent -1, which is what panicwrap was substituting.