Summary

Fixes #17537

Red Knot's CLI sometimes hangs after encountering a panic. This was due to non-determinsim related to how panics are propagated by rayon and Salsa.

• Our main loop spawns a job to run the check command to remain responsive to Ctrl + C commands and incoming file watcher changes
• This thread is spawned using rayon::spawn. Rayon terminates the entire process if the task scheduled with rayon::spawn panics (we don't have any other panic handling today).
• This thread calls db.check right away which calls project.check, but is wrapped in a salsa::Cancelled::catch. I assumed that this only catches cancellations because of a pending write (e.g. a file watcher change) but it turns out, that it also catches panics if thread A depends on a query running in thread B and thread B panics.
• The actual checking happens in project.check where we use a thread pool and spawn a task for every file.
• project.check uses rayon::scope: Unlike rayon::spawn, it propagates panics instead of terminating the process. However, it propagates an arbitrary panic if two threads panic (which is the case if thread A depends on a query in thread B and B panics).

What happened is that sometimes rayon propagated the Cancelled::PropagatedPanic (the panic raised by salsa in thread A that depends on the panicking query running in thread B) panic over the actual panic in thread B, which then got swallowed by our Cancelled::catch wrapper inside db.check.

We should probably change Salsa to use a different mechanism to handle thread-dependent panics because this is a logical error, whereas a pending write is not.

This PR fixes the hang by repeating the Cancelled::catch inside each spawned thread. This has the advantage that we propagate the real panic from thread B and never the placeholder Cancelled::PropagatedPanic from thread A (which doesn't contain any useful debug information). More specifically, we now catch panics inside check_file_impl and create a diagnostic for them.

error: panic: Panicked while checking `/Users/micha/astral/ecosystem/hydpy/hydpy/core/devicetools.py`: `dependency graph cycle querying try_metaclass_(Id(250b7)); set cycle_fn/cycle_initial to fixpoint iterate`
info: This indicates a bug in Red Knot.
info: If you could open an issue at https://github.com/astral-sh/ruff/issues/new?title=%5BRed%20Knot%20panic%5D, we'd be very appreciative!

Ideally, we'd capture the backtrace too but that's a) more complicated and b) mostly empty in production builds.

Test Plan

I ran red knot on hydpy for a couple of minutes and I couldn't reproduce the hang anymore (normally reproduces after 30s or so)