[eval] fix: make a second Ctrl-C exit without the interrupted-shutdown grace - #444

What

A second Ctrl-C during osmosis eval run now skips the interrupted-shutdown waits in both windows where it can land:

• During workflow cleanup (dispatch is still settling cancelled rollouts): the rollout server is stopped with the ordinary short grace (_SERVER_TERM_GRACE_SEC, 5 s) instead of the interrupted-shutdown grace (330 s).
• During server termination (the up-to-330 s wait after cleanup): the second Ctrl-C now reaches the supervisor. It terminates the server's process group again with the ordinary 5 s grace, then SIGKILL, which ends the 330 s wait. Shutdown still stops the tunnel and listener before exiting.
• A third Ctrl-C exits immediately, even mid-shutdown, and may skip tunnel and listener cleanup (test_a_third_interrupt_exits_even_while_shutdown_is_under_way).

Why

_request_cancel handles the second interrupt by printing "exiting now" and raising KeyboardInterrupt. Two gaps let the long grace survive it:

1. asyncio.run still cancels and finishes the supervisor task on the way out, so _stop_rollout_server saw a cancelled task and _cancelled set, and waited up to 330 s for the server. The handler now sets self._force_exit = True before raising, and _stop_rollout_server uses the short grace when it is set.
2. _schedule() removed its SIGINT/SIGTERM handlers before _stop_rollout_server() ran. A second Ctrl-C during the 330 s termination therefore went to Python's default handler: it never reached _request_cancel, and asyncio.run still waited for the termination thread to run out the grace (reported by cubic). After an interrupted dispatch the handlers now stay installed until _execute finishes shutting down. A second interrupt there runs terminate_process_group again with the ordinary grace, and the first wait returns as soon as the group is gone. It does not raise mid-shutdown, because that would skip _stop_tunnel() and leave a cloudflared quick tunnel running; _execute raises KeyboardInterrupt once shutdown completes, so both windows exit the same way.

A single Ctrl-C is unchanged: it still allows the 120 s settle and the 330 s server grace so workflow and sandbox cleanup can finish. A run that finishes or halts without a Ctrl-C keeps the old handler lifetime.

Trade-off: for a Harbor/Docker rollout, a server killed after the short grace may not finish its sandbox cleanup, so containers can be left behind until the orphan reaper cleans them up on the next run. That is what an explicit second Ctrl-C asks for, and a single Ctrl-C remains the clean path.

docs/eval-run-local.md said "A second Ctrl-C exits immediately". It now says what happens: the remaining waits are skipped, the server gets the ordinary five-second grace before it is force-stopped, and the tunnel and listener are still stopped.

Related: #443, #445

How to Test

• uv run pytest tests/unit/eval/local/test_runner_e2e.py -k second_interrupt runs two synchronous tests. Each runs a real supervisor, rollout server and stub model inside asyncio.run and sends real SIGINTs. Before each signal they check that the supervisor's handler owns SIGINT, so a failure can't interrupt pytest itself.
  • test_a_second_interrupt_stops_the_server_with_the_ordinary_grace: the second SIGINT arrives while the workflow is still cleaning up. On main it fails with [330.0] == [5.0].
  • test_a_second_interrupt_during_server_shutdown_cuts_its_grace_short: the second SIGINT arrives while the server is still draining under the 330 s grace. It checks that the server is terminated again with the 5 s grace, that the long wait ends because the server stopped, and that the server's ownership record is cleared. On this PR's first commit (1c8ab86) it fails with no interrupt handler during server shutdown.
  • Both pass 10/10 runs on Python 3.12. tests/unit/eval/local also passes 3/3 runs each on 3.13 and 3.14.
• Existing single-interrupt coverage still passes: test_cancel_settles_cleanup_before_stopping_the_server and test_server_shutdown_only_waits_for_cleanup_when_interrupted.
• uv run pytest -n 4 --dist worksteal: 3679 passed, 1 skipped. On main: 3676 passed, 1 skipped.

Checklist

• PR title follows [module] type: description format
  (labels are derived from it automatically — no need to add them by hand)
• ruff check . and ruff format --check . pass
• ty check --error-on-warning osmosis_ai/ passes; uv run --no-editable pyright --verifytypes osmosis_ai --ignoreexternal has no unexpected errors (see CONTRIBUTING.md for the Harbor baseline)
• pytest passes (new tests added if applicable)
• Public API changes are documented
• No secrets or credentials included

Pyright reports only the documented Harbor baseline errors. git diff --check is clean. This PR has no public API changes. docs/eval-run-local.md is updated to match the new behavior.

Summary by cubic

Makes Ctrl-C handling during osmosis eval run match the documented immediate-exit behavior: a second interrupt exits without the interrupted-shutdown grace (330 s), and a third exits immediately even mid-shutdown.

• The first Ctrl-C is unchanged and still allows the 120 s settle and 330 s server grace for workflow and sandbox cleanup.
• A second interrupt sets _force_exit: during dispatch it raises at once, and during server shutdown it re-terminates the server's process group with the ordinary 5 s grace and lets shutdown finish stopping the tunnel and listener.
• SIGINT handlers now stay installed during shutdown so a second interrupt reaches the supervisor, and _stop_rollout_server uses the short grace when _force_exit is set.
• Trade-off: a server killed after the short grace may not finish sandbox cleanup, leaving containers for the orphan reaper on the next run. docs/eval-run-local.md is updated to describe the three-step exit behavior.

^{Written for commit c3012c4. Summary will update on new commits.}