Fixes #17444

Summary

This PR addresses two related "session looks hung" failure modes in the PI embedded runner:

1. Compaction retry waits can block the session lane with no aggregate timeout.
2. SIGUSR1 in-process restart can proceed while embedded compaction is still in-flight, leaving session write locks held by the previous lifecycle.

What changed

1) Bound compaction retry waits (aggregate timeout)

• Add waitForCompactionRetryWithAggregateTimeout() in run/compaction-retry-aggregate-timeout.ts.
• In run/attempt.ts, replace the unbounded await abortable(waitForCompactionRetry()) with a 60s aggregate timeout.
• On aggregate timeout:
  • set timedOutDuringCompaction = true
  • log a warning and proceed using the existing pre-compaction snapshot selection logic
• Defensive normalization: non-finite aggregateTimeoutMs values fall back to 1ms rather than producing NaN timeouts.
• Timer cleanup uses timer !== undefined (not truthiness) to handle fake-timer environments where handle can be 0.

This prevents long compaction retry waits from freezing the session lane.

2) Harden SIGUSR1 restart against in-flight embedded runs/compaction

• Increase default SIGUSR1 deferral budget: DEFAULT_DEFERRAL_MAX_WAIT_MS 30s → 90s.
• Increase gateway run-loop drain budget: DRAIN_TIMEOUT_MS 30s → 90s.
• On SIGUSR1 restart in the gateway run loop:
  1. Abort compacting embedded runs (best-effort, surgical — only runs where handle.isCompacting() is true)
  2. Drain both active command-queue tasks and active embedded runs (up to 90s)
  3. If drain times out, abort all embedded runs (best-effort) and proceed

3) Exception-safe abort handling

• All three abortEmbeddedPiRun() code paths (single session, compacting-only, abort-all) now wrap handle.abort() in try/catch.
• A thrown abort handler no longer prevents server.close() from executing during SIGUSR1 restart drain.
• Failed aborts are logged at warn level with the session ID and error.

Why this is safe

• The aggregate timeout only affects cases where compaction retry waits would otherwise stall; normal compaction runs are unchanged.
• Restart already implies disruption; aborting compacting runs is a targeted best-effort to ensure session locks are released.
• Exception guards on abort are strictly additive — they only prevent cascading failures.
• Changes are localized to restart/compaction wait paths.

Tests

New:

• src/agents/pi-embedded-runner/run/compaction-retry-aggregate-timeout.e2e.test.ts — aggregate timeout helper
• src/agents/pi-embedded-runner/runs.e2e.test.ts — surgical abort + waitForActiveEmbeddedRuns

Updated:

• src/cli/gateway-cli/run-loop.test.ts — drain timeout constant
• src/infra/infra-runtime.test.ts — deferral timeout constant

Note: e2e tests require -c vitest.e2e.config.ts (excluded from default config).

pnpm -s lint
pnpm -s format:check
npx vitest run src/agents/pi-embedded-runner/ src/cli/gateway-cli/run-loop.test.ts src/infra/infra-runtime.test.ts --reporter=verbose
npx vitest run -c vitest.e2e.config.ts src/agents/pi-embedded-runner/runs.e2e.test.ts src/agents/pi-embedded-runner/run/compaction-retry-aggregate-timeout.e2e.test.ts --reporter=verbose

Notes / follow-ups

• This PR does not change per-attempt compaction safety timeouts (EMBEDDED_COMPACTION_TIMEOUT_MS); if desired, we can follow up with a smaller per-attempt ceiling and/or a compaction re-entry cooldown once we have more production data.
• Refactor commits (dedupe compaction failure, dedupe assistant delta parsing, reuse isPidAlive) are included to reduce code duplication touched during the investigation.

Greptile Summary

This PR fixes two related "session hung" failure modes in the PI embedded runner:

• Aggregate timeout for compaction retry waits: Adds a 60s aggregate timeout to waitForCompactionRetry() in the embedded run attempt loop, preventing unbounded blocking of the session lane. On timeout, the run proceeds with the pre-compaction snapshot. The new waitForCompactionRetryWithAggregateTimeout helper uses Promise.race with proper timer cleanup and defensive normalization for non-finite timeout values.
• Hardened SIGUSR1 restart against in-flight compaction: The gateway run loop now surgically aborts compacting embedded runs before draining, waits for both active tasks and embedded runs in parallel (up to 90s), and falls back to aborting all runs if the drain times out. The deferral budget and drain timeout are increased from 30s to 90s to match observed production compaction durations.
• Exception-safe abort handling: All abortEmbeddedPiRun code paths now wrap handle.abort() in try/catch, preventing a thrown abort handler from cascading into the shutdown sequence.

Changes are well-localized to restart/compaction paths, and existing behavior for normal compaction runs is unchanged. Test coverage is thorough with new e2e tests for the aggregate timeout helper and surgical abort modes.

Confidence Score: 5/5

• This PR is safe to merge — changes are defensive, well-scoped to restart/compaction paths, and thoroughly tested.
• The changes are strictly additive safety improvements: an aggregate timeout that only fires when compaction would otherwise stall indefinitely, exception guards on abort that only prevent cascading failures, and targeted abort of compacting runs during restart. Normal compaction flows are unaffected. All new behavior paths have dedicated e2e tests, existing tests are updated to match the new timeout constants, and the code follows established patterns in the codebase (polling drain, try/catch abort, Promise.race with cleanup). No logical or security issues found after thorough analysis of all 9 changed files and their related imports.
• No files require special attention

_{Last reviewed commit: 6d7ef5a}