Thinking Path

• bug(stability): persistent-host crashes — bootstrap.py double-fork breaks launchd + state.db FD leak #1458's bootstrap and state.db FD leak fixes are already shipped; the remaining production concern is the process-alive / port-listening / HTTP-unhealthy wedge.
• The issue comments called out independent hardening that would make that final failure mode diagnosable/recoverable: deep health, an accept-loop heartbeat, supervisor watchdog docs, and more FD headroom.
• This PR keeps the change narrowly scoped to those persistent-host safeguards instead of speculating about an unreported post-v0.50.272 root cause.
• The deep probe exercises the same state surfaces that made the UI unusable during the wedge (STREAMS_LOCK, sidebar/session state, projects, and state.db) while plain /health stays cheap.
• Closes #1458 once merged because the concrete fixed bugs are already covered and this adds the final residual hardening path.

What Changed

• Added accept-loop heartbeat counters to QuietHTTPServer and exposed them under /health.accept_loop.
• Added /health?deep=1 readiness checks for the streams lock, session/sidebar path, project state, and Hermes state.db connectivity.
• Made health return 503 with status: "degraded" when the stream lock probe cannot complete, so watchdogs do not treat a wedged process as healthy.
• Raised the server process RLIMIT_NOFILE soft limit to 4096 on supported platforms as defense in depth for persistent hosts.
• Documented a launchd/systemd-compatible HTTP watchdog recipe and the deep-health/FD-limit behavior in docs/supervisor.md.
• Added regression coverage in tests/test_issue1458_stability_hardening.py.

Why It Matters

The previous fixes removed the confirmed FD leaks and supervisor double-fork loop. This PR addresses the remaining operational gap: a process supervisor can only restart a crashed process, not one that is still alive but no longer serving usable HTTP responses. Deep health plus an accept-loop heartbeat gives persistent Mac mini / launchd deployments a concrete watchdog signal, and the FD soft-limit raise provides diagnostic headroom if a future leak regresses.

Verification

/home/michael/.hermes/hermes-agent/venv/bin/python -m pytest tests/test_issue1458_stability_hardening.py -q
/home/michael/.hermes/hermes-agent/venv/bin/python -m py_compile server.py api/routes.py
/home/michael/.hermes/hermes-agent/venv/bin/python -m pytest tests/test_issue1458_stability_hardening.py tests/test_sprint7.py tests/test_sprint19.py -q
git diff --check
/home/michael/.hermes/hermes-agent/venv/bin/python -m pytest tests/ -q

Result:

RED check before implementation: 3 failed as expected (missing accept_loop, missing deep checks, missing _raise_fd_soft_limit).
Targeted final: 3 passed in 1.61s.
Related health/security targeted: 35 passed in 1.86s.
Full suite: 4287 passed, 2 skipped, 3 xpassed, 1 warning, 8 subtests passed in 403.36s (0:06:43).
git diff --check: clean.

Manual verification, if applicable:

• Inspected the final diff for unrelated files and credential patterns; no secrets or unrelated work included.

UI media, if applicable:

• Not applicable — server health/readiness and docs change only; no UI/UX surface changed.

Risks / Follow-ups

• /health?deep=1 intentionally does more work than plain /health; watchdogs should use reasonable timeouts/intervals rather than polling it aggressively.
• The residual RST+LISTEN wedge has not been freshly reproduced on post-v0.50.272 builds. This PR adds observability/recovery hardening rather than claiming a new distinct root cause was found.
• If a new post-fix wedge report arrives with FD counts/thread samples, it should be tracked as a new targeted issue with that diagnostic bundle.

Model Used

AI assisted.

• Provider: OpenRouter
• Model: openai/gpt-5.4-mini
• Notable tool use: GitHub issue inspection via gh, isolated git worktree, strict RED/GREEN pytest loop, full local pytest suite, git diff/security review, PR creation via gh.

Closes #1458