feat(services): add continuous HTTP readiness checks - #4230

Summary

Exposed sandbox HTTP services now report continuously observed health. Services check listener responsiveness by default; configuring a readiness path requires a 2xx response, and the Codex app-server example uses /readyz.

Related Issue

Closes #4229

Changes

• Probe the service's existing loopback target through the supervisor relay every five seconds, with a one-second timeout, three failures to become unhealthy, and one success to recover.
• Return cached health from service APIs and CLI output. Keep observations ephemeral and return unknown for stale observations, changed configuration, and unavailable or replaced runtimes. Health does not gate routing or restart applications.
• Add optional HTTP readiness configuration to service exposure and sandbox creation, CLI flags, and Rust, Go, and TypeScript SDKs. Probes send no credentials, follow no redirects, and consume only response headers.
• Preserve conclusive target failures across gateway replicas without retrying or evicting healthy peer channels. Mark internal probe relays so routine attempts and expected bridge cancellation are logged at debug, and retain failure counts across delayed scans.
• Update the Codex app-server example, published service documentation, and CLI skill.

Service contract

A service remains an exposed loopback HTTP port identified by workspace, sandbox, and service name (empty name selects the unnamed endpoint). Readiness uses that same target port and relay; it does not require another listener or route.

Configuration and response

ExposeServiceRequest and create-time SandboxServiceExposure accept an optional readiness_check. The persisted ServiceEndpoint returns the effective configuration. Existing expose/get/list RPCs return health alongside the endpoint and URL:

message HttpReadinessCheck {
  string path = 1;
}

message ServiceEndpointResponse {
  ServiceEndpoint endpoint = 1;
  string url = 2;
  ServiceHealth health = 3;
}

message ServiceHealth {
  ServiceHealthState state = 1;
  google.protobuf.Timestamp last_checked_time = 2;
  string message = 3;
  optional uint32 http_status_code = 4;
}

enum ServiceHealthState {
  SERVICE_HEALTH_STATE_UNSPECIFIED = 0;
  SERVICE_HEALTH_STATE_UNKNOWN = 1;
  SERVICE_HEALTH_STATE_HEALTHY = 2;
  SERVICE_HEALTH_STATE_UNHEALTHY = 3;
}

An empty configured path resolves to /. Paths must begin with /, be at most 1,024 bytes, and contain no query, fragment, whitespace, or alternate host. Omitting the check when updating an existing service preserves its configuration. Removing the check requires deleting and recreating the endpoint in this version.

Continuous observation

• Checks run while the sandbox is ready, on a five-second timer with a one-second deadline covering relay setup and response headers. Each replica runs at most 16 checks concurrently; large fleets of slow services can delay the nominal cadence, and stale reads remain unknown. They send no application credentials, follow no redirects, and do not consume response bodies.
• State starts unknown. One passing check makes it healthy; three consecutive failed checks make it unhealthy. One passing check recovers immediately. A healthy service stays healthy through the first two failures.
• Configuration changes, replaced runtimes, unavailable supervisor relays, stopped/non-ready sandboxes, and observations older than 15 seconds produce unknown. An inconclusive relay observation resets the consecutive-failure count.
• last_checked_time, message, and optional http_status_code describe the latest attempt. The code is present only when response headers arrived. Clients should branch on state, because thresholds can leave the state healthy while the latest attempt failed.
• Health is an ephemeral cache per gateway replica. Get/list reads do not launch probes or mutate the endpoint's resource version. Gateway restart clears observations; durable mutation replay omits health. Missing health from an older gateway, and an unspecified state, are interpreted as unknown.
• A supervisor-reported target failure is conclusive on both owning and peer gateway replicas. The peer hop preserves the structured RELAY_TARGET_FAILED reason and does not retry it as a transport outage. Older owning gateways keep their previous peer-error behavior.
• Internal probe relays carry an additive RelayOpen.health_check marker. Updated supervisors log routine attempts at debug rather than emitting OCSF open/close events; older supervisors ignore the marker and keep their existing events. Probes still contribute to routed-relay metrics.
• Health is observational: it does not gate routing, change sandbox phase, restart the service, or certify credentials/model access.

The CLI adds service expose --readiness-path PATH and sandbox create --expose-readiness-path PATH (requires --expose). Service output shows Ready / Not ready for configured readiness and Responsive / Unresponsive otherwise. The Codex example sets --expose-readiness-path /readyz.

Testing

• Claude Code (claude-opus-5-5) reviewed the service implementation and bridge cancellation warning fix; no remaining actionable findings in that review.
• mise run pre-commit, TypeScript SDK CI, Go SDK CI, and docs checks pass.
• Server service tests (93), supervisor-session/relay tests (93), storage/schema tests (12), and supervisor runtime session tests (28) pass. CLI service/flag tests and Rust SDK request propagation tests pass.
• Unit regressions cover peer refusal without retry/channel eviction, infrastructure errors remaining unknown, old wire payloads, and failure counts across delayed scans.
• Docker readiness and authorization tests pass against rebuilt gateway/supervisor binaries: mise exec -- e2e/with-docker-gateway.sh cargo test --manifest-path e2e/rust/Cargo.toml --features e2e-docker --test service_bearer_passthrough -- --nocapture. Includes delayed 128 KiB response bodies, quiet sandbox audit and gateway warning logs, health transitions, closed targets, stopped sandboxes, unchanged routing, and authorization passthrough.
• Started the installed Codex app-server on loopback and confirmed /readyz returns HTTP 200 without credentials.

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Architecture docs updated (if applicable)