Parent: #1263 · Labels: enhancement, area/infra (auto, manifests under
infra/testgrid/**) · Size: M · Depends on: TG1 (host cluster), TG3 (workers),
TG4a (API), TG5 (publish path)

Goal. Make the running TestGrid instance operable by a small team — the
"operate" half of the epic title. TG7 is the single accountable owner of
operability for the running instance; TG1/TG3/TG4a/TG5 deliver the probes/metrics/signals
it monitors and gates on. Cover liveness/health of the host cluster, the
long-running workers, and the API; a runbook for the common failure modes; and
non-silent surfacing of publish-path and produced-but-unpublished failures.
CLAUDE.md's design principles make observability mandatory (structured logging,
metrics).

Scope.

• Host-cluster + worker health: liveness/readiness for the updater/tabulator/
  summarizer GKE pods (long-running, resumable queue/*.json); a check that a
  worker loop is making progress (queue not wedged), surfaced via structured
  logs/metrics; host-cluster/node-pool health (the TG1 cluster).
• API health: verify TG4a's liveness/readiness probes are wired and monitored.
• Publish-path failure surfacing: the TG5 ingestion job must not silently swallow
  a publish failure — emit a clear failure signal (job failure + structured log) so
  an operator sees a dropped bundle.
• Produced-but-unpublished reconciliation (the dynamic-clusters→TG5→TG3 seam). A lightweight
  check: for each completed UAT run with an emitted bundle (the dynamic-clusters side succeeded),
  assert a TestGrid column materializes within a bounded number of worker cycles;
  if not, emit a warning. This closes the cross-epic gap where a bundle exists but
  never reaches the feed (ingestion failed, queue wedged) — otherwise invisible for
  up to the 1-week alertStaleResultsHours=168 window. TG7 owns detecting this gap
  (the TestGrid epic owns the feed).
• Runbook (docs/contributor/ or infra/testgrid/RUNBOOK.md): stuck queue,
  wedged worker, failed publish, produced-but-unpublished bundle, stale columns,
  config-gen drift, host cluster down / node pool unschedulable — symptoms,
  diagnosis, recovery (resume from queue/*.json, re-run config-gen, re-trigger TG5
  for a missed bundle, recover the host cluster).

Out of scope. Regression alerting (dropped — see Deferred); cost governance
(deferred); the workers/API/publisher/host-cluster themselves (TG3/TG4a/TG2/TG1).

Key files. Probe/health additions consumed from infra/testgrid/k8s/**;
failure-surfacing in the TG5 workflow; new runbook doc (register in the Fern nav if
under docs/).

Tests. Probe-config validation (manifests declare liveness/readiness); a test
that the TG5 job fails loudly on a simulated publish error (no silent success); a
test of the produced-but-unpublished reconciliation (a completed run with no column
within the bounded window emits a warning); runbook builds clean under
fern-docs-ci if it lives in docs/.

Acceptance. Host cluster, workers, and API expose monitored liveness/readiness;
a failed publish surfaces as a visible job failure with a structured log; a
produced-but-unpublished bundle is detected within a bounded number of worker
cycles; the runbook covers stuck-queue / wedged-worker / failed-publish /
host-cluster-outage recovery; TG7 gates on TG3/TG4a/TG5 having emitted the signals it
monitors.

Child of #1263 · staged from docs/design/010-aicr-testgrid.md (TG7).