You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Add runtime revision GC backlog and failure metrics #2744
Follow-up to #2742, which now implements independent runtime revision GC under the existing controller leader election: startup and one-minute periodic sweeps, per-candidate deadlines, failure isolation, durable deletion recovery, and preparation polling while a digest is deleting.
This issue tracks the remaining cleanup observability work:
Expose the pending runtime cleanup count and oldest pending cleanup age.
Count cleanup failures without revision or template labels that create unbounded metric cardinality.
Retain revision-specific context in error logs and document how operators distinguish a growing backlog from a persistently failing Substrate deletion.
Verify metrics recover after successful retries and controller restarts. A failed candidate must remain visible while other candidates are collected.
The lifecycle E2E in #2742 now checks eventual ActorTemplate and golden-actor cleanup through the public status API after releasing the last checkpoint, without a later template change. Controller tests cover startup discovery, retries, failure isolation, deadlines, and cancellation. PostgreSQL tests retain the reference-acquisition and UID-safety coverage.
A concrete operational case from validation is a Substrate deletion returning Internal: ... runsc state: exit status 128 after an API/DNS outage. Its durable deletion marker survives restart. GC must report this backlog while continuing unrelated preparation and cleanup. Checkpoint-specific recovery remains separate in #2746.
The agreed scope landed in #2808: a pending-cleanup gauge, an attempt-duration histogram with bounded failure labels, retry/restart coverage, and operator documentation.
Oldest-pending age was explicitly deferred during review, and the histogram covers failure rates without a separate failure counter. Closing this as completed.
Activity
[-]Decouple runtime revision GC from pair reconciliation[/-][+]Add runtime revision GC backlog and failure metrics[/+]fix: safely collect runtime revisions in a leader-elected runnable (#…
feat: add runtime revision GC backlog and failure metrics (kagent-dev…
EItanya commentedon Oct 16, 2026
The agreed scope landed in #2808: a pending-cleanup gauge, an attempt-duration histogram with bounded failure labels, retry/restart coverage, and operator documentation.
Oldest-pending age was explicitly deferred during review, and the histogram covers failure rates without a separate failure counter. Closing this as completed.
🤖 AI-assisted triage.