atelet: poll ateom workload stats into template-level metrics - #961

AteomDialer caches one grpc.ClientConn per worker pod UID in a 256-entry
LRU, but the cache had no eviction function: a conn pushed out to make
room was forgotten, never closed. grpc does not reclaim an un-Closed
conn -- measured against a dead socket, each one holds 4 goroutines, and
the channel idle timeout later parks only one of them, so ~3 goroutines
and their buffers leak per eviction for the life of the process.

Today evictions need >256 distinct worker pods dialed since atelet
started, so the leak is a slow drip reset by restarts. The stats poller
proposed in agent-substrate#961 would sweep every ateoms/ directory (including the
stale ones nothing garbage-collects) through this cache every minute,
turning the drip into hundreds of leaked conns per tick on long-lived
nodes.

Close evicted conns via the cache's eviction hook. The trade is that an
RPC still in flight on a conn that aged to the LRU tail now fails
visibly instead of completing on a leaked conn; that needs the same
>256-pod churn, and the failure is retryable.

AteomDialer caches one grpc.ClientConn per worker pod UID in a 256-entry
LRU, but the cache had no eviction function: a conn pushed out to make
room was forgotten, never closed. grpc does not reclaim an un-Closed
conn -- measured against a dead socket, each one holds 4 goroutines, and
the channel idle timeout later parks only one of them, so ~3 goroutines
and their buffers leak per eviction for the life of the process.

Today evictions need >256 distinct worker pods dialed since atelet
started, so the leak is a slow drip reset by restarts. The stats poller
proposed in agent-substrate#961 would sweep every ateoms/ directory (including the
stale ones nothing garbage-collects) through this cache every minute,
turning the drip into hundreds of leaked conns per tick on long-lived
nodes.

Close evicted conns via the cache's eviction hook. The trade is that an
RPC still in flight on a conn that aged to the LRU tail now fails
visibly instead of completing on a leaked conn; that needs the same
>256-pod churn, and the failure is retryable.

AteomDialer caches one grpc.ClientConn per worker pod UID in a 256-entry
LRU, but the cache had no eviction function: a conn pushed out to make
room was forgotten, never closed. grpc does not reclaim an un-Closed
conn -- measured against a dead socket, each one holds 4 goroutines, and
the channel idle timeout later parks only one of them, so ~3 goroutines
and their buffers leak per eviction for the life of the process.

Today evictions need >256 distinct worker pods dialed since atelet
started, so the leak is a slow drip reset by restarts. The stats poller
proposed in agent-substrate#961 would sweep every ateoms/ directory (including the
stale ones nothing garbage-collects) through this cache every minute,
turning the drip into hundreds of leaked conns per tick on long-lived
nodes.

Close evicted conns via the cache's eviction hook. The trade is that an
RPC still in flight on a conn that aged to the LRU tail now fails
visibly instead of completing on a leaked conn; that needs the same
>256-pod churn, and the failure is retryable.

Fixes agent-substrate#1009.

AteomDialer caches one grpc.ClientConn per worker pod UID in a 256-entry
LRU, but the cache had no eviction function: a conn pushed out to make
room was forgotten, never closed. grpc does not reclaim an un-Closed
conn -- measured against a dead socket, each one holds 4 goroutines, and
the channel idle timeout later parks only one of them, so ~3 goroutines
and their buffers leak per eviction for the life of the process.

Today evictions need >256 distinct worker pods dialed since atelet
started, so the leak is a slow drip reset by restarts. The stats poller
proposed in agent-substrate#961 would sweep every ateoms/ directory (including the
stale ones nothing garbage-collects) through this cache every minute,
turning the drip into hundreds of leaked conns per tick on long-lived
nodes.

Close evicted conns via the cache's eviction hook. The trade is that an
RPC still in flight on a conn that aged to the LRU tail now fails
visibly instead of completing on a leaked conn; that needs the same
>256-pod churn, and the failure is retryable.

Fixes agent-substrate#1009.

Per-container CPU and memory limits for micro-VM actors. First slice of
#752.

An `ActorTemplate` container can cap its own cpu and memory so it cannot
starve or kill its siblings in the same actor. A container that exceeds
its memory limit is OOM-killed on its own; the rest of the actor is
unaffected.

```yaml
sandboxClass: microvm
containers:
  - name: trainer
    resources:
      limits: {memory: 1500Mi}
  - name: sidecar
    resources:
      limits: {memory: 256Mi, cpu: "0.2"}
```

#752 asks for per-container device selection as well. This PR does the
cpu and memory half; devices are the remaining half, and when they land,
GPU assignment should follow the same field rather than staying
actor-wide as it is today.

## Why micro-VM only

The two sandbox classes support opposite halves of #752. Micro-VM actors
have a real guest kernel, so each container gets its own cgroup and the
limits bind. gVisor applies cgroup limits at the sandbox level: one
sentry backs every container in the actor, so a per-container cgroup is
created and then stays empty (google/gvisor#190). Measured on a running
actor, the workload container's cgroup reported `memory.current=0` while
all 20 sandbox processes sat in the pause leaf. A template that sets
`resources` with `sandboxClass: gvisor` is rejected at admission.

## How a limit travels

`ActorTemplate` → CEL validation → `ate-api-server` resolves each
`resource.Quantity` once → `ateletpb` → atelet writes OCI
`linux.resources` → `ateom-microvm` merges kata's defaults and checks
the guest envelope → `SpecToAgentPB` → kata agent → guest cgroup.

The limit is carried as a standard OCI field rather than a
substrate-private concept, so a runtime that gains per-container
enforcement picks it up without new plumbing.

## Composition with #679

#679 sizes the sandbox itself from `ActorTemplate.spec.resources`: guest
RAM and vCPUs for a micro-VM, the sentry for gVisor. This PR subdivides
that sandbox. The two are different fields and complementary layers.

They collided in one place. #679 applied the actor-level size to
**every** container's OCI spec, overwriting the per-container limits
atelet writes into the same field, so a container asking for 64Mi
silently received the whole guest. This PR removes that call on the
micro-VM path. A container now gets a cgroup limit only when it declares
one; an undeclared container is bounded by guest RAM, which is the real
ceiling. On micro-VM the stamp never reached the guest at all:
`SpecToAgentPB` carried neither `Memory` nor `CPU.Quota`, so it stopped
at the bundle's `config.json`. Running the pre-PR pipeline shows
`Quota=50000 Memory=2Gi` on disk arriving as `Quota=0 Memory=<nil>` on
the wire.

gVisor is untouched. There the actor-level size is applied to the
`pause` container, and since one sentry backs every container, that is
the only cgroup that binds.

Two consequences worth calling out for reviewers of #679:

- The `sizing.SandboxSize` threaded into `buildActorContainers` and
`ensureKataCompatibleSpec` had no remaining reader, so it and the
`guestSize` helper are removed. `resolveGuestMemMiB` still sizes the VM
and still rejects a declared limit too small to boot.
- `internal/sizing`'s package doc claimed both runtimes shared
`ApplyToOCISpec` for container cgroups. That is now gVisor-only, and the
comments say so. No code in that file changed.

`checkResourceEnvelope` runs after `resolveGuestMemMiB`, so it validates
the per-container sum against the post-reserve guest rather than the
SandboxConfig default. Its error now names
`spec.resources.limits.memory` (or `.cpu`) when the actor declared a
size, and `SandboxConfig` when it did not.

`internal/e2e/suites/sizing` is unaffected: it asserts on `num_cpu` and
`mem_total_bytes`, both of which come from VM sizing, and only logs the
cgroup files.

## Verified on hardware

Same template, run twice, one commit apart on a micro-VM actor:

| | before | after |
| :--- | :--- | :--- |
| `hog_ovl/memory.max` | `max` | `67108864` (the declared 64Mi) |
| `bystander_ovl/memory.max` | `max` | `max` |
| hog allocates 128MB | survived | OOM-killed |
| bystander | alive | alive |

The bug in between: `SpecToAgentPB` converted only `Devices` and
`CPU.Shares` out of `Linux.Resources`, so a memory limit reached the
bundle's `config.json` and was dropped on the way to the agent. Every
unit test passed and the on-disk spec was correct while the feature did
nothing.

## Notes for review

- `ContainerResources` deliberately does not reuse
`corev1.ResourceRequirements`, which also carries `requests` and
`claims`. There is no scheduler inside an actor to hint at, and for
memory a soft request cannot express "must have this much to come back
at all". #679 reached the same conclusion from the other direction and
now rejects `spec.resources.requests` and `.claims` at admission.
- `Limits` is a bounded named type. A `MaxProperties` marker on the
field lands at the wrong schema level for a named map, and without a
bound the CEL cost estimator rejects the whole schema. This CRD is close
to its schema-wide CEL cost ceiling; a rule that iterates containers and
parses quantities exceeds it by more than 100x and takes the existing
rules down with it.
- A cpu limit below `10m` is raised to `10m`: the kernel rejects a CFS
quota under 1ms.
- `SpecToAgentPB` drops a non-positive quota and a zero period rather
than forwarding them. A non-positive quota means unlimited in OCI but
was sent as a literal zero, which the guest applies as no CPU at all; a
non-nil zero period overwrote the CFS default set for a live quota. Both
now read a spec the way `cpuLimitMillis` does, so the envelope check and
the conversion agree on the same input.
- `mergeKataResources` fills the gaps kata's defaults cover rather than
allowlisting known fields, so a field it does not recognise survives the
merge. That holds only as far as the merge: `SpecToAgentPB` converts
`Devices`, `Memory` and `CPU` only, so `Pids`, `BlockIO`,
`HugepageLimits` and `Network` stop at the ttrpc boundary. Nothing sets
them today.

## Known gaps

- An OOM-killed container is not reported above the guest. The actor
stays `STATUS_RUNNING` and nothing records which container died. Raised
on #550. Note that #961 settled workload stats at the sandbox level and
deliberately does not attribute per container, so surfacing which
container died needs a separate signal: the guest cgroup's
`memory.events`, not the stats path.
- A container's writable rootfs is a guest tmpfs, so filesystem writes
are charged to its memory limit and a container writing more than its
limit to `/tmp` is OOM-killed. Documented in `api-guide.md`; separating
the two budgets needs an `ephemeral-storage` field.
- Over-subscription across containers is caught when the actor starts,
not at apply time. An admission-time sum check is not implementable
within the CEL cost budget.
- The atelet hop that attaches resources to the bundle spec has no test:
replacing `ctr.GetResources()` with `nil` leaves the whole suite green.
Closing it needs an imagecache fixture for `prepareOCIDirectory`.
- Restored actors inherit the golden's cgroups rather than applying
their own spec. Correct today only because `ActorTemplateSpec` is
immutable.
- The GPU section of `api-guide.md` still says `ActorTemplate` has no
per-container resource fields, which this PR makes false. Correcting it
is left to the devices half of #752, which is what will change the GPU
behaviour that paragraph describes.

---

- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR

---------

Signed-off-by: Eliran Wolff <[email protected]>

Part of #896 (Phase 1 of #550): the events channel — per-actor usage
samples as structured log events. With this, both halves of #174's
cardinality split exist: template-level metrics in the TSDB (#961), and
everything carrying actor/atespace identity here, in the log store.

## The events

One JSON record per executing actor per sweep on atelet's stdout, riding
the poller's existing probe — no extra RPC load, and one knob governs
both channels (`--actor-stats-poll-interval 0` disables the subsystem).
Idle workers emit nothing: an idle fleet is silent by design.

Records use the same label vocabulary as actorlog's lifecycle events
(`ateattr.ActorLogLabels`, including the GCE
`logging.googleapis.com/labels` spelling) — so one Cloud Logging filter
on `labels."ate.actor.uid"` returns an actor's container logs, lifecycle
transitions, and usage samples interleaved. Identity comes solely from
each sample's echo, per the stats RPCs' attribution contract.
Measurements ride as payload fields (`kind`, class, source, the four
numbers, `observed_at_unix_nano`); the `kind` field (today always
`periodic`) lets future kinds join without reshaping the record.

Events also carry the `ate.workerpool.namespace/name` pair the metric
labels already carry, resolved by the sweep's own pod list, so a
pool-level metric spike can pivot to the actors behind it — pool
membership lives on the worker pod and is unrecoverable from logs once
the pod is gone, so it must be stamped at emission. An unresolved pod
emits without the pair, following the metric channel's rule.

## Scope

Earlier revisions also took first/final lifecycle samples from the
Run/Restore/Checkpoint handlers. Per the review discussion on
suspend/resume latency, those are descoped from this PR: it now touches
no lifecycle path at all. The bracket design (including taking the final
sample inside ateom's CheckpointWorkload and echoing it in the response)
moves to a follow-up under #896.

## Isolation

The poller dials its own short-lived connection per probe
(`dialAteomStats`) and never touches the lifecycle RPCs' cached clients
— after review on #961 the isolation is structural.

## Validated live on ate-dev

Periodic events read back from Cloud Logging via
`labels."ate.actor.name"`, one per executing actor per minute, with
`ate.actor.uid` confirmed promoted into `LogEntry.labels` (filterable) —
the claim only production could prove.

Consumer note: aggregate freely in the log store, but a log-based metric
built over these events must label only by the bounded set (template,
class, source, pool) — promoting actor identity into a metric label
would reintroduce exactly the cardinality #174 keeps out of the TSDB.

Part of #896. Part of #550.

…unter (#1479)

Fixes #1478.

One failed pod list per sweep grouped that tick's samples without
`ate.workerpool.*` labels. Gauges blink and heal; the CPU counter does
not: an OTel counter binds increments to their attribute set, so a
single flap re-homes that sweep's CPU increase onto a newly-born
pool-less series that coexists with the pooled one forever —
pool-filtered `rate()` dips on every flap, a phantom unlabeled series
bursts exactly during apiserver trouble, and cardinality doubles per
flapped group for the process lifetime.

## Fix

A pod's pool is **immutable for the pod's lifetime** (the
`ate.dev/worker-pool` label is stamped at creation; pods never move
between pools), so caching resolutions per pod UID carries zero
correctness risk. `resolveWorkerPools`:

* lets fresh resolutions win (harmless either way, given immutability),
* falls back to `cachedPools` when the list fails outright or misses one
pod,
* rebuilds the cache restricted to the pods whose ateom directories
exist this sweep — pruning departed pods and bounding memory with the
same prune-to-the-living pattern `lastCPU` uses.

The residual unlabeled case shrinks to "pod first seen while the
apiserver is unreachable", which heals on the next good list — and is
pinned by a test as the documented behavior.

## Testing

* List flap: seed the cache, fail the next list, assert the samples stay
on the pooled label set with no pool-less split.
* Prune: an entry whose ateom directory is gone leaves the cache; the
live pod's entry stays.
* Residual case: a pod first seen during an outage groups without pool
labels (one group, not an error).
* `hack/verify-all.sh` green.

Part of #896, toward #550.


## Note: cache lifetime is tied to ateom directories, which nothing
cleans up

`cachedPools` evicts an entry on the first sweep after its ateom
directory disappears — but today **nothing removes `ateoms/<podUID>/`
when a worker pod is deleted** (the only cleanup anywhere is an ateom
clearing its own stale socket at boot; directories accumulate until node
replacement). Repo archaeology found no decision behind this: the
directory set had no reader until the stats sweep (#961) promoted it
into a discovery registry, so GC was never needed before.

Consequences for this PR, sized honestly:

* **Correctness is unaffected.** Pod UIDs are never reused, and a
retained entry is only consulted when a sample arrives from that UID — a
dead socket produces none.
* **Memory is a rounding error on a pre-existing cost.** The cache is
bounded by the directory count (~150 bytes per departed pod, ~1.5 MB per
ten thousand); each stale directory already costs more than its cache
entry does — a dial attempt every sweep plus debug-log noise, on main
today, independent of this PR.
* Eviction is deliberately **not** tied to the pod list — "the list
stopped returning this pod" is exactly the flap signal this fix exists
to distrust. The filesystem stays the retention authority, consistent
with being the discovery authority.

The durable fix is directory GC (atelet holds both halves of the
evidence each sweep: the dir set and the node's pod list; "absent from
the pod list AND socket dead for N consecutive sweeps → remove" is safe
against half-born ateoms and transient list failures). That is a
pre-existing issue being filed separately; when it lands, cache eviction
follows it automatically with zero changes here.

…te#1678)

Part of agent-substrate#1677 — the graceful-shutdown half; the atelet-side janitor for
ungraceful exits is tracked there.

Nothing removes `ateoms/<podUID>/` when a worker pod goes away, so every
pool rollout leaks one directory (holding a dead `ateom.sock`) per
replaced worker — and since the stats sweep enumerates that directory as
its discovery registry (agent-substrate#961), each leak costs a dial+probe per sweep,
forever.

## Change

A `defer os.RemoveAll(ateomDir)` in both runtime mains, registered right
after the `MkdirAll` that creates the directory. ~7 lines each plus the
comment carrying the safety argument.

**Why this is safe:**
* **Same-pod container restart:** kubelet serializes containers — the
replacement is not started until this process, and therefore this defer,
has exited. The remove cannot race the successor's `MkdirAll`.
* **Shutdown ordering:** the SIGTERM path drains, `GracefulStop`s,
`Serve` returns, `do` returns — the defer runs after the listener is
already closed, so nothing can be dialing the socket it removes.
* **Boot-failure paths:** error returns before `Listen` also run the
defer; the pod restarts and re-creates the directory. Harmless.

**What it deliberately does not cover:** SIGKILL after the grace period,
OOM kills, node crashes — defers don't run there by nature. That residue
is the janitor's job (agent-substrate#1677); this change shrinks the janitor's caseload
to exactly those, since rollouts — the dominant growth driver —
terminate gracefully.

## Testing

Both runtime mains are untested boot plumbing (no unit seam exists for
`do()`); verified by linux builds and test-compiles of both packages.

**Live-verified on the ate-dev GKE cluster** (gVisor pool; the micro-VM
change is the identical lines):

* Baseline census of one node found **11 ateom directories with exactly
1 live pod** — ten stale leftovers from prior rollouts, the leak in the
wild.
* Rolled the pool onto an image built from this branch. The outgoing
pre-change pod (`445af5aa…`) **leaked its directory** on termination,
reproducing the bug side-by-side with the fix.
* Gracefully deleted a pod running this branch (`5618282c…`): its
directory **was removed** on shutdown, while the pre-change pod's leaked
directory remained in the same census — before/after behavior on the
same node, same listing.