Skip to content

Commit 6949841

Browse files
cloudonlycloudonly
authored
docs: average ate.workerpool.workers across replicas before summing (agent-substrate#3193)
The aggregation note from agent-substrate#3056 tells readers to take the `max` across ate-api-server replicas and then `sum`. As agent-substrate#3162 shows, the replicas read separate worker caches and can disagree for a few seconds: one reports `{ate_worker_state="partial"} 3` while the other still reports `{ate_worker_state="idle"} 3`, so `max by (ate_worker_state)` keeps both series and the pool sums to 6 workers. Take the average across replicas instead. It keeps the sum across the states equal to the pool size; the counts can be fractional for those few seconds. Updates the registry note and the observability guide. No code change: the locust collector already reads a single process's series, and nothing else aggregates this instrument. Fixes agent-substrate#3162 AI use: Claude Code analyzed the issue and wrote the change and this description. I reviewed the diff. Codex rebased the branch onto main and resolved the documentation conflicts. --------- Signed-off-by: cloudonly <[email protected]>
1 parent 427f2fb commit 6949841

2 files changed

Lines changed: 15 additions & 10 deletions

File tree

‎docs/guides/observability.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -290,7 +290,7 @@ Agent Substrate emits foundational OpenTelemetry system and server metrics to mo
290290
| `atelet.snapshot.size` | atelet | histogram | uncompressed allocated size in bytes of each snapshot image written during checkpoint (`st_blocks * 512`, excluding sparse holes in microVM `memory-ranges`) (labels `file.name`, `ate.template.atespace`, `ate.template.name`) |
291291
| `ate.workerpool.desired_workers` | atecontroller | up/down counter | number of worker pods requested for a WorkerPool, from `spec.replicas` (labels `ate.workerpool.namespace`, `ate.workerpool.name`) |
292292
| `ate.workerpool.ready_workers` | atecontroller | up/down counter | number of worker pods currently ready for a WorkerPool, from `status.readyReplicas` (labels `ate.workerpool.namespace`, `ate.workerpool.name`) |
293-
| `ate.workerpool.workers` | ateapi | up/down counter | live worker count per pool, split by state (`idle`/`partial`/`at_capacity`/`unschedulable`) and sandbox class to provide fleet capacity and saturation at a glance. Every ateapi replica reports the whole fleet: take the `max` by pool namespace, pool, state, and sandbox class first, then `sum` (example query below the table) |
293+
| `ate.workerpool.workers` | ateapi | up/down counter | live worker count per pool, split by state (`idle`/`partial`/`at_capacity`/`unschedulable`) and sandbox class to provide fleet capacity and saturation at a glance. Every ateapi replica reports the whole fleet: take the `avg` by pool namespace, pool, state, and sandbox class first, then `sum` (example query below the table) |
294294
| `ate.actor.lifecycle.operation.duration` | ateapi | histogram | how long each actor operation (create/resume/suspend/delete/revert) takes and whether it failed (`error.type` present = failure, absent = success); labeled by operation, template, pool (`ate.workerpool.namespace` + `ate.workerpool.name`), sandbox class, snapshot kind on resume, and snapshot fidelity on resume and suspend; already-running resume no-ops are not recorded so the histogram tracks actual activations, not router traffic |
295295
| `ate.scheduler.assignment.duration` | ateapi | histogram | time of one attempt to assign a worker to an actor, with one record for each resume (ateapi retries an attempt that loses a race for a worker and records only the last attempt), with the outcome (`assigned` / `no_capacity` / `error`), the assigned pool (`ate.workerpool.namespace` + `ate.workerpool.name`) and sandbox class to catch scheduling latency and capacity starvation problems |
296296
| `ate.actor.restore.duration` | atelet | histogram | how long each phase of a restore takes on the worker node, which is where cold-start latency actually goes once ateapi hands off (labels `ate.snapshot.phase`, `ate.snapshot.kind`, `ate.snapshot.fidelity`, `ate.template.atespace`, `ate.template.name`, `ate.sandbox.class`) |
@@ -313,11 +313,11 @@ For `ate.workerpool.desired_workers` and `ate.workerpool.ready_workers`:
313313
* **Autoscaling Control Loop & Anti-Windup**: `desired - ready > 0` sustained beyond a few minutes indicates undelivered capacity due to node pool exhaustion, quota limits, or stuck worker pods, serving as anti-windup input for demand-reactive capacity scaling.
314314

315315
For `ate.workerpool.workers`:
316-
* Every ateapi replica reports the whole fleet, so a plain `sum` counts each worker once per replica ([#3055](https://github.com/agent-substrate/substrate/issues/3055)). Take the `max` across replicas first, keeping the pool namespace, pool, state, and sandbox class, then `sum`. Workers per pool:
316+
* Every ateapi replica reports the whole fleet, so a plain `sum` counts each worker once per replica ([#3055](https://github.com/agent-substrate/substrate/issues/3055)). The replicas read separate worker caches and can disagree for a few seconds: a worker that changed state sits under its old state on one replica and its new state on the other ([#3162](https://github.com/agent-substrate/substrate/issues/3162)). Take the `avg` across replicas first, keeping the pool namespace, pool, state, and sandbox class, then `sum`. A `max` would keep that worker under both states and the summed pool outgrows its real size. The average keeps the sum equal to the pool size; the counts can be fractional while the replicas disagree. Workers per pool:
317317

318318
```promql
319319
sum by ("ate.workerpool.namespace", "ate.workerpool.name") (
320-
max by ("ate.workerpool.namespace", "ate.workerpool.name", "ate.worker.state", "ate.sandbox.class") (
320+
avg by ("ate.workerpool.namespace", "ate.workerpool.name", "ate.worker.state", "ate.sandbox.class") (
321321
{"ate.workerpool.workers"}
322322
)
323323
)

‎docs/metrics/registry/metrics.yaml‎

Lines changed: 12 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -846,13 +846,18 @@ groups:
846846
brief: The number of workers in each pool, divided by state and by sandbox class.
847847
note: >
848848
ateapi reads this value with a callback. Each ateapi replica reports the
849-
whole fleet, and the default install runs two replicas. Thus a sum across
850-
the replicas counts each worker once for each replica. Take the maximum
851-
across the replicas first, by pool namespace, pool, state and sandbox
852-
class, and add after that. Within one replica you can add the counts
853-
together. The sum across the states is the size of the pool. The sum
854-
across the pools is the size of the fleet. Thus this instrument is an
855-
UpDownCounter
849+
whole fleet from its own worker cache, and the default install runs two
850+
replicas. Thus a sum across the replicas counts each worker once for
851+
each replica. The caches can disagree for a few seconds, and a worker
852+
that changed state then sits under its old state on one replica and
853+
under its new state on the other. Take the average across the replicas
854+
first, by pool namespace, pool, state and sandbox class, and add after
855+
that. A maximum would count that worker twice, once under each state.
856+
The average keeps the sum across the states equal to the size of the
857+
pool, and the counts can be fractional while the replicas disagree.
858+
Within one replica you can add the counts together. The sum across the
859+
states is the size of the pool. The sum across the pools is the size of
860+
the fleet. Thus this instrument is an UpDownCounter
856861
and not a gauge. ateapi sets every state to 0 for each known pool. Thus a
857862
full pool or an empty pool reports 0. An absent series would stop an alert
858863
on idle == 0.

0 commit comments

Comments
 (0)