Skip to content

Commit 0f5a684

Browse files
committed
atelet: pin the multi-actor stats fold and keep CPU baselines across pending sweeps
Ateoms now return one GetActiveWorkloadStats entry per hosted actor. The atelet sweep already folded every entry, but nothing tested it above one actor per worker, and reviewing the fold found CPU accounting defects. A pending entry (boot, restore, teardown, or a guest the ateom did not reach) never wrote the actor's CPU baseline, and lastCPU is rebuilt each sweep, so measured -> pending -> measured lost the whole interval. Pending entries now carry the baseline forward; only actors absent from every response drop it. Carrying it unconditionally could overwrite a value another worker measured in the same sweep (a restore in flight: the source still reports the actor), charging the interval twice on the next sweep. The pending branch now writes only if no worker measured the actor this sweep. A CPU decrease was charged as the whole new value. That is right for a cgroup counter, which restarts at zero on restore, but not always for a guest-agent counter: a FULL or golden restore resumes a guest at that guest's counter, a cold boot restarts it at zero, and a container exit lowers the sum. A guest-agent decrease cannot tell these apart, so it now charges nothing and re-baselines. That loses the usage from the event to the first sample. Main counted it correctly only after a cold boot (a DATA restore, or a Run after a crash); in every other case main overcounted it. An epoch that starts above the last value stays undetectable, as the proto notes, and carrying the baseline extends that limit to restores that span a pending sweep. Tests cover the fold over a multi-actor worker, per-actor baselines, per-entry events, baseline carry across a pending sweep, the decrease rule per source with and without a pending sweep, and the two-worker restore case in both fold orders. That last one orders the probes on the fold itself, via the connection close that follows it, rather than a sleep. The poll floor (50s) is one actor's worst-case micro-VM read, 25 containers at 2s each. Sweeps are sequential, so it bounds guest-agent load, not overlap. The RPC timeout (55s) was documented as covering that single-actor worst case. With several actors per worker, the micro-VM ateom caps its read at a 45s budget and reports unreached guests as pending, so the timeout stays constant. The comments now say so.
1 parent 10a1bfb commit 0f5a684

2 files changed

Lines changed: 353 additions & 51 deletions

File tree

‎cmd/atelet/statspoller.go‎

Lines changed: 39 additions & 47 deletions
Original file line numberDiff line numberDiff line change
@@ -48,24 +48,19 @@ import (
4848
const workerPoolLabel = "ate.dev/worker-pool"
4949

5050
// minActorStatsPollInterval is the floor a configured poll interval is clamped
51-
// to. It is the worst-case duration of one ateom's sweep on the micro-VM
52-
// runtime -- maxActorContainers containers at statsCallTimeout each, constants
53-
// that live with that runtime -- so a shorter interval could start a new poll
54-
// into a guest agent still serving the previous one.
51+
// to: one actor's worst-case micro-VM read, maxActorContainers (25) containers
52+
// at statsCallTimeout (2s) each. Sweeps never overlap, so it bounds
53+
// guest-agent load, not correctness.
5554
const minActorStatsPollInterval = 50 * time.Second
5655

57-
// statsRPCTimeout bounds one ateom's GetActiveWorkloadStats call. It has to
58-
// cover the ateom's own worst-case sweep (see minActorStatsPollInterval);
59-
// anything still unanswered past that is a stuck socket, not a slow guest.
56+
// statsRPCTimeout bounds one ateom's GetActiveWorkloadStats call. The micro-VM
57+
// ateom sets its statsSweepBudget (45s) below this timeout and reports guests
58+
// it did not reach as pending, so the call does not grow with actor count.
6059
const statsRPCTimeout = 55 * time.Second
6160

62-
// statsSweepConcurrency bounds how many ateoms one sweep probes at once. The
63-
// interval floor protects a single guest from overlapping polls; probing
64-
// DISTINCT ateoms concurrently puts one probe on each guest, so the only
65-
// stacking the cap prevents is on atelet itself -- without it, a node of
66-
// stuck-but-accepting sockets would hold one hung call per ateom for the full
67-
// statsRPCTimeout. With it, such a node degrades the sweep to
68-
// ceil(n/statsSweepConcurrency) timeouts instead of n.
61+
// statsSweepConcurrency bounds how many ateoms one sweep probes at once, one
62+
// call per ateom whatever it hosts. It caps how many stuck sockets can hold a
63+
// hung call open on atelet at the same time.
6964
const statsSweepConcurrency = 8
7065

7166
// workerPoolListTimeout bounds the per-sweep pod list that fetches worker
@@ -74,11 +69,10 @@ const statsSweepConcurrency = 8
7469
const workerPoolListTimeout = 10 * time.Second
7570

7671
// clampActorStatsPollInterval enforces the floor on a nonzero configured
77-
// interval, warning rather than obeying: an interval below the worst-case
78-
// sweep would pile overlapping polls onto the same guest agent.
72+
// interval, warning rather than obeying.
7973
func clampActorStatsPollInterval(ctx context.Context, configured time.Duration) time.Duration {
8074
if configured > 0 && configured < minActorStatsPollInterval {
81-
slog.WarnContext(ctx, "actor-stats-poll-interval below the worst-case sweep; clamping",
75+
slog.WarnContext(ctx, "actor-stats-poll-interval below the floor; clamping",
8276
slog.Duration("configured", configured), slog.Duration("clamped_to", minActorStatsPollInterval))
8377
return minActorStatsPollInterval
8478
}
@@ -142,13 +136,9 @@ type statsPoller struct {
142136
// exist, the same bound lastCPU keeps.
143137
cachedPools map[string]workerPoolRef
144138

145-
// lastCPU is the previous sweep's cpu_usage_usec per actor uid, the
146-
// baseline the next sweep's deltas are computed against. Only the sweep
147-
// loop touches it (under collect's mutex), and entries for actors a sweep
148-
// did not see are dropped at its end -- an actor that leaves the node
149-
// stops occupying memory here, and one that comes BACK later simply
150-
// re-baselines. Empty after an atelet restart, so the first sweep
151-
// contributes zero deltas: an undercount, never an overcount.
139+
// lastCPU is the last measured cpu_usage_usec per actor uid, the baseline
140+
// for the next delta; read-only during a sweep and replaced whole at the
141+
// end. Pending actors keep their entry, unseen actors are dropped.
152142
lastCPU map[string]uint64
153143
}
154144

@@ -242,8 +232,9 @@ func (p *statsPoller) tick(ctx context.Context) {
242232
// worker pods (nothing garbage-collects them eagerly), ateoms that have made
243233
// their directory but not yet listened, and workers torn down mid-sweep. The
244234
// no-sample answers are equally routine: an empty samples list is an idle
245-
// worker, and a pending entry (source UNSPECIFIED) is a boot or restore in
246-
// progress -- both are skips by the RPC's own contract.
235+
// worker, and a pending entry (source UNSPECIFIED) is a workload the ateom
236+
// could not measure (boot, restore, teardown, or an unreached guest), which
237+
// adds nothing but keeps its CPU baseline.
247238
func (p *statsPoller) collect(ctx context.Context) map[templateKey]*templateAggregate {
248239
entries, err := os.ReadDir(p.ateomsDir)
249240
if err != nil {
@@ -291,17 +282,20 @@ func (p *statsPoller) collect(ctx context.Context) map[templateKey]*templateAggr
291282
return nil
292283
}
293284

294-
// One entry per workload the ateom is executing; empty when it is
295-
// available. Today that is at most one entry -- multi-actor
296-
// workers will grow it, and nothing here assumes otherwise.
285+
// One entry per workload the ateom is hosting; empty when it is
286+
// available.
297287
for _, sample := range resp.GetSamples() {
298288
if sample.GetSource() == ateompb.StatsSource_STATS_SOURCE_UNSPECIFIED {
299-
// A pending workload: attributed but not measured (boot,
300-
// restore, teardown, or a transition underneath the
301-
// ateom's read). It contributes nothing anywhere -- not
302-
// to the aggregates (sampled_actors keeps meaning
303-
// "measured"), not to the CPU baselines, and no event --
304-
// exactly as the old no-sample answers behaved.
289+
// Pending: hosted but not measured, so it adds nothing
290+
// and its CPU baseline carries forward. A measured value
291+
// from another worker this sweep (restore in flight) wins.
292+
if last, ok := p.lastCPU[sample.GetActorUid()]; ok {
293+
mu.Lock()
294+
if _, measured := seenCPU[sample.GetActorUid()]; !measured {
295+
seenCPU[sample.GetActorUid()] = last
296+
}
297+
mu.Unlock()
298+
}
305299
continue
306300
}
307301
p.eventEmitter.emit(ctx, eventKindPeriodic, sample, pools[podUID])
@@ -323,22 +317,20 @@ func (p *statsPoller) collect(ctx context.Context) map[templateKey]*templateAggr
323317
agg.memoryCurrentBytes = addSat(agg.memoryCurrentBytes, sample.GetMemoryCurrentBytes())
324318
agg.memoryWorkingSetBytes = addSat(agg.memoryWorkingSetBytes, sample.GetMemoryWorkingSetBytes())
325319

326-
// The counter increase this sample represents. A decrease means
327-
// the epoch reset underneath us (the cgroup source restarts at
328-
// zero on restore), so the new value IS the usage since the
329-
// reset. A sample with NO baseline charges nothing and only
330-
// records one: atelet cannot tell a new actor from its own
331-
// restart, and charging the whole epoch-so-far would re-count
332-
// hours of usage the previous atelet already counted, as one
333-
// artificial spike. The bounded price is that every actor's
334-
// boot-to-first-poll usage goes uncounted -- the events channel
335-
// carries per-actor precision.
320+
// The counter increase since the baseline. On a decrease, a
321+
// cgroup counter restarted at zero, so the new value is the
322+
// usage since then. A guest-agent decrease is ambiguous (a
323+
// resume at another guest's value, a restart at zero, or a
324+
// container exit), so it charges nothing. A sample with no
325+
// baseline charges nothing: atelet cannot tell a new actor
326+
// from its own restart.
336327
cpu := sample.GetCpuUsageUsec()
337328
seenCPU[sample.GetActorUid()] = cpu
338329
if last, ok := p.lastCPU[sample.GetActorUid()]; ok {
339-
if last <= cpu {
330+
switch {
331+
case last <= cpu:
340332
agg.cpuDeltaUsec = addSat(agg.cpuDeltaUsec, cpu-last)
341-
} else {
333+
case sample.GetSource() == ateompb.StatsSource_STATS_SOURCE_CGROUP:
342334
agg.cpuDeltaUsec = addSat(agg.cpuDeltaUsec, cpu)
343335
}
344336
}

0 commit comments

Comments
 (0)