Skip to content

Commit f6a1dc7

Browse files
authored
ateom-gvisor logs a warning when a running actor's sandbox dies (#2258)
Part of #2212 (o11y side of #2211). If the gVisor sandbox dies after start (sentry crash, kill, OOM kill), nothing notices today. The actor stays RUNNING and its usage samples keep reading like an idle actor's, because the pause cgroup leaf and its counters outlive the processes in it. This makes the death visible: - `cgroupstats` also reads `populated` from `cgroup.events` and `oom_kill` from `memory.events`. If either file is missing, it never reads as a dead sandbox. - The ateom's usage sweep (`sweepUsage`, every `--usage-sample-interval`, 1m by default) logs one WARN per activation when the leaf is empty. Since #2194, `GetActiveWorkloadStats` serves the sweep's cached sample and reads no cgroup, so the check lives in the sweep, the only periodic cgroup read; `measure` hands back the raw `cgroupstats.Sample` for it. The record uses `ateattr.ActorLogAttrs` plus `ate.sandbox.oom_kills`, so filtering on `ate.actor.uid` finds it with the rest of the actor's logs. - Actors with a lifecycle RPC holding or waiting on their lock are skipped, since checkpoint and terminate empty the cgroup on purpose. `actorlock` gets a non-blocking `Busy` for this. An activation whose final record is written (checkpoint, terminate, drain) is no longer swept at all. - `readKeyedUint` walks lines in place instead of using a `bufio.Scanner`, which allocated 4 KiB per file. With the two extra reads, reading 1000 actors' cgroups takes about 9% longer than before and allocates half the bytes. It does not change actor state, RPC responses or the stats samples. Moving the actor to CRASHED is #2211. It also does not catch an app that exits while the sentry stays up. Tested on kind and on GKE (substrate-relay) with the branch image on the egress pool, before the rebase onto #2194 (the check then ran in `GetActiveWorkloadStats`). Unit tests pass at the rebased head; live re-verification on kind is pending: | Case | kind | GKE | |---|---|---| | OOM kill via `memory.max` | 1 WARN, `oom_kills=3` | 1 WARN, `oom_kills=1` | | `kill -9` the sentry | 1 WARN, `oom_kills=0` | 1 WARN, `oom_kills=0` | | Normal suspend | none | none | | Suspend then resume | none | none | | Idle actors | none (5) | none (3) | Each warning showed up on the first sweep after the fault and was not repeated. On GKE it lands in Cloud Logging as severity WARNING with the `ate.*` fields. The stats pipeline was unchanged. One caveat: `oom_kill` counts over the cgroup's lifetime, so a leaf left behind by an earlier activation of the same actor (#2107) carries its count forward. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
1 parent 632e662 commit f6a1dc7

10 files changed

Lines changed: 427 additions & 28 deletions

File tree

‎cmd/ateom-gvisor/hosted.go‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@ import (
2121
"fmt"
2222
"log/slog"
2323
"net"
24+
"sync/atomic"
2425
"time"
2526

2627
"github.com/agent-substrate/substrate/internal/apierror"
@@ -42,6 +43,8 @@ type hostedActor struct {
4243
session *workloadSession
4344
// usage is this activation's epoch, CPU baseline, and latest sample.
4445
usage *ateomstats.Activation
46+
// deadReported keeps the usage sweep to one warning per activation.
47+
deadReported atomic.Bool
4548
}
4649

4750
// admitActor reserves capacity before network setup. An actor that is already

‎cmd/ateom-gvisor/internal/cgroupstats/cgroupstats.go‎

Lines changed: 27 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -21,11 +21,10 @@
2121
// Every read is scoped to a caller-supplied directory rather than a hardcoded
2222
// /sys/fs/cgroup path, which keeps the parsing testable from a fixture tree
2323
// without root or a live sandbox. The package deliberately does not know what a
24-
// sandbox is; it reads four numbers out of five files.
24+
// sandbox is; it reads what the kernel reports in a handful of files.
2525
package cgroupstats
2626

2727
import (
28-
"bufio"
2928
"bytes"
3029
"errors"
3130
"fmt"
@@ -65,6 +64,16 @@ type Sample struct {
6564
// to this cgroup (see ateomcgroup.Delegate, which enables controllers
6665
// best-effort and carries on when one cannot be enabled).
6766
CPUUsageUsec uint64
67+
68+
// OOMKills is memory.events' oom_kill: processes in the cgroup the kernel's
69+
// OOM killer has killed over the cgroup's lifetime. Zero if the file is
70+
// absent.
71+
OOMKills uint64
72+
73+
// Empty is true when cgroup.events reports populated 0: no process is left
74+
// in the cgroup. It reads false when the file is absent, so a kernel that
75+
// does not say is never taken for a dead sandbox.
76+
Empty bool
6877
}
6978

7079
// Read returns a Sample for the cgroup v2 directory at dir.
@@ -80,6 +89,7 @@ type Sample struct {
8089
// memory controller's full accounting, and cpu.stat is absent when the cpu
8190
// controller was not delegated. Failing the sample for any of them would mean
8291
// reporting no memory numbers because the node could not report CPU.
92+
// memory.events and cgroup.events are optional in the same way.
8393
func Read(dir string) (Sample, error) {
8494
current, ok, err := readUint(filepath.Join(dir, "memory.current"))
8595
if err != nil {
@@ -95,6 +105,8 @@ func Read(dir string) (Sample, error) {
95105
peak, _, _ := readUint(filepath.Join(dir, "memory.peak"))
96106
inactiveFile, haveInactiveFile, _ := readKeyedUint(filepath.Join(dir, "memory.stat"), "inactive_file")
97107
cpuUsage, _, _ := readKeyedUint(filepath.Join(dir, "cpu.stat"), "usage_usec")
108+
oomKills, _, _ := readKeyedUint(filepath.Join(dir, "memory.events"), "oom_kill")
109+
populated, havePopulated, _ := readKeyedUint(filepath.Join(dir, "cgroup.events"), "populated")
98110

99111
// Without inactive_file there is nothing to subtract, so the working set
100112
// collapses to memory.current. That over-reports by however much reclaimable
@@ -117,6 +129,8 @@ func Read(dir string) (Sample, error) {
117129
MemoryPeakBytes: peak,
118130
MemoryWorkingSetBytes: workingSet,
119131
CPUUsageUsec: cpuUsage,
132+
OOMKills: oomKills,
133+
Empty: havePopulated && populated == 0,
120134
}, nil
121135
}
122136

@@ -150,23 +164,26 @@ func readKeyedUint(path, key string) (uint64, bool, error) {
150164
return 0, false, fmt.Errorf("reading %q: %w", path, err)
151165
}
152166

153-
sc := bufio.NewScanner(bytes.NewReader(b))
154-
for sc.Scan() {
167+
// The files are a few hundred bytes and read for every actor on every
168+
// sweep, so lines are walked in place, and only one that starts with the
169+
// key is split.
170+
for line := range bytes.Lines(b) {
171+
line = bytes.TrimLeft(line, " \t")
172+
if !bytes.HasPrefix(line, []byte(key)) {
173+
continue
174+
}
155175
// Fields, then a length check, rather than indexing straight into the
156176
// split: a blank or single-token line is not worth a panic in an RPC
157177
// handler, and these files are read on a timer for the life of a workload.
158-
f := strings.Fields(sc.Text())
159-
if len(f) != 2 || f[0] != key {
178+
f := bytes.Fields(line)
179+
if len(f) != 2 || string(f[0]) != key {
160180
continue
161181
}
162-
v, err := strconv.ParseUint(f[1], 10, 64)
182+
v, err := strconv.ParseUint(string(f[1]), 10, 64)
163183
if err != nil {
164184
return 0, false, fmt.Errorf("parsing %q of %q: %w", key, path, err)
165185
}
166186
return v, true, nil
167187
}
168-
if err := sc.Err(); err != nil {
169-
return 0, false, fmt.Errorf("scanning %q: %w", path, err)
170-
}
171188
return 0, false, nil
172189
}

‎cmd/ateom-gvisor/internal/cgroupstats/cgroupstats_test.go‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -167,6 +167,44 @@ func TestRead(t *testing.T) {
167167
},
168168
want: Sample{},
169169
},
170+
{
171+
name: "live cgroup is not empty",
172+
files: map[string]string{
173+
"memory.current": "1000\n",
174+
"memory.events": "low 0\nhigh 0\nmax 0\noom 0\noom_kill 0\noom_group_kill 0\n",
175+
"cgroup.events": "populated 1\nfrozen 0\n",
176+
},
177+
want: Sample{
178+
MemoryCurrentBytes: 1000,
179+
MemoryWorkingSetBytes: 1000,
180+
},
181+
},
182+
{
183+
// What a sandbox leaf looked like on kind after the kernel OOM-killed
184+
// the sentry: the leaf and its counters outlive every process in it.
185+
name: "OOM-killed sandbox is empty",
186+
files: map[string]string{
187+
"memory.current": "16384\n",
188+
"memory.events": "low 0\nhigh 0\nmax 0\noom 5334\noom_kill 3\noom_group_kill 0\n",
189+
"cgroup.events": "populated 0\nfrozen 0\n",
190+
},
191+
want: Sample{
192+
MemoryCurrentBytes: 16384,
193+
MemoryWorkingSetBytes: 16384,
194+
OOMKills: 3,
195+
Empty: true,
196+
},
197+
},
198+
{
199+
name: "no cgroup.events is not taken for empty",
200+
files: map[string]string{
201+
"memory.current": "1000\n",
202+
},
203+
want: Sample{
204+
MemoryCurrentBytes: 1000,
205+
MemoryWorkingSetBytes: 1000,
206+
},
207+
},
170208
{
171209
// A short, blank, or over-long line must not panic: this runs in an RPC
172210
// handler on a timer, and grpc-go does not recover handler panics.
@@ -190,6 +228,9 @@ func TestRead(t *testing.T) {
190228
"memory.current": "157286400\n",
191229
"memory.peak": "not-a-number\n",
192230
"cpu.stat": "usage_usec eleventy\n",
231+
"memory.events": "oom_kill lots\n",
232+
// Garbage here must not read as a dead sandbox.
233+
"cgroup.events": "populated no\n",
193234
},
194235
want: Sample{
195236
MemoryCurrentBytes: 157286400,

‎cmd/ateom-gvisor/stats.go‎

Lines changed: 51 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -104,7 +104,9 @@ func (s *AteomService) GetWorkloadStats(ctx context.Context, req *ateompb.GetWor
104104
if hosted == nil {
105105
return nil, apierror.NotFound("ateom is not executing actor %q", req.GetActorUid())
106106
}
107-
sample, err := s.measure(ctx, hosted)
107+
// The raw reading is dropped: the dead-sandbox check runs in the usage
108+
// sweep, which reads every hosted actor.
109+
sample, _, err := s.measure(ctx, hosted)
108110
if err != nil {
109111
// The requested actor is the active one but its cgroup is not there.
110112
// Most often that is a poll landing in the boot: the ateom retains the
@@ -162,10 +164,11 @@ func (s *AteomService) sweepUsage(ctx context.Context) {
162164
// An actor with no numbers is reported as pending, so a failed read for
163165
// one does not lose the others. Past the initial reading, a missing
164166
// cgroup means the sandbox is going away.
165-
sample, err := s.measure(ctx, h)
167+
sample, raw, err := s.measure(ctx, h)
166168
if err != nil {
167169
if !errors.Is(err, fs.ErrNotExist) {
168-
slog.WarnContext(ctx, "Failed to read sandbox cgroup", slog.String("actorUID", h.attribution.UID), slog.Any("err", err))
170+
attrs := append(ateattr.ActorLogAttrs(h.attribution), slog.Any("err", err))
171+
slog.LogAttrs(ctx, slog.LevelWarn, "Failed to read sandbox cgroup", attrs...)
169172
}
170173
sample = h.usage.WithEpoch(pendingSample(&h.attribution))
171174
}
@@ -176,14 +179,17 @@ func (s *AteomService) sweepUsage(ctx context.Context) {
176179
if s.lookupActor(h.attribution.UID) != h {
177180
continue
178181
}
182+
if raw.Empty { // a failed read returns a zero Sample
183+
s.warnDeadSandbox(ctx, h, raw)
184+
}
179185
h.usage.Periodic(sample, func() { s.usage.Emit(ctx, ateattr.StatsKindPeriodic, sample) })
180186
}
181187
}
182188

183189
// recordInitial samples a new activation once its sandbox is up and writes its
184190
// initial record.
185191
func (s *AteomService) recordInitial(ctx context.Context, h *hostedActor) {
186-
sample, err := s.measure(ctx, h)
192+
sample, _, err := s.measure(ctx, h)
187193
if err != nil {
188194
slog.WarnContext(ctx, "No initial usage sample", slog.String(string(ateattr.ActorUIDKey), h.attribution.UID), slog.Any("err", err))
189195
sample = nil
@@ -195,7 +201,7 @@ func (s *AteomService) recordInitial(ctx context.Context, h *hostedActor) {
195201
// record a successful checkpoint or terminate writes. A failed read leaves the
196202
// newest measured sample to stand in.
197203
func (s *AteomService) readFinal(ctx context.Context, h *hostedActor) {
198-
if sample, err := s.measure(ctx, h); err == nil {
204+
if sample, _, err := s.measure(ctx, h); err == nil {
199205
h.usage.Store(sample)
200206
}
201207
}
@@ -239,29 +245,34 @@ func pendingSample(active *resources.ActorAttribution) *ateompb.WorkloadStatsSam
239245
}
240246
}
241247

242-
// measure reads h's sandbox cgroup as a reading of its activation.
243-
func (s *AteomService) measure(ctx context.Context, h *hostedActor) (*ateompb.WorkloadStatsSample, error) {
244-
return h.usage.Measure(ctx, func() (*ateompb.WorkloadStatsSample, map[string]uint64, error) {
245-
sample, err := s.sampleSandbox(&h.attribution)
248+
// measure reads h's sandbox cgroup as a reading of its activation, returning
249+
// the raw cgroup reading alongside it for the dead-sandbox check.
250+
func (s *AteomService) measure(ctx context.Context, h *hostedActor) (*ateompb.WorkloadStatsSample, cgroupstats.Sample, error) {
251+
var raw cgroupstats.Sample
252+
sample, err := h.usage.Measure(ctx, func() (*ateompb.WorkloadStatsSample, map[string]uint64, error) {
253+
sample, r, err := s.sampleSandbox(&h.attribution)
254+
raw = r
246255
return sample, nil, err
247256
})
257+
return sample, raw, err
248258
}
249259

250260
// sampleSandbox reads the sandbox cgroup and builds the sample attributed to
251-
// active. Errors come back raw -- notably fs.ErrNotExist for a cgroup that is
252-
// not there yet -- because the callers disagree on what that means: an error
253-
// code for the keyed read, a pending sample for the sweep.
261+
// active, returning the raw reading alongside it. Errors come back raw --
262+
// notably fs.ErrNotExist for a cgroup that is not there yet -- because the
263+
// callers disagree on what that means: an error code for the keyed read, a
264+
// pending sample for the sweep.
254265
// The read holds no lifecycle lock, so the keyed caller re-checks the actor
255266
// record it loaded after this returns.
256-
func (s *AteomService) sampleSandbox(active *resources.ActorAttribution) (*ateompb.WorkloadStatsSample, error) {
267+
func (s *AteomService) sampleSandbox(active *resources.ActorAttribution) (*ateompb.WorkloadStatsSample, cgroupstats.Sample, error) {
257268
read := s.readSandboxCgroup
258269
if read == nil {
259270
read = cgroupstats.Read
260271
}
261272
observedAt := time.Now()
262273
sample, err := read(filepath.Join(s.cgroupRoot, ocispec.GVisorCgroupLeaf(active.UID, sandboxCgroupContainer)))
263274
if err != nil {
264-
return nil, err
275+
return nil, cgroupstats.Sample{}, err
265276
}
266277

267278
return &ateompb.WorkloadStatsSample{
@@ -280,5 +291,30 @@ func (s *AteomService) sampleSandbox(active *resources.ActorAttribution) (*ateom
280291
CpuUsageUsec: sample.CPUUsageUsec,
281292

282293
ObservedAtUnixNano: observedAt.UnixNano(),
283-
}, nil
294+
}, sample, nil
295+
}
296+
297+
// warnDeadSandbox logs, once per activation, that h's sandbox cgroup has no
298+
// process left. Nothing else notices this today: the actor stays RUNNING and
299+
// its stats read like an idle actor's (#2211).
300+
//
301+
// A lifecycle RPC empties the cgroup on purpose when it tears the sandbox down,
302+
// and holds the actor's lock while it does, so a busy actor is skipped. The
303+
// cgroup was read before the lock was checked, and h is looked up again after,
304+
// so a teardown that starts or finishes in between is never reported.
305+
//
306+
// oom_kill counts over the cgroup's lifetime. A leaf left behind by an earlier
307+
// activation of the same actor carries its count forward.
308+
func (s *AteomService) warnDeadSandbox(ctx context.Context, h *hostedActor, raw cgroupstats.Sample) {
309+
if h.deadReported.Load() {
310+
return
311+
}
312+
if s.locks.Busy(h.attribution.UID) || s.lookupActor(h.attribution.UID) != h {
313+
return
314+
}
315+
if !h.deadReported.CompareAndSwap(false, true) {
316+
return
317+
}
318+
attrs := append(ateattr.ActorLogAttrs(h.attribution), slog.Uint64(string(ateattr.SandboxOOMKillsKey), raw.OOMKills))
319+
slog.LogAttrs(ctx, slog.LevelWarn, "Sandbox has no processes left while the actor is hosted", attrs...)
284320
}

0 commit comments

Comments
 (0)