Repository navigation
Commit f6a1dc7
authored
ateom-gvisor logs a warning when a running actor's sandbox dies (#2258)
Part of #2212 (o11y side of #2211).
If the gVisor sandbox dies after start (sentry crash, kill, OOM kill),
nothing notices today. The actor stays RUNNING and its usage samples
keep reading like an idle actor's, because the pause cgroup leaf and its
counters outlive the processes in it.
This makes the death visible:
- `cgroupstats` also reads `populated` from `cgroup.events` and
`oom_kill` from `memory.events`. If either file is missing, it never
reads as a dead sandbox.
- The ateom's usage sweep (`sweepUsage`, every
`--usage-sample-interval`, 1m by default) logs one WARN per activation
when the leaf is empty. Since #2194, `GetActiveWorkloadStats` serves the
sweep's cached sample and reads no cgroup, so the check lives in the
sweep, the only periodic cgroup read; `measure` hands back the raw
`cgroupstats.Sample` for it. The record uses `ateattr.ActorLogAttrs`
plus `ate.sandbox.oom_kills`, so filtering on `ate.actor.uid` finds it
with the rest of the actor's logs.
- Actors with a lifecycle RPC holding or waiting on their lock are
skipped, since checkpoint and terminate empty the cgroup on purpose.
`actorlock` gets a non-blocking `Busy` for this. An activation whose
final record is written (checkpoint, terminate, drain) is no longer
swept at all.
- `readKeyedUint` walks lines in place instead of using a
`bufio.Scanner`, which allocated 4 KiB per file. With the two extra
reads, reading 1000 actors' cgroups takes about 9% longer than before
and allocates half the bytes.
It does not change actor state, RPC responses or the stats samples.
Moving the actor to CRASHED is #2211. It also does not catch an app that
exits while the sentry stays up.
Tested on kind and on GKE (substrate-relay) with the branch image on the
egress pool, before the rebase onto #2194 (the check then ran in
`GetActiveWorkloadStats`). Unit tests pass at the rebased head; live
re-verification on kind is pending:
| Case | kind | GKE |
|---|---|---|
| OOM kill via `memory.max` | 1 WARN, `oom_kills=3` | 1 WARN,
`oom_kills=1` |
| `kill -9` the sentry | 1 WARN, `oom_kills=0` | 1 WARN, `oom_kills=0` |
| Normal suspend | none | none |
| Suspend then resume | none | none |
| Idle actors | none (5) | none (3) |
Each warning showed up on the first sweep after the fault and was not
repeated. On GKE it lands in Cloud Logging as severity WARNING with the
`ate.*` fields. The stats pipeline was unchanged.
One caveat: `oom_kill` counts over the cgroup's lifetime, so a leaf left
behind by an earlier activation of the same actor (#2107) carries its
count forward.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR1 parent 632e662 commit f6a1dc7
10 files changed
Lines changed: 427 additions & 28 deletions
File tree
- cmd/ateom-gvisor
- internal/cgroupstats
- docs
- metrics/registry
- internal
- actorlock
- ateattr
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
21 | 21 | | |
22 | 22 | | |
23 | 23 | | |
| 24 | + | |
24 | 25 | | |
25 | 26 | | |
26 | 27 | | |
| |||
42 | 43 | | |
43 | 44 | | |
44 | 45 | | |
| 46 | + | |
| 47 | + | |
45 | 48 | | |
46 | 49 | | |
47 | 50 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
21 | 21 | | |
22 | 22 | | |
23 | 23 | | |
24 | | - | |
| 24 | + | |
25 | 25 | | |
26 | 26 | | |
27 | 27 | | |
28 | | - | |
29 | 28 | | |
30 | 29 | | |
31 | 30 | | |
| |||
65 | 64 | | |
66 | 65 | | |
67 | 66 | | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
68 | 77 | | |
69 | 78 | | |
70 | 79 | | |
| |||
80 | 89 | | |
81 | 90 | | |
82 | 91 | | |
| 92 | + | |
83 | 93 | | |
84 | 94 | | |
85 | 95 | | |
| |||
95 | 105 | | |
96 | 106 | | |
97 | 107 | | |
| 108 | + | |
| 109 | + | |
98 | 110 | | |
99 | 111 | | |
100 | 112 | | |
| |||
117 | 129 | | |
118 | 130 | | |
119 | 131 | | |
| 132 | + | |
| 133 | + | |
120 | 134 | | |
121 | 135 | | |
122 | 136 | | |
| |||
150 | 164 | | |
151 | 165 | | |
152 | 166 | | |
153 | | - | |
154 | | - | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
155 | 175 | | |
156 | 176 | | |
157 | 177 | | |
158 | | - | |
159 | | - | |
| 178 | + | |
| 179 | + | |
160 | 180 | | |
161 | 181 | | |
162 | | - | |
| 182 | + | |
163 | 183 | | |
164 | 184 | | |
165 | 185 | | |
166 | 186 | | |
167 | 187 | | |
168 | | - | |
169 | | - | |
170 | | - | |
171 | 188 | | |
172 | 189 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
167 | 167 | | |
168 | 168 | | |
169 | 169 | | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
170 | 208 | | |
171 | 209 | | |
172 | 210 | | |
| |||
190 | 228 | | |
191 | 229 | | |
192 | 230 | | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
193 | 234 | | |
194 | 235 | | |
195 | 236 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
104 | 104 | | |
105 | 105 | | |
106 | 106 | | |
107 | | - | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
108 | 110 | | |
109 | 111 | | |
110 | 112 | | |
| |||
162 | 164 | | |
163 | 165 | | |
164 | 166 | | |
165 | | - | |
| 167 | + | |
166 | 168 | | |
167 | 169 | | |
168 | | - | |
| 170 | + | |
| 171 | + | |
169 | 172 | | |
170 | 173 | | |
171 | 174 | | |
| |||
176 | 179 | | |
177 | 180 | | |
178 | 181 | | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
179 | 185 | | |
180 | 186 | | |
181 | 187 | | |
182 | 188 | | |
183 | 189 | | |
184 | 190 | | |
185 | 191 | | |
186 | | - | |
| 192 | + | |
187 | 193 | | |
188 | 194 | | |
189 | 195 | | |
| |||
195 | 201 | | |
196 | 202 | | |
197 | 203 | | |
198 | | - | |
| 204 | + | |
199 | 205 | | |
200 | 206 | | |
201 | 207 | | |
| |||
239 | 245 | | |
240 | 246 | | |
241 | 247 | | |
242 | | - | |
243 | | - | |
244 | | - | |
245 | | - | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
246 | 255 | | |
247 | 256 | | |
| 257 | + | |
248 | 258 | | |
249 | 259 | | |
250 | 260 | | |
251 | | - | |
252 | | - | |
253 | | - | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
254 | 265 | | |
255 | 266 | | |
256 | | - | |
| 267 | + | |
257 | 268 | | |
258 | 269 | | |
259 | 270 | | |
260 | 271 | | |
261 | 272 | | |
262 | 273 | | |
263 | 274 | | |
264 | | - | |
| 275 | + | |
265 | 276 | | |
266 | 277 | | |
267 | 278 | | |
| |||
280 | 291 | | |
281 | 292 | | |
282 | 293 | | |
283 | | - | |
| 294 | + | |
| 295 | + | |
| 296 | + | |
| 297 | + | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
| 307 | + | |
| 308 | + | |
| 309 | + | |
| 310 | + | |
| 311 | + | |
| 312 | + | |
| 313 | + | |
| 314 | + | |
| 315 | + | |
| 316 | + | |
| 317 | + | |
| 318 | + | |
| 319 | + | |
284 | 320 | | |
0 commit comments