Skip to content

Commit 80e0659

Browse files
committed
imagecache: Refuse a layout version 1 pool and log diffID mismatches
Layers in a version 1 pool were never hashed against their diffID, and eviction never retires a layer that images keep using. Bump the layout version so atelet refuses the old pool, and document how to rebuild a node's image cache. Log a diffID mismatch at WARN with the image. The RPC error was its only trace.
1 parent 9a3c261 commit 80e0659

4 files changed

Lines changed: 42 additions & 6 deletions

File tree

‎docs/upgrade.md‎

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -103,7 +103,7 @@ kubectl -n ate-system rollout status ds/atelet
103103

104104
- **Progress:** `Waiting for daemon set "atelet" rollout to finish: 2 out of 5 new pods have been updated...`, then `... 4 of 5 updated pods are available...`.
105105
- **Done:** `daemon set "atelet" successfully rolled out`.
106-
- **Stuck:** the `N out of M` count does not move for more than about 6 minutes. A new atelet pod that cannot start stops the roll and leaves its node without atelet; if you cannot fix it quickly, [roll back step 2](#roll-back-step-2). A pod stuck in ContainerCreating usually means podcertificate-controller is not running. If `deploy atelet` times out, the roll still goes on; keep watching.
106+
- **Stuck:** the `N out of M` count does not move for more than about 6 minutes. A new atelet pod that cannot start stops the roll and leaves its node without atelet; if you cannot fix it quickly, [roll back step 2](#roll-back-step-2). If its log says the image cache has a layout version this atelet does not support, [rebuild the node's image cache](#rebuild-a-nodes-image-cache). A pod stuck in ContainerCreating usually means podcertificate-controller is not running. If `deploy atelet` times out, the roll still goes on; keep watching.
107107
- **Actors:** running actors keep running. A node's atelet is down for a few seconds while it restarts.
108108

109109
### Step 3. Move each WorkerPool to the new worker image
@@ -256,6 +256,18 @@ kubectl -n $NS rollout resume deployment/$WORKERPOOL
256256

257257
Workers already removed still get their 30 minutes. Do not leave a pool paused, and do not start step 4 while one is: a paused pool ignores every later change to its workers.
258258

259+
### Rebuild a node's image cache
260+
261+
atelet refuses to start on an image cache whose layout version it does not support, and its log names both versions. The cache holds no unique state, but the actors on the node run from it, so delete it only once no actor runs on the node. atelet rebuilds it as actors start:
262+
263+
```bash
264+
NODE=my-node # change me
265+
kubectl debug node/$NODE -it --image=busybox -- rm -rf /host/var/lib/ate/image-cache
266+
kubectl delete pod -n ate-system -l app=atelet --field-selector spec.nodeName=$NODE
267+
# kubectl debug leaves its finished pod behind.
268+
kubectl get pods -o name | grep "^pod/node-debugger-$NODE-" | xargs kubectl delete
269+
```
270+
259271
### Roll back
260272

261273
To go back to the old release, roll back every step you began, finished or not, starting with the last. To fix one pool, roll back only that pool. Rollback runs from the old checkout, with `old.yaml`:
@@ -308,6 +320,8 @@ Only workers already on the new image are replaced, and their actors get `SIGTER
308320
go run ./cmd/ate-setup deploy atelet --config ~/ate-upgrade/old.yaml --rollout-timeout 1h
309321
```
310322

323+
If the new release changed the image cache layout, the old atelet refuses the cache the new one wrote and does not start; [rebuild the node's image cache](#rebuild-a-nodes-image-cache) on each node where that happens.
324+
311325
#### Roll back step 1
312326

313327
```bash

‎internal/imagecache/README.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -58,7 +58,7 @@ anywhere.
5858

5959
```
6060
<cache-root>/ default: /var/lib/ate/image-cache
61-
version layout version marker ("1")
61+
version layout version marker ("2")
6262
layers/sha256/<diffid-hex>/
6363
fs/ the unpacked layer tree (an overlay lowerdir)
6464
whiteouts.json whiteout state recorded at unpack time
@@ -115,8 +115,8 @@ layer diffIDs in order — layers shared by N images exist once.
115115

116116
A layer already in the pool is reused without re-hashing, because it
117117
was checked when it was written. The reusing image's own blob is never
118-
downloaded, so it is never checked. Layers unpacked by an atelet that
119-
lacked this check stay in the pool until eviction retires them.
118+
downloaded, so it is never checked. atelet refuses a layout version 1
119+
pool, whose layers were never checked.
120120
6. **Record**: the image config + diffID list is written under the
121121
requested digest (and the per-platform child digest for multi-arch refs).
122122

‎internal/imagecache/imagecache.go‎

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -75,7 +75,9 @@ import (
7575
)
7676

7777
const (
78-
layoutVersion = "1"
78+
// Version 2 layers were hashed against their diffID at unpack. Version 1
79+
// layers never were, so a version 1 pool is refused.
80+
layoutVersion = "2"
7981
versionFileName = "version"
8082

8183
layerFSDirName = "fs"
@@ -338,7 +340,7 @@ func New(root string, opts ...Option) (*Store, error) {
338340
if got := strings.TrimSpace(string(b)); got != layoutVersion {
339341
// Fail loudly instead of silently mixing layouts; an operator can
340342
// delete the cache dir to rebuild it (it holds no unique state).
341-
return nil, fmt.Errorf("image cache at %q has layout version %q, this atelet supports %q", root, got, layoutVersion)
343+
return nil, fmt.Errorf("image cache at %q has layout version %q, this atelet supports %q: delete the directory once no actor on this node is running, and atelet rebuilds it", root, got, layoutVersion)
342344
}
343345
case errors.Is(err, os.ErrNotExist):
344346
if err := os.WriteFile(versionPath, []byte(layoutVersion+"\n"), 0o600); err != nil {
@@ -634,6 +636,11 @@ func (s *Store) pullWith(ctx context.Context, parsedRef name.Reference, digest v
634636
diffID := cfgFile.RootFS.DiffIDs[i]
635637
dir, err := s.ensureLayer(gctx, diffID, layer)
636638
if err != nil {
639+
if errors.Is(err, errDiffIDMismatch) {
640+
slog.WarnContext(gctx, "Image layer does not match its claimed diffID",
641+
slog.String("ref", parsedRef.String()), slog.String("digest", digest.String()),
642+
slog.String("diffid", diffID.String()), slog.Any("err", err))
643+
}
637644
return fmt.Errorf("while unpacking layer %s: %w", diffID, err)
638645
}
639646
layerDirs[i] = dir

‎internal/imagecache/verify_test.go‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@ import (
2020
"context"
2121
"errors"
2222
"io"
23+
"log/slog"
2324
"net/http"
2425
"os"
2526
"path/filepath"
@@ -134,6 +135,16 @@ func assertPoolEmpty(t *testing.T, s *Store) {
134135
}
135136
}
136137

138+
// captureLogs sends the default logger to a buffer until the test ends.
139+
func captureLogs(t *testing.T) *bytes.Buffer {
140+
t.Helper()
141+
var buf bytes.Buffer
142+
prev := slog.Default()
143+
slog.SetDefault(slog.New(slog.NewTextHandler(&buf, nil)))
144+
t.Cleanup(func() { slog.SetDefault(prev) })
145+
return &buf
146+
}
147+
137148
// A blob whose bytes differ from the manifest digest is rejected, even
138149
// though it decompresses to the claimed diffID.
139150
func TestPullRejectsBlobNotMatchingManifestDigest(t *testing.T) {
@@ -165,10 +176,14 @@ func TestPullRejectsContentNotMatchingClaimedDiffID(t *testing.T) {
165176
pushLyingImage(t, evilRef, evil, good)
166177
pushImage(t, goodRef, v1.Config{}, good)
167178

179+
logs := captureLogs(t)
168180
s := newTestStore(t)
169181
if _, err := s.EnsureImage(context.Background(), evilRef); !errors.Is(err, errDiffIDMismatch) {
170182
t.Fatalf("lying pull error = %v, want errDiffIDMismatch", err)
171183
}
184+
if !strings.Contains(logs.String(), "level=WARN msg=\"Image layer does not match its claimed diffID\"") {
185+
t.Errorf("no mismatch WARN in logs:\n%s", logs)
186+
}
172187
assertPoolEmpty(t, s)
173188

174189
img, err := s.EnsureImage(context.Background(), goodRef)

0 commit comments

Comments
 (0)