Skip to content

Commit 2d96fd6

Browse files
authored
CI: Add the upgrade E2E test for each release (agent-substrate#2712)
This adds an e2e suite and a workflow that upgrade a running install with actors on it, following `docs/upgrade.md` step by step. The workflow installs the newest earlier release on the branch with that release's own installer, and the suite starts counter actors that are running, paused, or suspended. One job runs them on gVisor and one on microVM. It then runs each runbook step with this checkout's `ate-setup` and a copy of the install record: podcertificate-controller, atelet, the WorkerPool's `workerImage`, the API server, ate-controller, atenet, the SandboxConfig, then `deploy ate-system --setup-csi=none`. While it runs, it acts as the actors' harness and suspends an actor as soon as its worker pod starts terminating. After each step, the step's component has to run new images and no actor may be `CRASHED`. Only the pool roll may roll the workers and evict actors, plus ate-controller if its pod template changed. At the end, every actor, plus one created after the upgrade, has to resume on a new worker with its counter intact. ----- ### Cadence Patch releases are cut by merging fixes into `release-X.Y` and tagging there. The workflow runs on every push to a `release-X.Y` branch and upgrades from the newest earlier tag on that branch, so the commit that becomes v0.6.1 is tested as `v0.6.0` → `v0.6.1` before it is tagged. A push with no earlier tag yet, such as the branch cut itself, is skipped with a notice. It also runs on demand. It doesn't run on pull requests or on pushes to main. After GA 1.0 we support upgrading to the next minor, for example 1.1.x → 1.2.y, so a push to `release-1.2` should also upgrade from the newest 1.1 release. That second lane comes with 1.0. -------- #### Tested - main → this PR, on GitHub runners in my fork: both lanes pass all 8 steps, in about 15 and 18 minutes. The microVM lane needs KVM, so I only runs it on GitHub runners. Not covered yet: rollback. I used Claude Code to help write the suite and to run the tests above. Part of agent-substrate#1784 - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR --------- Signed-off-by: Haven Xia <[email protected]>
1 parent 76cba32 commit 2d96fd6

9 files changed

Lines changed: 1271 additions & 2 deletions

File tree

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
#!/usr/bin/env bash
2+
3+
# Copyright 2026 Google LLC
4+
#
5+
# Licensed under the Apache License, Version 2.0 (the "License");
6+
# you may not use this file except in compliance with the License.
7+
# You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
17+
# Prints the state of the kind cluster after a failed upgrade: the worker
18+
# pools and pods, the system workloads, and the logs of the system pods and of
19+
# every worker pod. The actors' worker logs say what happened to each actor.
20+
#
21+
# Used by .github/workflows/upgrade-e2e.yaml. Every command is best effort.
22+
23+
set -o nounset -o pipefail
24+
25+
kubectl --context kind-kind get workerpool,pods -A -o wide || true
26+
kubectl --context kind-kind get ds,deploy -n ate-system -o wide || true
27+
dump() {
28+
echo "=== logs: $1/$2 ==="
29+
kubectl --context kind-kind logs -n "$1" "$2" --all-containers --tail=300 2>/dev/null || true
30+
}
31+
for ns in ate-system podcertificate-controller-system; do
32+
for p in $(kubectl --context kind-kind get pods -n "${ns}" -o name 2>/dev/null); do
33+
dump "${ns}" "${p}"
34+
done
35+
done
36+
kubectl --context kind-kind get pods -A -l ate.dev/worker-pool \
37+
-o 'custom-columns=:.metadata.namespace,:.metadata.name' --no-headers 2>/dev/null \
38+
| while read -r ns name; do dump "${ns}" "${name}"; done
Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
#!/usr/bin/env bash
2+
3+
# Copyright 2026 Google LLC
4+
#
5+
# Licensed under the Apache License, Version 2.0 (the "License");
6+
# you may not use this file except in compliance with the License.
7+
# You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
17+
# Grants the runner access to /dev/kvm, so hack/create-kind-cluster.sh mounts
18+
# it into the kind node for microVM workers.
19+
#
20+
# Used by .github/workflows/upgrade-e2e.yaml.
21+
22+
set -o errexit -o nounset -o pipefail
23+
24+
echo 'KERNEL=="kvm", GROUP="kvm", MODE="0666", OPTIONS+="static_node=kvm"' \
25+
| sudo tee /etc/udev/rules.d/99-kvm4all.rules
26+
sudo udevadm control --reload-rules
27+
sudo udevadm trigger --name-match=kvm
Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
#!/usr/bin/env bash
2+
3+
# Copyright 2026 Google LLC
4+
#
5+
# Licensed under the Apache License, Version 2.0 (the "License");
6+
# you may not use this file except in compliance with the License.
7+
# You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
17+
# Installs the release to upgrade from with its own installer, so the cluster
18+
# starts out as that release left it, plus the counter demo of SANDBOX
19+
# (gvisor or microvm) that the suite's actors run on.
20+
#
21+
# (cd <old checkout> && <new checkout>/.github/scripts/upgrade-e2e/install-old.sh SANDBOX)
22+
#
23+
# Runs in the checkout of the old release. A release whose deploy ate-system
24+
# does not install the microVM dependencies has
25+
# hack/run-microvm-demo-kind.sh, which installs them, the control plane and
26+
# counter-microvm.
27+
#
28+
# Used by .github/workflows/upgrade-e2e.yaml.
29+
30+
set -o errexit -o nounset -o pipefail
31+
32+
sandbox="${1:?usage: install-old.sh gvisor|microvm}"
33+
34+
case "${sandbox}" in
35+
gvisor)
36+
demo=counter
37+
;;
38+
microvm)
39+
if [[ -x hack/run-microvm-demo-kind.sh ]]; then
40+
exec hack/run-microvm-demo-kind.sh
41+
fi
42+
demo=counter-microvm
43+
;;
44+
*)
45+
echo "unknown sandbox ${sandbox}, want gvisor or microvm" >&2
46+
exit 1
47+
;;
48+
esac
49+
50+
go run ./cmd/ate-setup --kind deploy ate-system
51+
go run ./cmd/ate-setup --kind deploy demo "${demo}"
Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
#!/usr/bin/env bash
2+
3+
# Copyright 2026 Google LLC
4+
#
5+
# Licensed under the Apache License, Version 2.0 (the "License");
6+
# you may not use this file except in compliance with the License.
7+
# You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
17+
# Picks the release to upgrade from and writes it to GITHUB_OUTPUT as tag=.
18+
#
19+
# FROM=<tag or commit, may be empty> TO=<branch> EVENT=<event name> \
20+
# .github/scripts/upgrade-e2e/pick-from.sh
21+
#
22+
# FROM wins when set. Otherwise, on a release-X.Y branch, it is the newest
23+
# earlier vX.Y.Z release: a release branch only takes patch releases, and
24+
# users upgrade from a release. A minor with no release yet, such as one
25+
# between its release candidates, starts from its newest pre-release. With
26+
# nothing to upgrade from, a push writes an empty tag and the job skips its
27+
# steps, and a manual run fails.
28+
#
29+
# Runs in the checkout of the release under test. Used by
30+
# .github/workflows/upgrade-e2e.yaml.
31+
32+
set -o errexit -o nounset -o pipefail
33+
34+
if [[ -z "${FROM}" && "${TO}" =~ ^release-[0-9]+\.[0-9]+$ ]]; then
35+
# The minor's tags, newest first. versionsort.suffix sorts a pre-release
36+
# (v1.0.0-rc.1) below its release.
37+
tags="$(git -c versionsort.suffix=- tag -l "v${TO#release-}.*" --merged HEAD --no-contains HEAD --sort=-v:refname)"
38+
FROM="$(grep -v -e - <<<"${tags}" | head -n 1 || true)"
39+
if [[ -z "${FROM}" ]]; then
40+
FROM="$(head -n 1 <<<"${tags}")"
41+
fi
42+
fi
43+
if [[ -z "${FROM}" ]]; then
44+
if [[ "${EVENT}" == workflow_dispatch ]]; then
45+
echo "::error::Nothing to upgrade ${TO} from: set the from input."
46+
exit 1
47+
fi
48+
echo "::notice::No earlier release on ${TO} to upgrade from yet."
49+
echo "tag=" >> "${GITHUB_OUTPUT}"
50+
exit 0
51+
fi
52+
echo "Upgrading ${FROM} -> ${TO}"
53+
echo "tag=${FROM}" >> "${GITHUB_OUTPUT}"
Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
#!/usr/bin/env bash
2+
3+
# Copyright 2026 Google LLC
4+
#
5+
# Licensed under the Apache License, Version 2.0 (the "License");
6+
# you may not use this file except in compliance with the License.
7+
# You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
17+
# Runs the upgrade suite (internal/e2e/upgrade) from a copy of the old
18+
# release's install record, so this checkout's ate-setup upgrades the cluster
19+
# in the runbook's order.
20+
#
21+
# ATE_RECORD_DIR=<dir> E2E_ATE_SETUP_CONFIG=<file> E2E_JUNIT_FILE=<file> \
22+
# .github/scripts/upgrade-e2e/run-suite.sh
23+
#
24+
# Runs in the checkout of the release under test. The old release is installed
25+
# with no preview gates, and CI has to say so. Used by
26+
# .github/workflows/upgrade-e2e.yaml.
27+
28+
set -o errexit -o nounset -o pipefail
29+
30+
cp "${ATE_RECORD_DIR}"/installs/*.yaml "${E2E_ATE_SETUP_CONFIG}"
31+
hack/run-e2e-kind.sh ./internal/e2e/upgrade -timeout 90m -args --no-color --preview=""

‎.github/workflows/upgrade-e2e.yaml‎

Lines changed: 120 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,120 @@
1+
# Copyright 2026 Google LLC
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
# Upgrades a cluster running a release to this checkout, with actors on it,
16+
# following docs/guides/upgrade.md, and checks that every actor survives (see
17+
# internal/e2e/upgrade). Two installs on one kind cluster take too long for
18+
# every PR, so this runs for every push to a release branch, before a patch
19+
# release is tagged from it, plus on demand for any branch.
20+
name: upgrade-e2e
21+
on:
22+
push:
23+
branches: ['release-[0-9]+.[0-9]+']
24+
workflow_dispatch:
25+
inputs:
26+
from:
27+
description: 'Tag or commit to upgrade from. Required except on a release branch, which defaults to its newest earlier release'
28+
required: false
29+
default: ''
30+
permissions:
31+
contents: read
32+
jobs:
33+
upgrade:
34+
name: Upgrade e2e (${{ matrix.sandbox }})
35+
runs-on: ubuntu-latest
36+
timeout-minutes: 120
37+
strategy:
38+
fail-fast: false
39+
matrix:
40+
sandbox: [gvisor, microvm]
41+
env:
42+
# Points the suite at the counter-microvm fixture. Empty means gVisor.
43+
E2E_SANDBOX_CLASS: ${{ matrix.sandbox == 'microvm' && 'microvm' || '' }}
44+
# Where the previous release's ate-setup writes its install record.
45+
ATE_RECORD_DIR: ${{ github.workspace }}/ate-records
46+
# deploy ate-system requires a credential provider selection. The
47+
# record keeps it for the upgrade.
48+
ATE_CREDENTIAL_PROVIDER: '{"name":"k8s.io"}'
49+
# deploy ate-system also requires an ingress allowlist. This is the
50+
# suite's router client (internal/e2e/ingressauth.go).
51+
ATE_INGRESS_ALLOWED_SPIFFE_IDS: spiffe://cluster.local/ns/ate-e2e-router-client/sa/router-client
52+
ARTIFACTS: ${{ github.workspace }}/_artifacts
53+
steps:
54+
- name: Checkout the release under test
55+
uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09 # v5.1.0
56+
with:
57+
path: new
58+
# Full history brings the release tags the next step picks from.
59+
fetch-depth: 0
60+
- name: Pick the release to upgrade from
61+
id: from
62+
working-directory: new
63+
env:
64+
FROM: ${{ inputs.from }}
65+
TO: ${{ github.ref_name }}
66+
EVENT: ${{ github.event_name }}
67+
run: .github/scripts/upgrade-e2e/pick-from.sh
68+
- name: Checkout ${{ steps.from.outputs.tag }}
69+
if: steps.from.outputs.tag != ''
70+
uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09 # v5.1.0
71+
with:
72+
ref: ${{ steps.from.outputs.tag }}
73+
path: old
74+
- name: Setup Go
75+
if: steps.from.outputs.tag != ''
76+
uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
77+
with:
78+
go-version-file: new/go.mod
79+
cache-dependency-path: |
80+
new/go.sum
81+
old/go.sum
82+
- name: Enable KVM
83+
if: steps.from.outputs.tag != '' && matrix.sandbox == 'microvm'
84+
run: new/.github/scripts/upgrade-e2e/enable-kvm.sh
85+
- name: Create cluster
86+
if: steps.from.outputs.tag != ''
87+
working-directory: new
88+
run: hack/create-kind-cluster.sh
89+
- name: Install ${{ steps.from.outputs.tag }}
90+
if: steps.from.outputs.tag != ''
91+
working-directory: old
92+
run: ../new/.github/scripts/upgrade-e2e/install-old.sh ${{ matrix.sandbox }}
93+
- name: Upgrade to ${{ github.ref_name }}
94+
if: steps.from.outputs.tag != ''
95+
# The suite performs the upgrade with this checkout's ate-setup, in the
96+
# runbook's order and from a copy of the install record, and checks the
97+
# actors after every step.
98+
working-directory: new
99+
env:
100+
E2E_ATE_SETUP_CONFIG: ${{ github.workspace }}/new.yaml
101+
E2E_JUNIT_FILE: ${{ github.workspace }}/_artifacts/e2e-upgrade-${{ matrix.sandbox }}.xml
102+
run: .github/scripts/upgrade-e2e/run-suite.sh
103+
# A suite that stops matching would otherwise leave the run green and empty.
104+
- name: Require the suite to have run tests
105+
if: always() && steps.from.outputs.tag != ''
106+
working-directory: new
107+
run: go -C tools/junittool run . verify -manifest "${ARTIFACTS}/expected-junit.txt"
108+
- name: Upload test results
109+
if: always() && steps.from.outputs.tag != ''
110+
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
111+
with:
112+
name: test-results-upgrade-e2e-${{ matrix.sandbox }}
113+
# The install records carry no secrets.
114+
path: |
115+
${{ env.ARTIFACTS }}
116+
${{ env.ATE_RECORD_DIR }}
117+
if-no-files-found: error
118+
- name: Dump diagnostics on failure
119+
if: failure() && steps.from.outputs.tag != ''
120+
run: new/.github/scripts/upgrade-e2e/dump-diagnostics.sh

‎internal/e2e/README.md‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,9 @@ $ go test -v ./internal/e2e/suites/... -args --e2e
88
## Principles
99

1010
* Keep it simple -- use go test for the harness.
11-
* e2e tests live under `internal/e2e/suites/<suite>`
11+
* e2e tests live under `internal/e2e/suites/<suite>`. The upgrade suite lives in
12+
`internal/e2e/upgrade` instead, so the default run leaves it out: it upgrades
13+
the cluster it runs on. `.github/workflows/upgrade-e2e.yaml` runs it.
1214
* Each suite should implement TestMain using e2e.RunTestMain()
1315
* e2e tests will be skipped for ordinary unit tests unless the `--e2e` flag
1416
is set e.g. `go test ./internal/e2e/suites/... -args --e2e`
@@ -55,7 +57,10 @@ installs the fixtures elsewhere: `E2E_SUBSTRATE_TEMPLATE_ATESPACE` /
5557
`E2E_SUBSTRATE_POOL_NAME` point the counter fixture somewhere else, and
5658
`E2E_TEMPLATE_READY_TIMEOUT` replaces the golden-snapshot budget (90s on
5759
gVisor, 10m on microVM, where the golden is a cloud-hypervisor cold boot plus
58-
a checkpoint).
60+
a checkpoint). The upgrade suite runs `ate-setup` with `--config
61+
$E2E_ATE_SETUP_CONFIG` when it is set, and otherwise with `--kind`, which
62+
`E2E_ATE_SETUP_FLAGS` replaces. With a record of pre-built images, it rolls the
63+
pool to the worker image for its sandbox class at the record's tag.
5964

6065
## Settings
6166

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
// Copyright 2026 Google LLC
2+
//
3+
// Licensed under the Apache License, Version 2.0 (the "License");
4+
// you may not use this file except in compliance with the License.
5+
// You may obtain a copy of the License at
6+
//
7+
// http://www.apache.org/licenses/LICENSE-2.0
8+
//
9+
// Unless required by applicable law or agreed to in writing, software
10+
// distributed under the License is distributed on an "AS IS" BASIS,
11+
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
// See the License for the specific language governing permissions and
13+
// limitations under the License.
14+
15+
package upgrade
16+
17+
import (
18+
"os"
19+
"testing"
20+
21+
"go.ate.dev/substrate/internal/e2e"
22+
)
23+
24+
func TestMain(m *testing.M) { os.Exit(e2e.RunTestMain(m)) }

0 commit comments

Comments
 (0)