You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Walked docs/upgrade.md from v0.5.0 to main4b4fbb2 on a fresh 3-node GKE cluster under a 12-user pause/resume load, as a rehearsal for what an upgrade e2e has to catch. The runbook says a pre-1.0 minor bump is a reinstall, and the walk confirms it: three steps break in order, each from a change merged after v0.5.0 that was allowed under the no-compat policy. They are the concrete cases a version-to-version e2e should cover once upgrades are supported.
Step 2, atelet roll (50 s, clean by itself). From then on every suspend, resume and delete failed: the v0.5.0 ate-api-server dials atelet.DefaultPort = 8085, the new atelet listens on 21998 (atelet: move default port to an IANA-unassigned port #2665). workflow failed at step CallAteletRestore: ... dial tcp 10.37.192.9:8085: i/o timeout, 49 in the first 3 minutes, all actors stuck RESUMING, later CRASHED by the Locust client's retries. atelet: move default port to an IANA-unassigned port #2665 notes that ateapi and atelet have to move together, which the runbook order (atelet before API server) cannot do.
Step 3, worker roll. The main workers' registration is rejected by the old API server: capacity: Invalid value: unknown field with protobuf tag 3 (strict unknown-field validation per the skew policy), so no worker has room for the actor for every resume.
Step 4a, API server. The new ate-api-server pod crashloops on the install's existing config: parse authentication config: ... json: unknown field "actorIdentityJWTProvider" (field dropped in Drop the unused actorIdentityJWTProvider setting and fix the actor identity docs #2597; the ConfigMap is only rewritten in step 4e). ate-setup deploy apiserver times out at 1/2 replicas and the runbook's answer is to roll back.
Timeline: step 1 at 20:08Z clean; step 2 at 20:10Z, 100% of actor operations failing from 20:10Z; step 4a stuck at 20:21Z. Client-side the v0.5.0 kubectl ate and Locust worked against the old server, while the main kubectl ate could not list actors (cannot parse invalid wire-format data) and the main Locust image could not create them (unknown field with protobuf tag 2), so the e2e also needs the matching client for each side of the skew.
Artifacts: install records and old.yaml/new.yaml in ~/ate-upgrade on my machine, Cloud Logging for cluster stab-upgrade in gke-ai-eco-dev.
Fourth break, step 4e: ate-setup deploy ate-system --config new.yaml from the main checkout refuses the v0.5.0 install record because main added a required setting after v0.5.0: at least one of ATE_INGRESS_ALLOWED_SPIFFE_IDS, config.ingressAllowedSPIFFEIDs or --ingress-allowed-spiffe-ids must be set, got none. The runbook's sed on the record only changes the image tag, so any setting a new release makes mandatory has to be added to new.yaml by hand, and nothing in the runbook says so. An upgrade e2e should run the new ate-setup against the old record before any step touches the cluster.
Same class, one more: after supplying the ingress allowlist, the main ate-setup deploy ate-system next refuses the record for --snapshot-location (SNAPSHOT_LOCATION) must be set unless --preview=PerTemplateSnapshotLocation is enabled. Two settings became mandatory between v0.5.0 and main; both only surface at step 4e, after steps 2 and 3 have already changed the cluster.
Last one from this walk: even a full ate-setup deploy ate-system from main left ate-api-authentication with the v0.5.0 content (actorIdentityJWTProvider: kubernetes still present afterwards), so the main ate-api-server kept crashlooping until I removed the line by hand. If the installer preserves that ConfigMap on purpose so operator edits survive, then a release that drops a field needs either a tolerant parser for one release or a migration step in the runbook. Either way it is a case for the e2e: old config, new binary.
Data layer, after forcing the cluster onto main (ConfigMap fixed by hand, missing settings supplied, authz bootstrap owners added):
Every ActorTemplate written by v0.5.0 is unreadable: CreateActor fails with unmarshaling actor template benchmark-workloads/glutton: proto: cannot parse invalid wire-format data. No actor can be created from a pre-upgrade template.
Actors written by v0.5.0 come back without a status at all through the main API, and the ActorState enum was renumbered in between (CRASHED 7 to 5, DELETING 8 to 6, REVERTING 9 to 7), so a stored CRASHED actor would read as REVERTING if the row were decoded.
The first supported upgrade path therefore also needs a schema and proto compatibility check of stored rows, not only of the wire. Leaving the stab-upgrade cluster and its database up for a day in case anyone wants to look.
Thanks for the walk. v0.5.0 → main is a pre-1.0 minor upgrade, which docs/upgrade.md says needs a reinstall, so these breaks are expected.
#2712 only tests patch upgrades within one minor. Checking the old install record with the new ate-setup before any step touches the cluster is a good idea. I'll track it as a follow-up.
Activity
aditya-shantanu commentedon Oct 21, 2026
Walked
docs/upgrade.mdfrom v0.5.0 tomain4b4fbb2 on a fresh 3-node GKE cluster under a 12-user pause/resume load, as a rehearsal for what an upgrade e2e has to catch. The runbook says a pre-1.0 minor bump is a reinstall, and the walk confirms it: three steps break in order, each from a change merged after v0.5.0 that was allowed under the no-compat policy. They are the concrete cases a version-to-version e2e should cover once upgrades are supported.atelet.DefaultPort= 8085, the new atelet listens on 21998 (atelet: move default port to an IANA-unassigned port #2665).workflow failed at step CallAteletRestore: ... dial tcp 10.37.192.9:8085: i/o timeout, 49 in the first 3 minutes, all actors stuck RESUMING, later CRASHED by the Locust client's retries. atelet: move default port to an IANA-unassigned port #2665 notes that ateapi and atelet have to move together, which the runbook order (atelet before API server) cannot do.capacity: Invalid value: unknown field with protobuf tag 3(strict unknown-field validation per the skew policy), sono worker has room for the actorfor every resume.parse authentication config: ... json: unknown field "actorIdentityJWTProvider"(field dropped in Drop the unused actorIdentityJWTProvider setting and fix the actor identity docs #2597; the ConfigMap is only rewritten in step 4e).ate-setup deploy apiservertimes out at 1/2 replicas and the runbook's answer is to roll back.Timeline: step 1 at 20:08Z clean; step 2 at 20:10Z, 100% of actor operations failing from 20:10Z; step 4a stuck at 20:21Z. Client-side the v0.5.0
kubectl ateand Locust worked against the old server, while the mainkubectl atecould not list actors (cannot parse invalid wire-format data) and the main Locust image could not create them (unknown field with protobuf tag 2), so the e2e also needs the matching client for each side of the skew.Artifacts: install records and
old.yaml/new.yamlin~/ate-upgradeon my machine, Cloud Logging for clusterstab-upgradein gke-ai-eco-dev.aditya-shantanu commentedon Oct 21, 2026
Fourth break, step 4e:
ate-setup deploy ate-system --config new.yamlfrom the main checkout refuses the v0.5.0 install record because main added a required setting after v0.5.0:at least one of ATE_INGRESS_ALLOWED_SPIFFE_IDS, config.ingressAllowedSPIFFEIDs or --ingress-allowed-spiffe-ids must be set, got none. The runbook'ssedon the record only changes the image tag, so any setting a new release makes mandatory has to be added tonew.yamlby hand, and nothing in the runbook says so. An upgrade e2e should run the new ate-setup against the old record before any step touches the cluster.aditya-shantanu commentedon Oct 21, 2026
Same class, one more: after supplying the ingress allowlist, the main
ate-setup deploy ate-systemnext refuses the record for--snapshot-location (SNAPSHOT_LOCATION) must be set unless --preview=PerTemplateSnapshotLocation is enabled. Two settings became mandatory between v0.5.0 and main; both only surface at step 4e, after steps 2 and 3 have already changed the cluster.aditya-shantanu commentedon Oct 21, 2026
Last one from this walk: even a full
ate-setup deploy ate-systemfrom main leftate-api-authenticationwith the v0.5.0 content (actorIdentityJWTProvider: kubernetesstill present afterwards), so the main ate-api-server kept crashlooping until I removed the line by hand. If the installer preserves that ConfigMap on purpose so operator edits survive, then a release that drops a field needs either a tolerant parser for one release or a migration step in the runbook. Either way it is a case for the e2e: old config, new binary.aditya-shantanu commentedon Oct 21, 2026
Data layer, after forcing the cluster onto main (ConfigMap fixed by hand, missing settings supplied, authz bootstrap owners added):
CreateActorfails withunmarshaling actor template benchmark-workloads/glutton: proto: cannot parse invalid wire-format data. No actor can be created from a pre-upgrade template.statusat all through the main API, and theActorStateenum was renumbered in between (CRASHED7 to 5,DELETING8 to 6,REVERTING9 to 7), so a stored CRASHED actor would read as REVERTING if the row were decoded.The first supported upgrade path therefore also needs a schema and proto compatibility check of stored rows, not only of the wire. Leaving the
stab-upgradecluster and its database up for a day in case anyone wants to look.HavenXia commentedon Oct 22, 2026
Thanks for the walk. v0.5.0 → main is a pre-1.0 minor upgrade, which
docs/upgrade.mdsays needs a reinstall, so these breaks are expected.#2712 only tests patch upgrades within one minor. Checking the old install record with the new
ate-setupbefore any step touches the cluster is a good idea. I'll track it as a follow-up.CI: Add the upgrade E2E test for each release (agent-substrate#2712)
CI: Add the upgrade E2E test for each release (#2712)
zlammerts-svg commentedon Nov 5, 2026
Haven Xia (@HavenXia) #2712 is merged. Close this issue now. Thanks for all the work!