Repository navigation
ate-setup: port microvm-deps installation from shell to Go - #2092
Conversation
3d4777d to
1727148
Compare
Wiz Scan Summary
To detect these findings earlier in the dev lifecycle, try the Wiz Code extension for VS Code, JetBrains, or Visual Studio. |
Huy Pham (hdp617)
left a comment
Huy Pham (hdp617)
left a comment
There was a problem hiding this comment.
Instead of having deploy/delete microvm-deps, how about doing these steps as part of deploy ate-system?
I like it because we want microVM SandboxClass installed by default. Also, the staging steps are needed at the moment because we don't have the microVM assets publicly available yet. Once they are, we can adjust deploy ate-system without any user facing changes.
| from, and applies the cluster-wide `microvm` SandboxConfig. `deploy ate-system` | ||
| installs only `gvisor-default`, so this is needed before any micro-VM demo or |
There was a problem hiding this comment.
I think we want to install microvm SandboxConfig by default om deploy ate-system.
There was a problem hiding this comment.
I think this is a good change, I've updated the code to match. I also added the opt-out flag like you mentioned in the PR this is stacked upon. It also simplifies the changes to make substrate-gke match, so that's a nice plus.
There was a problem hiding this comment.
LGTM from the upgrade side: step 4d now picks up the microvm SandboxConfig through deploy sandboxconfig.
deploy ate-system runs at upgrade step 4e so it will redeploy the microvm sandboxConfig again, which is no-op but is also ok. (I am thinking about merging upgrade 4d and 4e though).
Also cc Zoe Zhao (@zoez7) as SandboxConfig will carry different name per release soon.
|
/assign |
| go install ./cmd/kubectl-ate | ||
|
|
||
| # Create the actor from the counter template. | ||
| kubectl ate create actor my-counter-1 -a ate-demo-counter --template counter |
There was a problem hiding this comment.
Again I believe this is slightly wrong for the microvm case. kubectl ate create actor my-counter-microvm-1 -a ate-demo-counter-microvm --template counter-microvm
There was a problem hiding this comment.
(same as other README.md comment - this is for the gvisor demo, adjusted the doc to make it more clear)
| defaultCloudHypervisorVersion = "v53.0" | ||
| // virtiofsdVersion is the version bundled inside kata-static for | ||
| // defaultKataVersion (not env-overridable; verified before stamping). | ||
| virtiofsdVersion = "1.14.0" |
There was a problem hiding this comment.
Ditto
hack/microvm-assets/assemble.sh:30:# to and including 4.0.0; kata 4.1.0 bundles v1.14.0, the first release carrying the
hack/microvm-assets/assemble.sh:56:VIRTIOFSD_VER="1.14.0"
bin/microvm-assets/amd64/.asset-versions:4:virtiofsd=1.14.0
There was a problem hiding this comment.
same here, .../steps/microvm.go becomes the single source of truth for the virtiofsd version from this PR. assemble.sh gets deleted and .asset-versions is a local gitignored file from build time.
|
need rebase |
a39fc51 to
1d184c3
Compare
Walter Fender (cheftako)
left a comment
Walter Fender (cheftako)
left a comment
There was a problem hiding this comment.
The port from shell to Go in cmd/ate-setup/internal/steps/microvm.go is well-implemented and maintains backwards compatibility through the flag-translation shim.
Two minor discrepancies between the PR description and implementation should be noted:
- The PR description claims Cloud Hypervisor SHA-256 verification and staging to rustfs via
docker cp+ an in-cluster Job, whereas the implementation mirrors the original scripts (logging the calculated asset hashes for manifest authoring and streaming stdin directly to theamazon/aws-clicontainer).
Walter Fender (cheftako)
left a comment
Walter Fender (cheftako)
left a comment
There was a problem hiding this comment.
Looks good.
e685e2d to
ae44cc0
Compare
Huy Pham (hdp617)
left a comment
Huy Pham (hdp617)
left a comment
There was a problem hiding this comment.
LGTM. Thank you!
ae44cc0 to
666ca16
Compare
Bowei Du (bowei)
added
area/dev-infra
area/microVM
kind/cleanup
|
need rebase |
666ca16 to
ed3e67f
Compare
Tim Hockin (thockin)
left a comment
Tim Hockin (thockin)
left a comment
There was a problem hiding this comment.
Agent comments, seem minor
| func microvmAssetsNeedAssemble(outDir, wantStamp string, digests map[string]string) (needAssemble, stale bool, err error) { | ||
| if err := checkMicroVMAssets(outDir); err != nil { | ||
| return true, false, nil | ||
| } | ||
| got, err := os.ReadFile(filepath.Join(outDir, assetStampFile)) | ||
| if err != nil || string(got) != wantStamp { | ||
| return true, true, nil | ||
| } | ||
| if err := verifyMicroVMAssetDigests(outDir, digests); err != nil { | ||
| return false, false, err | ||
| } | ||
| return false, false, nil | ||
| } |
There was a problem hiding this comment.
🤖 should-fix 🟡 – Now that prepareMicroVMDeps loads the per-file SHA-256 digests from sandboxconfig-microvm.yaml.tmpl and hashes all four files on every run, .asset-versions (microvmAssetStamp) is redundant and introduces two failure modes:
- If the four files in
bin/microvm-assets/<arch>/already match the pinned digests, but.asset-versionsis missing or in the old 4-line format fromassemble.sh,microvmAssetsNeedAssembletreats a valid directory as stale and re-downloads ~600MB. - If
.asset-versionsmatcheskata=4.3.1andcloud-hypervisor=v53.0, but a file on disk is corrupt orsandboxconfig-microvm.yaml.tmplupdated a digest under the same version,microvmAssetsNeedAssemblereturns a hard error instead of settingneedAssemble = true. That wedgesdeploy ate-systemuntil the user manually deletesbin/microvm-assets.
Please drop .asset-versions (assetStampFile, microvmAssetStamp, and the stamp tests) and treat verifyMicroVMAssetDigests(outDir, digests) != nil before assembly as the trigger to re-assemble, then run verifyMicroVMAssetDigests again after downloading.
| if microvmPrep != nil { | ||
| if err := e.applyMicroVMSandboxConfig(ctx, microvmPrep); err != nil { | ||
| return err | ||
| } | ||
| } |
There was a problem hiding this comment.
🤖 should-fix 🟡 – DeploySandboxConfig is split across three places in DeployAteSystem (prepareMicroVMDeps at cmd/ate-setup/internal/steps/deploy.go:102, applyInitialSandboxConfigs at cmd/ate-setup/internal/steps/deploy.go:119, and applyMicroVMSandboxConfig here), and the responsibilities do not match the function names:
prepareMicroVMDepssounds like local preflight and assembly, yet on GKE it uploads the assets to GCS (stageMicroVMAssetsToGCS), while on Kind it does not stage anything.applyMicroVMSandboxConfigsounds like it only applies theSandboxConfigmanifest, yet on Kind it stages the assets torustfs(stageMicroVMAssetsToRustfs).
Can we keep prepareMicroVMDeps purely local (preflight + local asset assembly) and do object-store staging (GCS or rustfs) + manifest apply together in one step after the system bundle is applied?
Regardless, please add a comment here stating the precondition for why applyMicroVMSandboxConfig runs after applyWithAPIServerEnvHash rather than alongside applyInitialSandboxConfigs at line 119: we applied the system bundle above, so on Kind the rustfs Deployment and rustfs-bucket-init Job now exist for asset staging.
| for _, p := range pods.Items { | ||
| if p.DeletionTimestamp != nil || p.Status.Phase == corev1.PodSucceeded || p.Status.Phase == corev1.PodFailed { | ||
| continue | ||
| } | ||
| if p.Spec.NodeName != "" { | ||
| return p.Spec.NodeName, nil | ||
| } | ||
| } | ||
| return e.kindNodeName(ctx) |
There was a problem hiding this comment.
🤖 should-fix 🟡 – stageMicroVMAssetsToRustfs calls rustfsNodeName immediately after RolloutStatus succeeds for deploy/rustfs, so we know at least one non-terminating app=rustfs pod is scheduled on a node (p.Spec.NodeName != "").
If that precondition is violated, falling back to e.kindNodeName(ctx) picks an arbitrary worker node. On a multi-node Kind cluster where rustfs is on another node, rustfs's NetworkPolicy will drop the upload container's traffic and fail later with an opaque S3 connection timeout. Please state the precondition in a comment and return an explicit error (fmt.Errorf("no scheduled app=rustfs pod found in namespace %s", ns)) instead of falling back to e.kindNodeName(ctx).
| log.Infof(">> Done. Verify:") | ||
| return awsCLI(nil, "s3", "ls", "--recursive", prefixURL) |
There was a problem hiding this comment.
🤖 nit 🟢 – Both stageMicroVMAssetsToRustfs and stageMicroVMAssetsToGCS (cmd/ate-setup/internal/steps/microvm.go:731) already run ls --recursive up front and check the exit code of every cp call. Spawning a second docker run ... aws s3 ls container (or gcloud storage ls subprocess) at the end to print stdout under >> Done. Verify: is a leftover from the interactive shell scripts. Since deploy ate-system runs this on every install, even when all four assets were already present and skipped, please drop both trailing ls calls.
| func fieldsSet(s string) map[string]bool { | ||
| fields := strings.Fields(s) | ||
| out := make(map[string]bool, len(fields)) | ||
| for _, f := range fields { | ||
| out[f] = true | ||
| } | ||
| return out | ||
| } |
There was a problem hiding this comment.
🤖 nit 🟢 – stageMicroVMAssetsToRustfs looks up bare keys (microvm-assets/<arch>/<sha>) in existing, whereas stageMicroVMAssetsToGCS looks up full URLs (gs://<bucket>/microvm-assets/<arch>/<sha>). That works because aws s3 ls --recursive prints <date> <time> <size> <key> on each line while gcloud storage ls --recursive prints <gs-url>, and our keys never contain whitespace. Please add a brief comment on fieldsSet explaining this so the difference between the two callers is clear.
| func downloadAndExtractKataAssets(ctx context.Context, kataURL, workDir, outDir string) error { | ||
| req, err := http.NewRequestWithContext(ctx, http.MethodGet, kataURL, nil) | ||
| if err != nil { | ||
| return err | ||
| } | ||
| resp, err := http.DefaultClient.Do(req) | ||
| if err != nil { | ||
| return fmt.Errorf("while downloading %s: %w", kataURL, err) | ||
| } | ||
| defer resp.Body.Close() | ||
| if resp.StatusCode != http.StatusOK { | ||
| return fmt.Errorf("while downloading %s: unexpected HTTP status %s", kataURL, resp.Status) | ||
| } |
There was a problem hiding this comment.
🤖 should-fix 🟡 – downloadAndExtractKataAssets and downloadToFile (cmd/ate-setup/internal/steps/microvm.go:583) duplicate the http.NewRequestWithContext, Do, and StatusOK checks, and both use http.DefaultClient (which has no timeout, while deploy ate-system runs with a background context that has no deadline). On non-Kind clusters a cold cache downloads ~1.2 GB across four release artifacts, so a stalled connection will hang deploy ate-system indefinitely. Please extract a small helper that opens the HTTP response stream with a bounded timeout.
| func (o DeployOptions) Validate() error { | ||
| return ValidateCSIDriver(o.SetupCSI) | ||
| if err := ValidateCSIDriver(o.SetupCSI); err != nil { | ||
| return err | ||
| } | ||
| return ValidateSandboxClasses(o.SandboxClasses) | ||
| } |
There was a problem hiding this comment.
🤖 nit 🟢 – Two small cleanups on DeployOptions and parseSandboxClasses (cmd/ate-setup/internal/steps/microvm.go:102):
DeployOptions.Validate()has no callers anywhere in the repository (cmd/deploy.govalidates each field directly so it can wrap errors inconfig.InvalidError). Please deleteDeployOptions.Validate().- Because
ValidateSandboxClassesdelegates toparseSandboxClasses, passing an explicit empty--sandbox-classes=""flag on the CLI passes validation and silently installs bothgvisorandmicrovm. IfparseSandboxClassesrejects empty strings andDeployAteSystemdefaults an emptyopts.SandboxClassestoDefaultSandboxClasses,--sandbox-classes=""will fail loudly as expected.
agent-substrate/substrate#2092 removes hack/install-microvm-deps.sh, and `ate-setup deploy sandboxconfig` stages the micro-VM assets instead. It goes into v1.0.0-rc.2. The micro-VM step runs `ate-setup deploy sandboxconfig`. The control-plane deploy sets ATE_INSTALL_SANDBOX_CLASSES=gvisor, so it stages nothing for micro-VM before the user picks it. Every ate-setup deploy replaces the install record the upgrade flow reads, so the micro-VM step passes the control-plane settings and records both classes. Signed-off-by: Haven Xia <[email protected]>
…k/run-microvm-demo-kind.sh (agent-substrate#3509) Since agent-substrate#2092, `deploy ate-system` stages the microVM guest assets and applies the `microvm` SandboxConfig, so both scripts can be replaced by `deploy ate-system` now. This PR deletes `hack/run-microvm-demo.sh` and `hack/run-microvm-demo-kind.sh`. Tested on kind, this fixed the broken counter microVM demo, which failed before due to missing `ATE_CREDENTIAL_PROVIDER`. AI usage: test. - [ ] Tests pass - [ ] Docs updated Signed-off-by: Haven Xia <[email protected]>
Stacked on #1838. ## Summary Ports the microVM dependency setup (`hack/install-microvm-deps.sh` and `hack/microvm-assets/`) to Go, and folds microVM asset assembly, staging, and `SandboxConfig` management directly into `deploy ate-system`, `deploy sandboxconfig`, and `delete ate-system`. ## Changes - **Fold microVM setup into control-plane deploy/delete (`cmd/ate-setup/internal/steps/deploy.go`, `delete.go`, `benchmarks.go`)**: - Calls `prepareMicroVMDeps` (before cluster resources are modified) and `applyMicroVMSandboxConfig` inside `DeployAteSystem` (after applying the system bundle so Kind's in-cluster `rustfs` Deployment and `rustfs-bucket-init` Job exist before asset staging) and inside `DeploySandboxConfig`. - Adds `--sandbox-classes` (default `gvisor,microvm`) to `ate-setup deploy ate-system` and `ate-setup deploy sandboxconfig` (`--sandbox-classes=gvisor` opts out of staging microVM assets and applying `SandboxConfig/microvm`). - Releases `SandboxConfig` protection finalizers inside `DeleteAteSystem` before CRD teardown when no `WorkerPool` references the config. - Updates `DeployBenchmarks` / `DeleteBenchmarks` since `deploy ate-system` now installs both `gvisor-default` and `microvm` `SandboxConfig`s up front, and checks that `sandboxconfig/microvm` exists when `--sandbox-class=microvm`. - **Remove `hack/install-microvm-deps.sh` and `hack/microvm-assets/`**: - Deletes `hack/install-microvm-deps.sh` and the entire `hack/microvm-assets/` directory (`assemble.sh`, `stage-to-gcs.sh`, `stage-to-rustfs.sh`, and `README.md`). - Updates `hack/run-microvm-demo.sh`, `benchmarking/automation/orchestrator.py`, and `.github/workflows/pr-workflow.yaml` to rely on `deploy ate-system`. - **Native Go multi-arch microVM asset assembly and staging (`cmd/ate-setup/internal/steps/microvm.go`)**: - Streams and extracts the Kata `.tar.zst` archive (`vmlinux.container`, `kata-containers-ubuntu.img` as `rootfs.img`, `virtiofsd`) and Cloud Hypervisor binary in Go (`runtime.GOARCH` on Kind; both `amd64` and `arm64` on non-Kind clusters) into `bin/microvm-assets/<arch>/`. - Checks the SHA-256 digests of all four assets (`cloud-hypervisor`, `virtiofsd`, `vmlinux`, `rootfs.img`) against the digests pinned in `manifests/ate-install/sandboxconfig-microvm.yaml.tmpl` (both after assembly and when reusing a cached `.asset-versions` directory). - Stages assets by content digest under `microvm-assets/<arch>/<sha256>` in Kind's `rustfs` bucket or GKE's `<bucket>-assets` GCS bucket, then renders and applies `manifests/ate-install/sandboxconfig-microvm.yaml.tmpl`. - **Docs, manifests & unit tests**: - Moves `sandboxconfig-microvm.yaml.tmpl` to `manifests/ate-install/sandboxconfig-microvm.yaml.tmpl` with per-arch `microvm-assets/<arch>/<sha256>` URLs and removes stale version-number comments. - Folds the remaining asset-set and snapshot-location notes from `hack/microvm-assets/README.md` into `docs/dev/microvm-local.md` and updates `docs/reference/api-guide.md`, `docs/guides/operator-install.md`, `cmd/ate-setup/README.md`, `cmd/ate-setup/commands.md`, `cmd/ate-setup/differences.md`, `demos/counter/README.md`, and `hack/release/draft-release.sh`. - Adds unit tests in `cmd/ate-setup/internal/steps/microvm_test.go`, `cmd/ate-setup/internal/steps/benchmarks_test.go`, `cmd/ate-setup/internal/kube/wait_test.go`, and `cmd/ate-setup/internal/cmd/deploy_test.go`. ## Testing - Run `go test -race ./cmd/ate-setup/... ./internal/e2e/...` and `hack/verify-all.sh`. - Local Kind cluster verification (`go run ./cmd/ate-setup --kind`): - Verified `deploy ate-system --sandbox-classes=gvisor` deploys the control plane and installs only `SandboxConfig/gvisor-default`. - Verified `deploy ate-system` assembles the microVM guest assets, stages them into the in-cluster `rustfs` bucket, and applies `SandboxConfig/microvm` alongside `SandboxConfig/gvisor-default` (reusing cached assets via `.asset-versions` on subsequent runs). - Verified `delete all` removes `SandboxConfig/microvm` during control-plane teardown before deleting CRDs. - Deployed `deploy demo counter` and ran `./hack/run-e2e-kind.sh ./internal/e2e/suites/demo`. ## Folded in from open PRs Because this PR replaces `hack/microvm-assets/stage-to-rustfs.sh` with `stageMicroVMAssetsToRustfs`, it folds in the `stage-to-rustfs.sh` fixes from two open PRs: - #2905 (@dims): look up the `app=rustfs` pod's `spec.nodeName` so the S3 upload container runs in the node network namespace admitted by rustfs's NetworkPolicy on multi-node Kind clusters. - #1122 (@ygao-g): use `net.JoinHostPort(svc.Spec.ClusterIP, "9000")` for `AWS_ENDPOINT_URL` so IPv6 ClusterIPs are bracketed. AI Use: AI assisted implementation & unit testing, human reviewed, refined, and locally tested. --------- Signed-off-by: Noah West <[email protected]> Co-authored-by: jybao <[email protected]> (cherry picked from commit 8edd9ae) Signed-off-by: Eitan Yarmush <[email protected]>
…k/run-microvm-demo-kind.sh (#3509) Since #2092, `deploy ate-system` stages the microVM guest assets and applies the `microvm` SandboxConfig, so both scripts can be replaced by `deploy ate-system` now. This PR deletes `hack/run-microvm-demo.sh` and `hack/run-microvm-demo-kind.sh`. Tested on kind, this fixed the broken counter microVM demo, which failed before due to missing `ATE_CREDENTIAL_PROVIDER`. AI usage: test. - [ ] Tests pass - [ ] Docs updated Signed-off-by: Haven Xia <[email protected]> (cherry picked from commit 29af06e) Signed-off-by: Eitan Yarmush <[email protected]>
Core agent-substrate/substrate#2092 ported hack/install-microvm-deps.sh into ate-setup and deleted the script. The pinned v1.0.0-rc.1 predates it, so the micro-VM step now branches on the script's presence: the script on older trees, 'ate-setup deploy sandboxconfig --sandbox-classes=microvm' on newer ones. deploy ate-system gets ATE_INSTALL_SANDBOX_CLASSES=gvisor so micro-VM stays opt-in; older trees ignore the variable. The progress checklist matches both outputs, the dry run uses the -assets bucket, and the README covers both paths. Part of #99. Signed-off-by: Grant McCloskey <[email protected]>
- Check the asset bucket with 'gcloud storage ls', which needs only storage.objects.list, so object-scoped grants pass the pre-check. - Require BUCKET_NAME only for --install; --delete removes the config by name and needs no bucket. - Name the too-new case: a tree from agent-substrate/substrate#2092 on has cmd/ate-setup but no assemble.sh or template, so point at 'ate-setup deploy sandboxconfig' instead of calling it the wrong directory. Signed-off-by: Grant McCloskey <[email protected]>
Step 4's ingress allowlist was a placeholder, so the counter demo's driver, which calls its actor through the router, was refused. List the demo drivers' IDs, as the installer does. Fix the moved operator-install link. Describe the micro-VM staging deploy ate-system does by default from agent-substrate/substrate#2092, --sandbox-classes, pre-built release images, and the micro-VM demo. Signed-off-by: Grant McCloskey <[email protected]>
The "stage micro-VM assets" step failed on a workstation with gcloud's context-aware access enabled (context_aware/use_client_certificate = true): after the 50 s asset download, `gcloud storage cp` died with `[SSL] PEM lib (_ssl.c:4181)` three times and `Temporary components were not uploaded correctly`. gcloud splits each large upload into components sent by worker processes, and each worker reads the per-invocation client certificate file the parent gcloud writes and removes. Half the assets and the gcloud/tmp/parallel_composite_uploads/ parts were left in the asset bucket, the SandboxConfig was never applied, and the wizard's [s] reset the sandbox choice to gVisor. Replaying the same step with CLOUDSDK_STORAGE_PARALLEL_COMPOSITE_UPLOAD_ENABLED=False in the environment uploaded all four assets and created the SandboxConfig. microVMEnv now sets that variable, so both the staging step and the demo counter-microvm step carry it. It reaches both staging paths: the pinned v1.0.0-rc.1 tree's hack/install-microvm-deps.sh (whose stage-to-gcs.sh runs a bare `gcloud storage cp`) and, on trees from agent-substrate/substrate#2092 on, `ate-setup deploy sandboxconfig`, whose Go code shells out to the same command with an environment built from os.Environ(). gcloud storage cp has no flag for this; the storage/parallel_composite_upload_enabled property is the only switch, and sequential upload of four objects costs nothing noticeable. The env test for the two micro-VM specs asserts the variable, and the README's hand-run re-staging commands say to set it too. Fixes #111 Co-authored-by: Aditya Shantanu <[email protected]>
Move Commit from the v1.0.0-rc.1 tag (f86fde9) to v1.0.0-rc.2 (e59281c766d547689482a1fce666ca3f0f6bd008), 55 commits later. go.mod is unchanged (go 1.27.0), so MinGoVersion stays; substrate-pin-check agrees. What rc.2 changes for the installer: - hack/install-microvm-deps.sh is gone (agent-substrate/substrate#2092): micro-VM staging is `ate-setup deploy sandboxconfig --sandbox-classes`, which assembles both architectures in-process and uploads them with `gcloud storage cp` to <bucket>-assets/microvm-assets/<arch>/<sha256>. InstallMicroVMDeps already branched on the script's presence, so the pin now takes the ate-setup branch with no change to the command; the dry run replays that branch's output instead of the script's, and a test pins it there. The legacy branch stays for v1.0.0-rc.1 and older commits typed into the images step. The step's environment already carries BUCKET_NAME, ATE_INSTALL_KIND=false and CLOUDSDK_STORAGE_PARALLEL_COMPOSITE_UPLOAD_ENABLED=False, which ate-setup passes through to gcloud. - `deploy ate-system` defaults --sandbox-classes to gvisor,microvm and stages the micro-VM assets before touching the cluster. The control-plane step already passes ATE_INSTALL_SANDBOX_CLASSES=gvisor, so a gVisor-only install still stages nothing. - On GKE the only tool ate-setup's micro-VM path needs is gcloud; the doctor's sha256sum/zstd/unzip/jq/make row now says it applies to trees before v1.0.0-rc.2 only, and stays for them. - Upstream removed its tools/setup-gcp and hack/teardown.sh (#3505); this repository runs its own copies, so bootstrap and cleanup-gcp are unaffected. docs/guides/upgrade.md is unchanged, so the upgrade track mirrors the same runbook. The CSIDriverConfig CRD still serves only ate.dev/v1, so the Filestore step still stops at its API check. Nothing else the installer passes or parses changed: the ate-setup flags, the delete and demo commands, the cordon and placement checks, and the install record format are as at rc.1. Verified: make substrate-pin-check, make verify, go test ./... in tools/setup-gcp, and in installer/ gofmt, go vet and go test for the root, internal and uishots packages, the ui suite included. The dry-run frames render; guard.svg is regenerated for the new short commit and the other four are byte-identical. TestFetchTreesFetchesBothCommits fails on this machine before and after the change, for an environmental reason unrelated to the pin.
Stacked on #1838.
Summary
Ports the microVM dependency setup (
hack/install-microvm-deps.shandhack/microvm-assets/) to Go, and folds microVM asset assembly, staging, andSandboxConfigmanagement directly intodeploy ate-system,deploy sandboxconfig, anddelete ate-system.Changes
cmd/ate-setup/internal/steps/deploy.go,delete.go,benchmarks.go):prepareMicroVMDeps(before cluster resources are modified) andapplyMicroVMSandboxConfiginsideDeployAteSystem(after applying the system bundle so Kind's in-clusterrustfsDeployment andrustfs-bucket-initJob exist before asset staging) and insideDeploySandboxConfig.--sandbox-classes(defaultgvisor,microvm) toate-setup deploy ate-systemandate-setup deploy sandboxconfig(--sandbox-classes=gvisoropts out of staging microVM assets and applyingSandboxConfig/microvm).SandboxConfigprotection finalizers insideDeleteAteSystembefore CRD teardown when noWorkerPoolreferences the config.DeployBenchmarks/DeleteBenchmarkssincedeploy ate-systemnow installs bothgvisor-defaultandmicrovmSandboxConfigs up front, and checks thatsandboxconfig/microvmexists when--sandbox-class=microvm.hack/install-microvm-deps.shandhack/microvm-assets/:hack/install-microvm-deps.shand the entirehack/microvm-assets/directory (assemble.sh,stage-to-gcs.sh,stage-to-rustfs.sh, andREADME.md).hack/run-microvm-demo.sh,benchmarking/automation/orchestrator.py, and.github/workflows/pr-workflow.yamlto rely ondeploy ate-system.cmd/ate-setup/internal/steps/microvm.go):.tar.zstarchive (vmlinux.container,kata-containers-ubuntu.imgasrootfs.img,virtiofsd) and Cloud Hypervisor binary in Go (runtime.GOARCHon Kind; bothamd64andarm64on non-Kind clusters) intobin/microvm-assets/<arch>/.cloud-hypervisor,virtiofsd,vmlinux,rootfs.img) against the digests pinned inmanifests/ate-install/sandboxconfig-microvm.yaml.tmpl(both after assembly and when reusing a cached.asset-versionsdirectory).microvm-assets/<arch>/<sha256>in Kind'srustfsbucket or GKE's<bucket>-assetsGCS bucket, then renders and appliesmanifests/ate-install/sandboxconfig-microvm.yaml.tmpl.sandboxconfig-microvm.yaml.tmpltomanifests/ate-install/sandboxconfig-microvm.yaml.tmplwith per-archmicrovm-assets/<arch>/<sha256>URLs and removes stale version-number comments.hack/microvm-assets/README.mdintodocs/dev/microvm-local.mdand updatesdocs/reference/api-guide.md,docs/guides/operator-install.md,cmd/ate-setup/README.md,cmd/ate-setup/commands.md,cmd/ate-setup/differences.md,demos/counter/README.md, andhack/release/draft-release.sh.cmd/ate-setup/internal/steps/microvm_test.go,cmd/ate-setup/internal/steps/benchmarks_test.go,cmd/ate-setup/internal/kube/wait_test.go, andcmd/ate-setup/internal/cmd/deploy_test.go.Testing
go test -race ./cmd/ate-setup/... ./internal/e2e/...andhack/verify-all.sh.go run ./cmd/ate-setup --kind):deploy ate-system --sandbox-classes=gvisordeploys the control plane and installs onlySandboxConfig/gvisor-default.deploy ate-systemassembles the microVM guest assets, stages them into the in-clusterrustfsbucket, and appliesSandboxConfig/microvmalongsideSandboxConfig/gvisor-default(reusing cached assets via.asset-versionson subsequent runs).delete allremovesSandboxConfig/microvmduring control-plane teardown before deleting CRDs.deploy demo counterand ran./hack/run-e2e-kind.sh ./internal/e2e/suites/demo.Folded in from open PRs
Because this PR replaces
hack/microvm-assets/stage-to-rustfs.shwithstageMicroVMAssetsToRustfs, it folds in thestage-to-rustfs.shfixes from two open PRs:app=rustfspod'sspec.nodeNameso the S3 upload container runs in the node network namespace admitted by rustfs's NetworkPolicy on multi-node Kind clusters.net.JoinHostPort(svc.Spec.ClusterIP, "9000")forAWS_ENDPOINT_URLso IPv6 ClusterIPs are bracketed.AI Use: AI assisted implementation & unit testing, human reviewed, refined, and locally tested.