microvm-assets: upload the kata assets through the node that hosts rustfs - #2905

…strate#2092)

Stacked on agent-substrate#1838.

## Summary
Ports the microVM dependency setup (`hack/install-microvm-deps.sh` and
`hack/microvm-assets/`) to Go, and folds microVM asset assembly,
staging, and `SandboxConfig` management directly into `deploy
ate-system`, `deploy sandboxconfig`, and `delete ate-system`.

## Changes
- **Fold microVM setup into control-plane deploy/delete
(`cmd/ate-setup/internal/steps/deploy.go`, `delete.go`,
`benchmarks.go`)**:
- Calls `prepareMicroVMDeps` (before cluster resources are modified) and
`applyMicroVMSandboxConfig` inside `DeployAteSystem` (after applying the
system bundle so Kind's in-cluster `rustfs` Deployment and
`rustfs-bucket-init` Job exist before asset staging) and inside
`DeploySandboxConfig`.
- Adds `--sandbox-classes` (default `gvisor,microvm`) to `ate-setup
deploy ate-system` and `ate-setup deploy sandboxconfig`
(`--sandbox-classes=gvisor` opts out of staging microVM assets and
applying `SandboxConfig/microvm`).
- Releases `SandboxConfig` protection finalizers inside
`DeleteAteSystem` before CRD teardown when no `WorkerPool` references
the config.
- Updates `DeployBenchmarks` / `DeleteBenchmarks` since `deploy
ate-system` now installs both `gvisor-default` and `microvm`
`SandboxConfig`s up front, and checks that `sandboxconfig/microvm`
exists when `--sandbox-class=microvm`.
- **Remove `hack/install-microvm-deps.sh` and `hack/microvm-assets/`**:
- Deletes `hack/install-microvm-deps.sh` and the entire
`hack/microvm-assets/` directory (`assemble.sh`, `stage-to-gcs.sh`,
`stage-to-rustfs.sh`, and `README.md`).
- Updates `hack/run-microvm-demo.sh`,
`benchmarking/automation/orchestrator.py`, and
`.github/workflows/pr-workflow.yaml` to rely on `deploy ate-system`.
- **Native Go multi-arch microVM asset assembly and staging
(`cmd/ate-setup/internal/steps/microvm.go`)**:
- Streams and extracts the Kata `.tar.zst` archive (`vmlinux.container`,
`kata-containers-ubuntu.img` as `rootfs.img`, `virtiofsd`) and Cloud
Hypervisor binary in Go (`runtime.GOARCH` on Kind; both `amd64` and
`arm64` on non-Kind clusters) into `bin/microvm-assets/<arch>/`.
- Checks the SHA-256 digests of all four assets (`cloud-hypervisor`,
`virtiofsd`, `vmlinux`, `rootfs.img`) against the digests pinned in
`manifests/ate-install/sandboxconfig-microvm.yaml.tmpl` (both after
assembly and when reusing a cached `.asset-versions` directory).
- Stages assets by content digest under `microvm-assets/<arch>/<sha256>`
in Kind's `rustfs` bucket or GKE's `<bucket>-assets` GCS bucket, then
renders and applies
`manifests/ate-install/sandboxconfig-microvm.yaml.tmpl`.
- **Docs, manifests & unit tests**:
- Moves `sandboxconfig-microvm.yaml.tmpl` to
`manifests/ate-install/sandboxconfig-microvm.yaml.tmpl` with per-arch
`microvm-assets/<arch>/<sha256>` URLs and removes stale version-number
comments.
- Folds the remaining asset-set and snapshot-location notes from
`hack/microvm-assets/README.md` into `docs/dev/microvm-local.md` and
updates `docs/reference/api-guide.md`,
`docs/guides/operator-install.md`, `cmd/ate-setup/README.md`,
`cmd/ate-setup/commands.md`, `cmd/ate-setup/differences.md`,
`demos/counter/README.md`, and `hack/release/draft-release.sh`.
- Adds unit tests in `cmd/ate-setup/internal/steps/microvm_test.go`,
`cmd/ate-setup/internal/steps/benchmarks_test.go`,
`cmd/ate-setup/internal/kube/wait_test.go`, and
`cmd/ate-setup/internal/cmd/deploy_test.go`.

## Testing
- Run `go test -race ./cmd/ate-setup/... ./internal/e2e/...` and
`hack/verify-all.sh`.
- Local Kind cluster verification (`go run ./cmd/ate-setup --kind`):
- Verified `deploy ate-system --sandbox-classes=gvisor` deploys the
control plane and installs only `SandboxConfig/gvisor-default`.
- Verified `deploy ate-system` assembles the microVM guest assets,
stages them into the in-cluster `rustfs` bucket, and applies
`SandboxConfig/microvm` alongside `SandboxConfig/gvisor-default`
(reusing cached assets via `.asset-versions` on subsequent runs).
- Verified `delete all` removes `SandboxConfig/microvm` during
control-plane teardown before deleting CRDs.
- Deployed `deploy demo counter` and ran `./hack/run-e2e-kind.sh
./internal/e2e/suites/demo`.

## Folded in from open PRs
Because this PR replaces `hack/microvm-assets/stage-to-rustfs.sh` with
`stageMicroVMAssetsToRustfs`, it folds in the `stage-to-rustfs.sh` fixes
from two open PRs:
- agent-substrate#2905 (@dims): look up the `app=rustfs` pod's `spec.nodeName` so the
S3 upload container runs in the node network namespace admitted by
rustfs's NetworkPolicy on multi-node Kind clusters.
- agent-substrate#1122 (@ygao-g): use `net.JoinHostPort(svc.Spec.ClusterIP, "9000")`
for `AWS_ENDPOINT_URL` so IPv6 ClusterIPs are bracketed.

AI Use: AI assisted implementation & unit testing, human reviewed,
refined, and locally tested.

---------

Signed-off-by: Noah West <[email protected]>
Co-authored-by: jybao <[email protected]>

Stacked on #1838.

## Summary
Ports the microVM dependency setup (`hack/install-microvm-deps.sh` and
`hack/microvm-assets/`) to Go, and folds microVM asset assembly,
staging, and `SandboxConfig` management directly into `deploy
ate-system`, `deploy sandboxconfig`, and `delete ate-system`.

## Changes
- **Fold microVM setup into control-plane deploy/delete
(`cmd/ate-setup/internal/steps/deploy.go`, `delete.go`,
`benchmarks.go`)**:
- Calls `prepareMicroVMDeps` (before cluster resources are modified) and
`applyMicroVMSandboxConfig` inside `DeployAteSystem` (after applying the
system bundle so Kind's in-cluster `rustfs` Deployment and
`rustfs-bucket-init` Job exist before asset staging) and inside
`DeploySandboxConfig`.
- Adds `--sandbox-classes` (default `gvisor,microvm`) to `ate-setup
deploy ate-system` and `ate-setup deploy sandboxconfig`
(`--sandbox-classes=gvisor` opts out of staging microVM assets and
applying `SandboxConfig/microvm`).
- Releases `SandboxConfig` protection finalizers inside
`DeleteAteSystem` before CRD teardown when no `WorkerPool` references
the config.
- Updates `DeployBenchmarks` / `DeleteBenchmarks` since `deploy
ate-system` now installs both `gvisor-default` and `microvm`
`SandboxConfig`s up front, and checks that `sandboxconfig/microvm`
exists when `--sandbox-class=microvm`.
- **Remove `hack/install-microvm-deps.sh` and `hack/microvm-assets/`**:
- Deletes `hack/install-microvm-deps.sh` and the entire
`hack/microvm-assets/` directory (`assemble.sh`, `stage-to-gcs.sh`,
`stage-to-rustfs.sh`, and `README.md`).
- Updates `hack/run-microvm-demo.sh`,
`benchmarking/automation/orchestrator.py`, and
`.github/workflows/pr-workflow.yaml` to rely on `deploy ate-system`.
- **Native Go multi-arch microVM asset assembly and staging
(`cmd/ate-setup/internal/steps/microvm.go`)**:
- Streams and extracts the Kata `.tar.zst` archive (`vmlinux.container`,
`kata-containers-ubuntu.img` as `rootfs.img`, `virtiofsd`) and Cloud
Hypervisor binary in Go (`runtime.GOARCH` on Kind; both `amd64` and
`arm64` on non-Kind clusters) into `bin/microvm-assets/<arch>/`.
- Checks the SHA-256 digests of all four assets (`cloud-hypervisor`,
`virtiofsd`, `vmlinux`, `rootfs.img`) against the digests pinned in
`manifests/ate-install/sandboxconfig-microvm.yaml.tmpl` (both after
assembly and when reusing a cached `.asset-versions` directory).
- Stages assets by content digest under `microvm-assets/<arch>/<sha256>`
in Kind's `rustfs` bucket or GKE's `<bucket>-assets` GCS bucket, then
renders and applies
`manifests/ate-install/sandboxconfig-microvm.yaml.tmpl`.
- **Docs, manifests & unit tests**:
- Moves `sandboxconfig-microvm.yaml.tmpl` to
`manifests/ate-install/sandboxconfig-microvm.yaml.tmpl` with per-arch
`microvm-assets/<arch>/<sha256>` URLs and removes stale version-number
comments.
- Folds the remaining asset-set and snapshot-location notes from
`hack/microvm-assets/README.md` into `docs/dev/microvm-local.md` and
updates `docs/reference/api-guide.md`,
`docs/guides/operator-install.md`, `cmd/ate-setup/README.md`,
`cmd/ate-setup/commands.md`, `cmd/ate-setup/differences.md`,
`demos/counter/README.md`, and `hack/release/draft-release.sh`.
- Adds unit tests in `cmd/ate-setup/internal/steps/microvm_test.go`,
`cmd/ate-setup/internal/steps/benchmarks_test.go`,
`cmd/ate-setup/internal/kube/wait_test.go`, and
`cmd/ate-setup/internal/cmd/deploy_test.go`.

## Testing
- Run `go test -race ./cmd/ate-setup/... ./internal/e2e/...` and
`hack/verify-all.sh`.
- Local Kind cluster verification (`go run ./cmd/ate-setup --kind`):
- Verified `deploy ate-system --sandbox-classes=gvisor` deploys the
control plane and installs only `SandboxConfig/gvisor-default`.
- Verified `deploy ate-system` assembles the microVM guest assets,
stages them into the in-cluster `rustfs` bucket, and applies
`SandboxConfig/microvm` alongside `SandboxConfig/gvisor-default`
(reusing cached assets via `.asset-versions` on subsequent runs).
- Verified `delete all` removes `SandboxConfig/microvm` during
control-plane teardown before deleting CRDs.
- Deployed `deploy demo counter` and ran `./hack/run-e2e-kind.sh
./internal/e2e/suites/demo`.

## Folded in from open PRs
Because this PR replaces `hack/microvm-assets/stage-to-rustfs.sh` with
`stageMicroVMAssetsToRustfs`, it folds in the `stage-to-rustfs.sh` fixes
from two open PRs:
- #2905 (@dims): look up the `app=rustfs` pod's `spec.nodeName` so the
S3 upload container runs in the node network namespace admitted by
rustfs's NetworkPolicy on multi-node Kind clusters.
- #1122 (@ygao-g): use `net.JoinHostPort(svc.Spec.ClusterIP, "9000")`
for `AWS_ENDPOINT_URL` so IPv6 ClusterIPs are bracketed.

AI Use: AI assisted implementation & unit testing, human reviewed,
refined, and locally tested.

---------

Signed-off-by: Noah West <[email protected]>
Co-authored-by: jybao <[email protected]>
(cherry picked from commit 8edd9ae)
Signed-off-by: Eitan Yarmush <[email protected]>