You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Found in an end-to-end run of the installer (v1.0.0-rc.1 install, micro-VM sandbox chosen).
Symptom
The "stage micro-VM assets" step runs upstream's hack/install-microvm-deps.sh --install (via hack/microvm-assets/stage-to-gcs.sh), which uploads each asset with a bare gcloud storage cp. On a machine with gcloud's context-aware access enabled (context_aware/use_client_certificate = true), the upload fails after the ~50 s asset download:
ERROR: [SSL] PEM lib (_ssl.c:4181)
ERROR: [SSL] PEM lib (_ssl.c:4181)
ERROR: [SSL] PEM lib (_ssl.c:4181)
ERROR: Temporary components were not uploaded correctly. Please retry this upload.
The step fails, so the SandboxConfig is never applied and the wizard's [s] resets the sandbox choice to gVisor. Half the assets plus gcloud/tmp/parallel_composite_uploads/… component objects are left behind in gs://<bucket>-assets.
Root cause
gcloud storage cp uses parallel composite uploads for large objects. The worker processes it forks each read gcloud's per-invocation context-aware client certificate file (caa_cert.pem), which the parent removes on exit; with the cert gone or half-written the workers' TLS handshakes fail with PEM lib, and the composite upload is abandoned.
Verified workaround
Replaying the identical step with CLOUDSDK_STORAGE_PARALLEL_COMPOSITE_UPLOAD_ENABLED=False in the environment succeeded: all four assets uploaded at ~30 MiB/s and sandboxconfig.ate.dev/microvm created.
Fix
Set CLOUDSDK_STORAGE_PARALLEL_COMPOSITE_UPLOAD_ENABLED=False in the environment the installer passes to the micro-VM steps (microVMEnv in installer/internal/snapshot/snapshot.go). gcloud storage cp has no per-command flag for this; the storage/parallel_composite_upload_enabled property (and its CLOUDSDK_STORAGE_… env form) is the only switch.
This covers both staging paths the step can take:
the pinned v1.0.0-rc.1 tree's hack/install-microvm-deps.sh → stage-to-gcs.sh, and
newer trees' ate-setup deploy sandboxconfig, whose Go code also shells out to gcloud storage cp (cmd/ate-setup/internal/steps/microvm.go) with an environment derived from os.Environ(), so the variable reaches it too.
#88 brings a local copy of the staging script into tools/setup-gcp/install-microvm-deps.sh with the same gcloud storage cp call; when it lands, its invocation should either inherit this environment or set the property itself.
Evidence
Observed in an end-to-end run; the failing step and the successful replay with the variable set were captured from the same cluster and bucket.
Activity
[-]Micro-VM asset upload fails on corp machines: gcloud parallel composite uploads break with context-aware client certs[/-][+]Micro-VM asset upload fails with context-aware client certificates: gcloud parallel composite uploads break[/+]Stage micro-VM assets without gcloud's parallel composite uploads (ai…