Skip to content

Micro-VM asset upload fails with context-aware client certificates: gcloud parallel composite uploads break #111

Closed

Description

@aditya-shantanu

Found in an end-to-end run of the installer (v1.0.0-rc.1 install, micro-VM sandbox chosen).

Symptom

The "stage micro-VM assets" step runs upstream's hack/install-microvm-deps.sh --install (via hack/microvm-assets/stage-to-gcs.sh), which uploads each asset with a bare gcloud storage cp. On a machine with gcloud's context-aware access enabled (context_aware/use_client_certificate = true), the upload fails after the ~50 s asset download:

ERROR: [SSL] PEM lib (_ssl.c:4181)
ERROR: [SSL] PEM lib (_ssl.c:4181)
ERROR: [SSL] PEM lib (_ssl.c:4181)
ERROR: Temporary components were not uploaded correctly. Please retry this upload.

The step fails, so the SandboxConfig is never applied and the wizard's [s] resets the sandbox choice to gVisor. Half the assets plus gcloud/tmp/parallel_composite_uploads/… component objects are left behind in gs://<bucket>-assets.

Root cause

gcloud storage cp uses parallel composite uploads for large objects. The worker processes it forks each read gcloud's per-invocation context-aware client certificate file (caa_cert.pem), which the parent removes on exit; with the cert gone or half-written the workers' TLS handshakes fail with PEM lib, and the composite upload is abandoned.

Verified workaround

Replaying the identical step with CLOUDSDK_STORAGE_PARALLEL_COMPOSITE_UPLOAD_ENABLED=False in the environment succeeded: all four assets uploaded at ~30 MiB/s and sandboxconfig.ate.dev/microvm created.

Fix

Set CLOUDSDK_STORAGE_PARALLEL_COMPOSITE_UPLOAD_ENABLED=False in the environment the installer passes to the micro-VM steps (microVMEnv in installer/internal/snapshot/snapshot.go). gcloud storage cp has no per-command flag for this; the storage/parallel_composite_upload_enabled property (and its CLOUDSDK_STORAGE_… env form) is the only switch.

This covers both staging paths the step can take:

  • the pinned v1.0.0-rc.1 tree's hack/install-microvm-deps.sh → stage-to-gcs.sh, and
  • newer trees' ate-setup deploy sandboxconfig, whose Go code also shells out to gcloud storage cp (cmd/ate-setup/internal/steps/microvm.go) with an environment derived from os.Environ(), so the variable reaches it too.

#88 brings a local copy of the staging script into tools/setup-gcp/install-microvm-deps.sh with the same gcloud storage cp call; when it lands, its invocation should either inherit this environment or set the property itself.

Evidence

Observed in an end-to-end run; the failing step and the successful replay with the variable set were captured from the same cluster and bucket.

Activity

  1. changed the title [-]Micro-VM asset upload fails on corp machines: gcloud parallel composite uploads break with context-aware client certs[/-] [+]Micro-VM asset upload fails with context-aware client certificates: gcloud parallel composite uploads break[/+] on Nov 3, 2026
  2. added a commit that references this issue on Nov 5, 2026
    1fb05e9
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions