Skip to content

Stage micro-VM assets without gcloud's parallel composite uploads - #112

Merged
annapendleton merged 1 commit into
mainfrom
no-parallel-composite-uploads
Nov 3, 2026
Merged

annapendleton merged 1 commit into
mainfrom
no-parallel-composite-uploads

Conversation

@aditya-shantanu

Copy link
Copy Markdown
Contributor

On a workstation with gcloud's context-aware access enabled, the "stage micro-VM assets" step fails after the asset download: gcloud storage cp splits each large upload into components sent by worker processes, each worker reads the per-invocation client certificate file the parent writes and removes, and the upload dies with [SSL] PEM lib (_ssl.c:4181) ×3 and Temporary components were not uploaded correctly. Half the assets and gcloud/tmp/parallel_composite_uploads/… parts are left in the bucket, the SandboxConfig is never applied, and the wizard's [s] resets the choice to gVisor. Replaying the same step with CLOUDSDK_STORAGE_PARALLEL_COMPOSITE_UPLOAD_ENABLED=False uploaded all four assets and created the SandboxConfig.

This change:

  • microVMEnv sets CLOUDSDK_STORAGE_PARALLEL_COMPOSITE_UPLOAD_ENABLED=False, so both the staging step and the demo counter-microvm step carry it. It covers both staging paths: the pinned v1.0.0-rc.1 tree's hack/install-microvm-deps.sh (stage-to-gcs.sh runs a bare gcloud storage cp) and newer trees' ate-setup deploy sandboxconfig, whose Go code shells out to the same command with an environment built from os.Environ(). gcloud storage cp has no per-command flag for this; the storage/parallel_composite_upload_enabled property is the only switch.
  • TestMicroVMDepsPinArchAndStagingTarget asserts the variable on both specs.
  • The README's hand-run re-staging commands say to set it.

#88's local copy of the staging script makes the same gcloud storage cp call; once it lands, its step should carry this environment too.

Tests: gofmt -l, go vet ./..., go test ./internal/snapshot/... (only the pre-existing, machine-specific TestFetchTreesFetchesBothCommits failure).

Fixes #111

The "stage micro-VM assets" step failed on a workstation with gcloud's
context-aware access enabled (context_aware/use_client_certificate =
true): after the 50 s asset download, `gcloud storage cp` died with
`[SSL] PEM lib (_ssl.c:4181)` three times and `Temporary components were
not uploaded correctly`. gcloud splits each large upload into components
sent by worker processes, and each worker reads the per-invocation client
certificate file the parent gcloud writes and removes. Half the assets and
the gcloud/tmp/parallel_composite_uploads/ parts were left in the asset
bucket, the SandboxConfig was never applied, and the wizard's [s] reset
the sandbox choice to gVisor. Replaying the same step with
CLOUDSDK_STORAGE_PARALLEL_COMPOSITE_UPLOAD_ENABLED=False in the
environment uploaded all four assets and created the SandboxConfig.

microVMEnv now sets that variable, so both the staging step and the demo
counter-microvm step carry it. It reaches both staging paths: the pinned
v1.0.0-rc.1 tree's hack/install-microvm-deps.sh (whose stage-to-gcs.sh
runs a bare `gcloud storage cp`) and, on trees from
agent-substrate/substrate#2092 on, `ate-setup deploy sandboxconfig`,
whose Go code shells out to the same command with an environment built
from os.Environ(). gcloud storage cp has no flag for this; the
storage/parallel_composite_upload_enabled property is the only switch,
and sequential upload of four objects costs nothing noticeable.

The env test for the two micro-VM specs asserts the variable, and the
README's hand-run re-staging commands say to set it too.

Fixes #111
@annapendleton
annapendleton merged commit 1fb05e9 into main Nov 3, 2026
7 checks passed
@annapendleton
annapendleton deleted the no-parallel-composite-uploads branch November 3, 2026 23:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Micro-VM asset upload fails with context-aware client certificates: gcloud parallel composite uploads break

3 participants