Skip to content

e2e: vmdisk-clone downloads its golden image from the public Alpine CDN inside a 10m assert, so a slow CDN fails the suite #5301

Open

Description

The VMDisk clone suite downloads its golden image from the public Alpine CDN inside a 10 minute assert, so a slow CDN fails the suite even when the import itself is healthy. This happened in in-tree E2E run 43310967137, on a change that touches only seaweedfs. The create-golden-image step applies hack/e2e-chainsaw/vminstance/golden-image.yaml, an HTTP import of nocloud_alpine-3.21.7-x86_64-bios-cloudinit-r0.qcow2 (172,032,000 bytes) from dl-cdn.alpinelinux.org. At the deadline the DataVolume was still ImportInProgress at 66.75%, with Ready reason TransferRunning.

* status.(conditions[?type == 'Ready'])[0].status: Invalid value: "False": Expected value: "True"
operation terminated with error: context deadline exceeded

The import did not fail, it was just slow. The importer log shows TransferScratch from 00:39:34Z and a steady climb with no errors or stalls: 3.14% at 00:40:00, 62.71% at 00:48:32, 66.10% at 00:49:01. That is about 114 MB in 567 s, roughly 200 KB/s from the first second. At that rate the whole file needs about 14 minutes.

Other downloads were fast on the same day and in the same run. The same step took 40 to 45 s in six green PR runs (43309715411, 43299732594, 43295829373, 43295775822, 43292143233, 43284911026), so about 4 to 5 MB/s. In the failing run, vmdisk imported the Ubuntu cloud image from cloud-images.ubuntu.com in 77 s (64 to 86 s in the green runs), and the 231 MB cdi-importer image pulled in 7 s. Nothing else was running in the suite at the time. Only the Alpine CDN leg was slow.

Raising the timeout alone only moves the threshold. The tenant Talos image had the same problem and is now served from an in-sandbox mirror. hack/e2e-talos-image-cache.yaml seeds the file once at install, with retries and a stall abort, and then serves it with range support. hack/e2e-chainsaw/_lib/talos-image-cache.sh checks that the mirror is reachable and falls back to the public URL. Seeding the Alpine qcow2 the same way and pointing golden-image.yaml at the mirror takes the download out of the assert window. A slow CDN would then cost install time with retries instead of failing a test. A static check that golden-image.yaml names no external host would fail on the current file and keep it from coming back.

The Ubuntu imports in vmdisk.yaml and vmdisk-vmi.yaml and the cirros URL in chainsaw-test.yaml have the same exposure, but none of them failed this way in the runs above.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    triage/needs-triageIndicates an issue needs triage by a maintainer

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions