Flaky test: TestBuildWCOWSandboxSize (insufficient disk space?) #42743
Description
This test is failing frequently with a There is not enough space on the disk. failure;
=== RUN TestBuildWCOWSandboxSize
build_test.go:572: assertion failed: value "{\"stream\":\"Step 1/8 : FROM busybox AS intermediate\"}\r\n{\"stream\":\"\\n\"}\r\n{\"stream\":\" ---\\u003e ea8f1b6e5c44\\n\"}\r\n{\"stream\":\"Step 2/8 : WORKDIR C:\\\\\\\\stuff\"}\r\n{\"stream\":\"\\n\"}\r\n{\"stream\":\" ---\\u003e Running in a2d453ddd861\\n\"}\r\n{\"stream\":\"Removing intermediate container a2d453ddd861\\n\"}\r\n{\"stream\":\" ---\\u003e 395a1b07f269\\n\"}\r\n{\"stream\":\"Step 3/8 : RUN fsutil file createnew C:\\\\\\\\stuff\\\\\\\\bigfile_0.txt 22548578304 \\u0026\\u0026 del bigfile_0.txt\"}\r\n{\"stream\":\"\\n\"}\r\n{\"stream\":\" ---\\u003e Running in 1f6957162b60\\n\"}\r\n{\"stream\":\"File C:\\\\stuff\\\\bigfile_0.txt is created\\r\\n\"}\r\n{\"stream\":\"Removing intermediate container 1f6957162b60\\n\"}\r\n{\"stream\":\" ---\\u003e 0a7a087b0669\\n\"}\r\n{\"stream\":\"Step 4/8 : RUN fsutil file createnew C:\\\\\\\\stuff\\\\\\\\bigfile_1.txt 7516192768\"}\r\n{\"stream\":\"\\n\"}\r\n{\"stream\":\" ---\\u003e Running in 478b5492a998\\n\"}\r\n{\"stream\":\"File C:\\\\stuff\\\\bigfile_1.txt is created\\r\\n\"}\r\n{\"stream\":\"Removing intermediate container 478b5492a998\\n\"}\r\n{\"stream\":\" ---\\u003e d26ac7985ca8\\n\"}\r\n{\"stream\":\"Step 5/8 : RUN fsutil file createnew C:\\\\\\\\stuff\\\\\\\\bigfile_2.txt 7516192768\"}\r\n{\"stream\":\"\\n\"}\r\n{\"stream\":\" ---\\u003e Running in a776a76797c8\\n\"}\r\n{\"stream\":\"File C:\\\\stuff\\\\bigfile_2.txt is created\\r\\n\"}\r\n{\"stream\":\"Removing intermediate container a776a76797c8\\n\"}\r\n{\"stream\":\" ---\\u003e 0a0dd247d26f\\n\"}\r\n{\"stream\":\"Step 6/8 : RUN fsutil file createnew C:\\\\\\\\stuff\\\\\\\\bigfile_3.txt 7516192768\"}\r\n{\"stream\":\"\\n\"}\r\n{\"stream\":\" ---\\u003e Running in 8dcb313b2e5c\\n\"}\r\n{\"stream\":\"File C:\\\\stuff\\\\bigfile_3.txt is created\\r\\n\"}\r\n{\"stream\":\"Removing intermediate container 8dcb313b2e5c\\n\"}\r\n{\"stream\":\" ---\\u003e 9111caa5c35e\\n\"}\r\n{\"aux\":{\"ID\":\"sha256:9111caa5c35ea45c8a2bcc410a5e2a62532ce48fe877fd065c1e457c2fafd7f6\"}}\r\n{\"stream\":\"Step 7/8 : FROM busybox\"}\r\n{\"stream\":\"\\n\"}\r\n{\"stream\":\" ---\\u003e ea8f1b6e5c44\\n\"}\r\n{\"stream\":\"Step 8/8 : COPY --from=intermediate C:\\\\\\\\stuff C:\\\\\\\\stuff\"}\r\n{\"stream\":\"\\n\"}\r\n{\"errorDetail\":{\"message\":\"re-exec error: exit status 1: output: hcsshim::ImportLayer - failed failed in Win32: The system cannot find the path specified. (0x3)\"},\"error\":\"re-exec error: exit status 1: output: hcsshim::ImportLayer - failed failed in Win32: The system cannot find the path specified. (0x3)\"}\r\n" does not match regexp "Successfully built|re-exec error: exit status 1: output: write.*daemon\\\\\\\\tmp\\\\\\\\hcs.*bigfile_[1-3].txt: There is not enough space on the disk."
--- FAIL: TestBuildWCOWSandboxSize (547.71s)
Cleaned-up output:
{"stream":"Step 1/8 : FROM busybox AS intermediate"}
{"stream":"\\n"}
{"stream":" ---\\u003e ea8f1b6e5c44\\n"}
{"stream":"Step 2/8 : WORKDIR C:\\\\\\\\stuff"}
{"stream":"\\n"}
{"stream":" ---\\u003e Running in a2d453ddd861\\n"}
{"stream":"Removing intermediate container a2d453ddd861\\n"}
{"stream":" ---\\u003e 395a1b07f269\\n"}
{"stream":"Step 3/8 : RUN fsutil file createnew C:\\\\\\\\stuff\\\\\\\\bigfile_0.txt 22548578304 \\u0026\\u0026 del bigfile_0.txt"}
{"stream":"\\n"}
{"stream":" ---\\u003e Running in 1f6957162b60\\n"}
{"stream":"File C:\\\\stuff\\\\bigfile_0.txt is created\\r\\n"}
{"stream":"Removing intermediate container 1f6957162b60\\n"}
{"stream":" ---\\u003e 0a7a087b0669\\n"}
{"stream":"Step 4/8 : RUN fsutil file createnew C:\\\\\\\\stuff\\\\\\\\bigfile_1.txt 7516192768"}
{"stream":"\\n"}
{"stream":" ---\\u003e Running in 478b5492a998\\n"}
{"stream":"File C:\\\\stuff\\\\bigfile_1.txt is created\\r\\n"}
{"stream":"Removing intermediate container 478b5492a998\\n"}
{"stream":" ---\\u003e d26ac7985ca8\\n"}
{"stream":"Step 5/8 : RUN fsutil file createnew C:\\\\\\\\stuff\\\\\\\\bigfile_2.txt 7516192768"}
{"stream":"\\n"}
{"stream":" ---\\u003e Running in a776a76797c8\\n"}
{"stream":"File C:\\\\stuff\\\\bigfile_2.txt is created\\r\\n"}
{"stream":"Removing intermediate container a776a76797c8\\n"}
{"stream":" ---\\u003e 0a0dd247d26f\\n"}
{"stream":"Step 6/8 : RUN fsutil file createnew C:\\\\\\\\stuff\\\\\\\\bigfile_3.txt 7516192768"}
{"stream":"\\n"}
{"stream":" ---\\u003e Running in 8dcb313b2e5c\\n"}
{"stream":"File C:\\\\stuff\\\\bigfile_3.txt is created\\r\\n"}
{"stream":"Removing intermediate container 8dcb313b2e5c\\n"}
{"stream":" ---\\u003e 9111caa5c35e\\n"}
{"aux":{"ID":"sha256:9111caa5c35ea45c8a2bcc410a5e2a62532ce48fe877fd065c1e457c2fafd7f6"}}
{"stream":"Step 7/8 : FROM busybox"}
{"stream":"\\n"}
{"stream":" ---\\u003e ea8f1b6e5c44\\n"}
{"stream":"Step 8/8 : COPY --from=intermediate C:\\\\\\\\stuff C:\\\\\\\\stuff"}
{"stream":"\\n"}
{"errorDetail":{"message":"re-exec error: exit status 1: output: hcsshim::ImportLayer - failed failed in Win32: The system cannot find the path specified. (0x3)"},"error":"re-exec error: exit status 1: output: hcsshim::ImportLayer - failed failed in Win32: The system cannot find the path specified. (0x3)"}
We should
- check if the error is genuine (is there indeed insufficient disk space, or is it an error that's converted incorrectly?
- if disk space is insufficient for the test, we should check space before running the test, and skip if if there's not enough space.
It's possible that the Jenkins agents run out of space over time, but they are purged/rotated frequently, which should still give us coverage for that specific test (I think?)