Hmm... okay; that didn't work

[2021-06-16T08:38:12.715Z] ERROR: Failed 'ERROR: Failed to docker pull mcr.microsoft.com/windows/servercore/insider:10.0.20344.1.' at 06/16/2021 08:38:12
[2021-06-16T08:38:12.715Z] At D:\gopath\src\github.com\docker\docker\hack\ci\windows.ps1:368 char:17
[2021-06-16T08:38:12.715Z] + ...             Throw $("ERROR: Failed to docker pull $($env:WINDOWS_BASE ...
[2021-06-16T08:38:12.715Z] +                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Ah.. 🤦 I guess the machine itself would have to be updated to 10.0.20295 or above;

no matching manifest for windows/amd64 10.0.20295 in the manifest list entries

Are all of the newer tags also for the upcoming "windows 2022"? Or are they for "after that"?

    "10.0.20295.1",
    "10.0.20295.1-amd64",
    "10.0.20298.1",
    "10.0.20298.1-amd64",
    "10.0.20303.1",
    "10.0.20303.1-amd64",
    "10.0.20308.1",
    "10.0.20308.1-amd64",
    "10.0.20313.1",
    "10.0.20313.1-amd64",
    "10.0.20317.1",
    "10.0.20317.1-amd64",
    "10.0.20324.3",
    "10.0.20324.3-amd64",
    "10.0.20329.1",
    "10.0.20329.1-amd64",
    "10.0.20334.1",
    "10.0.20334.1-amd64",
    "10.0.20339.2",
    "10.0.20339.2-amd64",
    "10.0.20344.1",
    "10.0.20344.1-amd64"

Are all of the newer tags also for the upcoming "windows 2022"? Or are they for "after that"?

Looks like it is; https://techcommunity.microsoft.com/t5/windows-server-insiders/announcing-windows-server-preview-build-20344/m-p/2303472

Today we are pleased to release a new build of the Windows Server Long-Term Servicing Channel (LTSC) Preview that contains both the Desktop Experience and Server Core installation options for Datacenter and Standard editions.

Yeah, those are all Windows 2022 Server Insider builds. Everything over 19041 technically has been for Windows Server 2022 LTSC, but build ID 20201 is the first one actually rigged up as an LTSC build instead of vNext.

Unless we are running Hyper-V isolation in CI, the base image needs to match the host version's build ID, and even for Hyper-V isolation, I recall the build ID of the host needs to be ahead of the container base.

[2021-06-16T08:38:11.308Z] INFO: Branch:fe_release Build:20295.1.210212-1600

So you'll need to bump the host version to before you can bump this tag.

Since it's looking pretty stable, now's probably a good time to get the host bumped to 20344, as I suspect we won't see another insider build, and there's not yet container base images for 20348, the version on the MS Evaluation Center.

Thanks! Yes, I need to check with @StefanScherer to get the machines updated (I found the repository that's used to build / configure them). Once they're updated we can merge this PR.

Following up, there's still no container base images posted for the Windows Server 2022 Preview (Build 20348), and it'll reach General Availability "later this summer"; I assume that means "before September 2021". Since the current build is considered "RTM", build 20348 seems likely to be the final release build ID for Windows Server 2022, that needs to be matched between host and container.

I have opened a Feedback Hub issue for the missing container base images, if people want to upvote it, based on the directions here.

Edit: Sorry, I didn't realise that Feedback Hub link only works if your Microsoft Account (or Organisational/AAD Account) is signed up as a Windows Insider.

Windows Server, Windows Server Core, and Nano Server insider images for 10.0.20348.1 have now been published, so it's feasible (at least from this perspective) to update the CI infrastructure to the Windows Server 2022 Preview (non-Insider) releases based on build ID 20348.

@StefanScherer can you look about if we can get CI servers updated to 10.0.20348.1 ?

It'll probably be 10.0.20348.143 or higher by the time Windows Update is done. That's fine, only the 10.0.20348 needs to match for our purposes.

@olljanat @TBBle Thanks for the heads up. I've tested a newer Jenkins agent with 10.0.20348.112 and it looks good to run the mcr.microsoft.com/windows/servercore/insider:10.0.20348.1 image in process isolation.
I've deployed the new Jenkins agent so you should be able to test it.

Thanks! Updated 🤞 hope it works now

Running into these again. Wondering what's causing it (regression in BuildKit / Dockerfile 1.3 ?)

#32 [swagger 2/2] RUN --mount=type=cache,target=/root/.cache/go-build     --mount=type=cache,target=/go/pkg/mod     --mount=type=tmpfs,target=/go/src/         set -x         && git clone https://github.com/kolyshkin/go-swagger.git .         && git checkout -q "c56166c036004ba7a3a321e5951ba472b9ae298c"         && go build -o /build/swagger github.com/go-swagger/go-swagger/cmd/swagger
#32 sha256:b91ffc0025b99e48e874df39aa649930bfdbb27d79b80cdc6e030f11987f2e15
#32 625.1 error: RPC failed; curl 56 GnuTLS recv error (-54): Error in the pull function.
#32 625.1 fatal: the remote end hung up unexpectedly
#32 625.1 fatal: early EOF
#32 625.1 fatal: index-pack failed
#32 ERROR: executor failed running [/bin/sh -c set -x         && git clone https://github.com/kolyshkin/go-swagger.git .         && git checkout -q "$GO_SWAGGER_COMMIT"         && go build -o /build/swagger github.com/go-swagger/go-swagger/cmd/swagger]: exit code: 128
------
 > [swagger 2/2] RUN --mount=type=cache,target=/root/.cache/go-build     --mount=type=cache,target=/go/pkg/mod     --mount=type=tmpfs,target=/go/src/         set -x         && git clone https://github.com/kolyshkin/go-swagger.git .         && git checkout -q "c56166c036004ba7a3a321e5951ba472b9ae298c"         && go build -o /build/swagger github.com/go-swagger/go-swagger/cmd/swagger:
------
executor failed running [/bin/sh -c set -x         && git clone https://github.com/kolyshkin/go-swagger.git .         && git checkout -q "$GO_SWAGGER_COMMIT"         && go build -o /build/swagger github.com/go-swagger/go-swagger/cmd/swagger]: exit code: 128
script returned exit code 1

Looks to be just a generic networking / connection issue; https://lynxbee.com/solved-error-rpc-failed-curl-56-gnutls-recv-error-54-error-in-the-pull-function/

curl/curl#1707

O_O

Win-RS5:

[2021-08-09T17:49:28.444Z] DONE 1092 tests, 546 skipped in 3068.068s
[2021-08-09T17:49:28.444Z] INFO: Integration tests ended at 08/09/2021 17:49:28. Duration:00:51:08.1898356

Win-2022:

[2021-08-09T17:19:01.727Z] DONE 1092 tests, 546 skipped in 1916.498s
[2021-08-09T17:19:01.727Z] INFO: Integration tests ended at 08/09/2021 17:19:01. Duration:00:31:56.5866128

Are we running larger VMs for WIn-2022? Or is Windows Server 2022 just faster? Looking at a known-long test (TestBuildWCOWSandboxSize) that single test went from 16m to 11m, and other tests nearby showed about the same improvement, which matches the overall improvement.

The machine configuration for RS5 and Windows Server 2022 is the same. Looks like it got faster than 2019.

Gonna bring this one in, because without it, CI will be failing (looks like Microsoft added a check, and these images now require the host to be on an insider build to be able to run them?) /cc @kevpar

[2021-08-09T18:06:13.240Z] 29d5b6a226d2: Download complete
[2021-08-09T18:07:54.461Z] powershell.exe : failed to register layer: re-exec error: exit status 1: output: hcsshim::ProcessUtilityVMImage 
[2021-08-09T18:07:54.461Z] \\?\D:\docker\windowsfilter\1258f08accef9b7f046ebfe52bddc3814446163e842dca420fe8242533da0e06\UtilityVM: To use this container image, you must join the Windows Insider Program. Please see 
[2021-08-09T18:07:54.461Z] https://go.microsoft.com/fwlink/?linkid=850659 for more information.

The machine configuration for RS5 and Windows Server 2022 is the same. Looks like it got faster than 2019.

I recall we saw an improvement in speed between RS1 and RS5 as well. Not sure though if the speed "improved" or if the older Windows versions became slower over time.

FYI, it looks to be that RTM version is out https://docs.microsoft.com/en-us/windows-server/get-started/windows-server-release-info and mcr.microsoft.com/windows/servercore:ltsc2022 image tag too https://hub.docker.com/_/microsoft-windows-servercore

Thanks @olljanat for the heads-up!! I also found a VM image in Azure and preparing a Jenkins agent with it. I think we can switch to the new Windows Server 2022 Server Core LTSC very soon.

I opened #42765 to update the Jenksinfile