appveyor: make CI happy by fuweid · Pull Request #3228 · containerd/containerd

fuweid · 2019-04-18T08:23:06Z

microsoft/nanoserver:latest is not found. We need to use existing tag
which doen't change frequently.

Fix: #3227

Signed-off-by: Wei Fu [email protected]

fuweid · 2019-04-18T10:43:07Z

CI is hang at pulling seed image

I set 30s timeout and dump goroutine stack after pulling seed image.

Because the microsoft/nanoserver:latest is not found, the ctrd will stop and wait.

But the wait is hang....

[running]:
github.com/containerd/containerd.TestMain(0xc000121e80)
\tC:/gopath/src/github.com/containerd/containerd/client_test.go:164 +0x1ab3
main.main()
\t_testmain.go:128 +0x22a

goroutine 5 [syscall, locked to thread]:
syscall.Syscall6(0x7ffe319e46b0, 0x5, 0x1d4, 0xc0003b0000, 0x8000, 0xc0003add24, 0x0, 0x0, 0x0, 0x0, ...)
\tC:/go/src/runtime/syscall_windows.go:199 +0xed
syscall.ReadFile(0x1d4, 0xc0003b0000, 0x8000, 0x8000, 0xc0003add24, 0x0, 0xc0003add40, 0x4309e6)
\tC:/go/src/syscall/zsyscall_windows.go:312 +0x102
syscall.Read(0x1d4, 0xc0003b0000, 0x8000, 0x8000, 0x54, 0x447a70, 0x447a70)
\tC:/go/src/syscall/syscall_windows.go:296 +0x83
internal/poll.(*FD).Read(0xc000374780, 0xc0003b0000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
\tC:/go/src/internal/poll/fd_windows.go:494 +0x2df
os.(*File).read(...)
\tC:/go/src/os/file_windows.go:207
os.(*File).Read(0xc000006400, 0xc0003b0000, 0x8000, 0x8000, 0x54, 0x0, 0x0)
\tC:/go/src/os/file.go:108 +0xaf
io.copyBuffer(0x13078e0, 0xc0002f29e0, 0x1307d60, 0xc000006400, 0xc0003b0000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
\tC:/go/src/io/io.go:402 +0x14b
io.Copy(...)
\tC:/go/src/io/io.go:364
os/exec.(*Cmd).writerDescriptor.func1(0x4612f1, 0x0)
\tC:/go/src/os/exec/exec.go:288 +0x82
os/exec.(*Cmd).Start.func1(0xc0000d54a0, 0xc0002f2ae0)
\tC:/go/src/os/exec/exec.go:409 +0x3c
created by os/exec.(*Cmd).Start
\tC:/go/src/os/exec/exec.go:408 +0xaa5

goroutine 7 [select]:
github.com/containerd/containerd/vendor/google.golang.org/grpc.(*ccResolverWrapper).watcher(0xc000370c30)
\tC:/gopath/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/resolver_conn_wrapper.go:109 +0x228
created by github.com/containerd/containerd/vendor/google.golang.org/grpc.(*ccResolverWrapper).start
\tC:/gopath/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/resolver_conn_wrapper.go:95 +0x54

goroutine 8 [select]:
github.com/containerd/containerd/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc000289080)
\tC:/gopath/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:122 +0x205
created by github.com/containerd/containerd/vendor/google.golang.org/grpc.newCCBalancerWrapper
\tC:/gopath/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:113 +0x294

goroutine 9 [select]:
github.com/containerd/containerd/vendor/google.golang.org/grpc.(*addrConn).transportMonitor(0xc0001562c0)
\tC:/gopath/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/clientconn.go:1373 +0x2ab
github.com/containerd/containerd/vendor/google.golang.org/grpc.(*addrConn).connect.func1(0xc0001562c0)
\tC:/gopath/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/clientconn.go:949 +0x27c
created by github.com/containerd/containerd/vendor/google.golang.org/grpc.(*addrConn).connect
\tC:/gopath/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/clientconn.go:940 +0x198

goroutine 12 [select]:
github.com/containerd/containerd/vendor/github.com/Microsoft/go-winio.(*win32File).asyncIo(0xc00021b720, 0xc000159d70, 0xc00021b750, 0x0, 0x1308e40, 0xfc14f0, 0x1308e40, 0xfc14f0, 0x0)
\tC:/gopath/src/github.com/containerd/containerd/vendor/github.com/Microsoft/go-winio/file.go:186 +0x19b
github.com/containerd/containerd/vendor/github.com/Microsoft/go-winio.(*win32File).Read(0xc00021b720, 0xc000408000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
\tC:/gopath/src/github.com/containerd/containerd/vendor/github.com/Microsoft/go-winio/file.go:224 +0x18a
bufio.(*Reader).Read(0xc00022b620, 0xc0002f01f8, 0x9, 0x9, 0x800010601, 0x1060100000000, 0x462ca6)
\tC:/go/src/bufio/bufio.go:223 +0x7c3
io.ReadAtLeast(0x1306e00, 0xc00022b620, 0xc0002f01f8, 0x9, 0x9, 0x9, 0xc000296ab8, 0xc0003abd30, 0x441212)
\tC:/go/src/io/io.go:310 +0x9d
io.ReadFull(...)
\tC:/go/src/io/io.go:329
github.com/containerd/containerd/vendor/golang.org/x/net/http2.readFrameHeader(0xc0002f01f8, 0x9, 0x9, 0x1306e00, 0xc00022b620, 0x0, 0x0, 0x40a4d75c5b735d71, 0x4010000000000000)
\tC:/gopath/src/github.com/containerd/containerd/vendor/golang.org/x/net/http2/frame.go:237 +0xb4
github.com/containerd/containerd/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc0002f01c0, 0xc00022f280, 0xc00022f200, 0x0, 0x0)
\tC:/gopath/src/github.com/containerd/containerd/vendor/golang.org/x/net/http2/frame.go:492 +0x103
github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.(*http2Client).reader(0xc0000b9680)
\tC:/gopath/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/transport/http2_client.go:1123 +0x205
created by github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.newHTTP2Client
\tC:/gopath/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/transport/http2_client.go:265 +0x1215

goroutine 11 [syscall, locked to thread]:
syscall.Syscall6(0x7ffe319d6c00, 0x5, 0x204, 0xc000405f84, 0xc000405f88, 0xc000405f98, 0xffffffff, 0x0, 0x0, 0x0, ...)
\tC:/go/src/runtime/syscall_windows.go:199 +0xed
github.com/containerd/containerd/vendor/github.com/Microsoft/go-winio.getQueuedCompletionStatus(0x204, 0xc000405f84, 0xc000405f88, 0xc000405f98, 0xffffffff, 0x0, 0x0)
\tC:/gopath/src/github.com/containerd/containerd/vendor/github.com/Microsoft/go-winio/zsyscall_windows.go:104 +0xe0
github.com/containerd/containerd/vendor/github.com/Microsoft/go-winio.ioCompletionProcessor(0x204)
\tC:/gopath/src/github.com/containerd/containerd/vendor/github.com/Microsoft/go-winio/file.go:159 +0xf1
created by github.com/containerd/containerd/vendor/github.com/Microsoft/go-winio.initIo
\tC:/gopath/src/github.com/containerd/containerd/vendor/github.com/Microsoft/go-winio/file.go:72 +0xa5

goroutine 13 [select]:
github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.(*controlBuffer).get(0xc0002891c0, 0x1, 0x0, 0x0, 0x0, 0x0)
\tC:/gopath/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/transport/controlbuf.go:289 +0x203
github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.(*loopyWriter).run(0xc00022b7a0)
\tC:/gopath/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/transport/controlbuf.go:374 +0x26b
github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.newHTTP2Client.func3(0xc0000b9680)
\tC:/gopath/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/transport/http2_client.go:298 +0x11d
created by github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.newHTTP2Client
\tC:/gopath/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/transport/http2_client.go:296 +0x162a

goroutine 14 [syscall, locked to thread]:
syscall.Syscall(0x7ffe319e42c0, 0x2, 0x1f0, 0xffffffff, 0x0, 0x0, 0x0, 0x0)
\tC:/go/src/runtime/syscall_windows.go:186 +0xea
syscall.WaitForSingleObject(0x1f0, 0xffffffff, 0x0, 0x0, 0x0)
\tC:/go/src/syscall/zsyscall_windows.go:737 +0x89
os.(*Process).wait(0xc00037cd80, 0x0, 0x0, 0x0)
\tC:/go/src/os/exec_windows.go:18 +0x90
os.(*Process).Wait(...)
\tC:/go/src/os/exec.go:125
os/exec.(*Cmd).Wait(0xc0000d54a0, 0x11f7140, 0x19e4f00)
\tC:/go/src/os/exec/exec.go:474 +0xfb
github.com/containerd/containerd.(*daemon).Wait(0x19e4f00, 0x0, 0x0)
\tC:/gopath/src/github.com/containerd/containerd/daemon_test.go:100 +0xbf
github.com/containerd/containerd.TestMain.func2(0xc0003e8ba0, 0x1322ae0, 0xc0003704b0, 0xc0000a6dc0, 0xc0002e8370, 0xc000370420)
\tC:/gopath/src/github.com/containerd/containerd/client_test.go:146 +0x2dd
created by github.com/containerd/containerd.TestMain
\tC:/gopath/src/github.com/containerd/containerd/client_test.go:141 +0x7a2

goroutine 24 [IO wait]:
internal/poll.runtime_pollWait(0x3d312c8, 0x72, 0x1308e40)
\tC:/go/src/runtime/netpoll.go:182 +0x5d
internal/poll.(*pollDesc).wait(0xc000156a08, 0x72, 0xfc1400, 0x0, 0x0)
\tC:/go/src/internal/poll/fd_poll_runtime.go:87 +0xec
internal/poll.(*ioSrv).ExecIO(0x19e1fa8, 0xc000156858, 0x11f65b0, 0x165b, 0x1, 0xc000423230)
\tC:/go/src/internal/poll/fd_windows.go:228 +0x2f5
internal/poll.(*FD).Read(0xc000156840, 0xc000472000, 0x1746, 0x1746, 0x0, 0x0, 0x0)
\tC:/go/src/internal/poll/fd_windows.go:502 +0x38a
net.(*netFD).Read(0xc000156840, 0xc000472000, 0x1746, 0x1746, 0xc00012bf20, 0xc, 0xc)
\tC:/go/src/net/fd_windows.go:152 +0x6d
net.(*conn).Read(0xc000434048, 0xc000472000, 0x1746, 0x1746, 0x55d074, 0xc0003d8cd8, 0x2030000)
\tC:/go/src/net/net.go:177 +0xa9
crypto/tls.(*atLeastReader).Read(0xc00038d760, 0xc000472000, 0x1746, 0x1746, 0xc00038d760, 0x1307060, 0x0)
\tC:/go/src/crypto/tls/conn.go:761 +0xae
bytes.(*Buffer).ReadFrom(0xc0003d8cd8, 0x1306f60, 0xc00038d760, 0x1198480, 0x3f700b8, 0xc000472005)
\tC:/go/src/bytes/buffer.go:207 +0x16c
crypto/tls.(*Conn).readFromUntil(0xc0003d8a80, 0x3f700b8, 0xc000434048, 0x5, 0xc000434048, 0xf24)
\tC:/go/src/crypto/tls/conn.go:783 +0x229
crypto/tls.(*Conn).readRecordOrCCS(0xc0003d8a80, 0x11f7100, 0xc0003d8bb8, 0x8cc004)
\tC:/go/src/crypto/tls/conn.go:590 +0x2eb
crypto/tls.(*Conn).readRecord(...)
\tC:/go/src/crypto/tls/conn.go:558
crypto/tls.(*Conn).Read(0xc0003d8a80, 0xc000154000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
\tC:/go/src/crypto/tls/conn.go:1236 +0x1fd
net/http.(*persistConn).Read(0xc0004365a0, 0xc000154000, 0x1000, 0x1000, 0x0, 0xf22, 0x1)
\tC:/go/src/net/http/transport.go:1524 +0x24f
bufio.(*Reader).fill(0xc000452900)
\tC:/go/src/bufio/bufio.go:100 +0x1a1
bufio.(*Reader).Peek(0xc000452900, 0x1, 0x2, 0x0, 0x0, 0xc00042ac00, 0x0)
\tC:/go/src/bufio/bufio.go:138 +0x9c
net/http.(*persistConn).readLoop(0xc0004365a0)
\tC:/go/src/net/http/transport.go:1677 +0x283
created by net/http.(*Transport).dialConn
\tC:/go/src/net/http/transport.go:1357 +0xc3a

goroutine 25 [select]:
net/http.(*persistConn).writeLoop(0xc0004365a0)
\tC:/go/src/net/http/transport.go:1958 +0x1ba
created by net/http.(*Transport).dialConn
\tC:/go/src/net/http/transport.go:1358 +0xc5f

goroutine 37 [IO wait]:
internal/poll.runtime_pollWait(0x3d311f8, 0x72, 0x1308e40)
\tC:/go/src/runtime/netpoll.go:182 +0x5d
internal/poll.(*pollDesc).wait(0xc000156cc8, 0x72, 0xfc1400, 0x0, 0x0)
\tC:/go/src/internal/poll/fd_poll_runtime.go:87 +0xec
internal/poll.(*ioSrv).ExecIO(0x19e1fa8, 0xc000156b18, 0x11f65b0, 0x61e, 0x1, 0xc000089230)
\tC:/go/src/internal/poll/fd_windows.go:228 +0x2f5
internal/poll.(*FD).Read(0xc000156b00, 0xc000473800, 0x1746, 0x1746, 0x0, 0x0, 0x0)
\tC:/go/src/internal/poll/fd_windows.go:502 +0x38a
net.(*netFD).Read(0xc000156b00, 0xc000473800, 0x1746, 0x1746, 0xc00038d680, 0xc, 0xc)
\tC:/go/src/net/fd_windows.go:152 +0x6d
net.(*conn).Read(0xc000006050, 0xc000473800, 0x1746, 0x1746, 0x55d074, 0xc000063758, 0x2030000)
\tC:/go/src/net/net.go:177 +0xa9
crypto/tls.(*atLeastReader).Read(0xc00038d720, 0xc000473800, 0x1746, 0x1746, 0xc00038d720, 0x1307060, 0x0)
\tC:/go/src/crypto/tls/conn.go:761 +0xae
bytes.(*Buffer).ReadFrom(0xc000063758, 0x1306f60, 0xc00038d720, 0x1198480, 0x3f700b8, 0xc000473805)
\tC:/go/src/bytes/buffer.go:207 +0x16c
crypto/tls.(*Conn).readFromUntil(0xc000063500, 0x3f700b8, 0xc000006050, 0x5, 0xc000006050, 0x40d1fa)
\tC:/go/src/crypto/tls/conn.go:783 +0x229
crypto/tls.(*Conn).readRecordOrCCS(0xc000063500, 0x11f7100, 0xc000063638, 0x8cc004)
\tC:/go/src/crypto/tls/conn.go:590 +0x2eb
crypto/tls.(*Conn).readRecord(...)
\tC:/go/src/crypto/tls/conn.go:558
crypto/tls.(*Conn).Read(0xc000063500, 0xc0002e4000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
\tC:/go/src/crypto/tls/conn.go:1236 +0x1fd
net/http.(*persistConn).Read(0xc000436240, 0xc0002e4000, 0x1000, 0x1000, 0x0, 0xee5, 0x1)
\tC:/go/src/net/http/transport.go:1524 +0x24f
bufio.(*Reader).fill(0xc00009d080)
\tC:/go/src/bufio/bufio.go:100 +0x1a1
bufio.(*Reader).Peek(0xc00009d080, 0x1, 0x0, 0x0, 0x1, 0xc00042aa00, 0x0)
\tC:/go/src/bufio/bufio.go:138 +0x9c
net/http.(*persistConn).readLoop(0xc000436240)
\tC:/go/src/net/http/transport.go:1677 +0x283
created by net/http.(*Transport).dialConn
\tC:/go/src/net/http/transport.go:1357 +0xc3a

goroutine 38 [select]:
net/http.(*persistConn).writeLoop(0xc000436240)
\tC:/go/src/net/http/transport.go:1958 +0x1ba
created by net/http.(*Transport).dialConn
\tC:/go/src/net/http/transport.go:1358 +0xc5f

job url is here

cc @estesp

fuweid · 2019-04-18T12:00:19Z

ping @jterry75

root@ubuntu-xenial ~/g/s/g/p/failpoint# docker pull microsoft/nanoserver                                                                                                        1 master
Using default tag: latest
Error response from daemon: manifest for microsoft/nanoserver:latest not found

any idea about the seed image name here?

thaJeztah · 2019-04-18T12:29:44Z

Ah, yes, microsoft/nanoserver:latest no longer exists, and must now match the version of Windows that the machine is running on;

We are deprecating the 'latest' tag across all our Windows base images to encourage
better container practices. At the beginning of the 2019 calendar year, we will
no longer publish the tag; We'll yank it from the available tags list.

(see https://techcommunity.microsoft.com/t5/Containers/Windows-Server-2019-Now-Available/ba-p/382430)

So if CI is using Windows Server 2019, that image would need to be something like

mcr.microsoft.com/windows/nanoserver:1809

If it's running Windows Server 2016, then it's gonna be more tricky because nano server for Windows Server 2016 no longer exists.

`microsoft/nanoserver:latest` is not found. We need to use existing tag which doen't change frequently. Fix: containerd#3227 Signed-off-by: Wei Fu <[email protected]>

fuweid · 2019-04-18T13:20:00Z

time="2019-04-18T13:16:24Z" level=info msg="apply failure, attempting cleanup" error="failed to extract layer sha256:67903cf26ef4095868687002e3dc6f78ad275677704bf0d11524f16209cec48e: ProcessUtilityVMImage \\\\?\\C:\\Program Files\\containerd\\root-test\\io.containerd.snapshotter.v1.windows\\snapshots\\1\\UtilityVM: The system cannot find the path specified.: unknown" key="extract-276784500-ByGC sha256:67903cf26ef4095868687002e3dc6f78ad275677704bf0d11524f16209cec48e"
failed to unpack image on snapshotter windows: failed to extract layer sha256:67903cf26ef4095868687002e3dc6f78ad275677704bf0d11524f16209cec48e: ProcessUtilityVMImage \\?\C:\Program Files\containerd\root-test\io.containerd.snapshotter.v1.windows\snapshots\1\UtilityVM: The system cannot find the path specified.: unknown: time="2019-04-18T13:16:09.026283100Z" level=info msg="starting containerd"

no idea about this error. ping @jterry75 @jhowardmsft

fuweid · 2019-04-18T13:41:22Z

 	if _, err = client.Pull(ctx, testImage, WithPullUnpack); err != nil {
-		ctrd.Stop()
-		ctrd.Wait()
 		fmt.Fprintf(os.Stderr, "%s: %s\n", err, buf.String())


show the error first and then kill it.

I'm going to pull this fix into #3222 because if this ever happens again, it causes quite a backup as it requires the full 60 minutes to expire before AppVeyor kills the run.

estesp · 2019-04-18T14:26:38Z

#3222 now includes the fix for not hanging CI on pull error and includes an image reference that appears to work. Thanks for your help @fuweid! Closing this PR now.

fuweid force-pushed the donotmerge branch 3 times, most recently from 42abb98 to 71b0555 Compare April 18, 2019 09:47

fuweid changed the title ~~don't merge it~~ don't merge it - investigating AppVeyor CI issue Apr 18, 2019

fuweid force-pushed the donotmerge branch 3 times, most recently from 9fd160c to a715c7d Compare April 18, 2019 12:23

fuweid changed the title ~~don't merge it - investigating AppVeyor CI issue~~ appveyor: make CI happy Apr 18, 2019

fuweid force-pushed the donotmerge branch from a715c7d to 8c0774d Compare April 18, 2019 12:40

appveyor: make CI happy

32a0a78

`microsoft/nanoserver:latest` is not found. We need to use existing tag which doen't change frequently. Fix: containerd#3227 Signed-off-by: Wei Fu <[email protected]>

fuweid force-pushed the donotmerge branch from 8c0774d to 32a0a78 Compare April 18, 2019 13:08

fuweid commented Apr 18, 2019

View reviewed changes

estesp mentioned this pull request Apr 18, 2019

Update to sha-specific nanoserver image reference to fix AppVeyor CI runs #3222

Merged

estesp closed this Apr 18, 2019

fuweid deleted the donotmerge branch August 22, 2019 08:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

appveyor: make CI happy #3228

appveyor: make CI happy #3228
fuweid wants to merge 1 commit intocontainerd:masterfrom
fuweid:donotmerge

fuweid commented Apr 18, 2019 •

edited

Loading

Uh oh!

fuweid commented Apr 18, 2019 •

edited

Loading

Uh oh!

fuweid commented Apr 18, 2019

Uh oh!

thaJeztah commented Apr 18, 2019

Uh oh!

fuweid commented Apr 18, 2019

Uh oh!

fuweid Apr 18, 2019

Uh oh!

estesp Apr 18, 2019

Uh oh!

estesp commented Apr 18, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

fuweid commented Apr 18, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fuweid commented Apr 18, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fuweid commented Apr 18, 2019

Uh oh!

thaJeztah commented Apr 18, 2019

Uh oh!

fuweid commented Apr 18, 2019

Uh oh!

fuweid Apr 18, 2019

Choose a reason for hiding this comment

Uh oh!

estesp Apr 18, 2019

Choose a reason for hiding this comment

Uh oh!

estesp commented Apr 18, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

fuweid commented Apr 18, 2019 •

edited

Loading

fuweid commented Apr 18, 2019 •

edited

Loading