Skip to content

Comments

Extend go test CI from 10 to 15 miniutes#1418

Merged
lmb merged 1 commit intomainfrom
pr/ci-increase-timeout
Apr 8, 2024
Merged

Extend go test CI from 10 to 15 miniutes#1418
lmb merged 1 commit intomainfrom
pr/ci-increase-timeout

Conversation

@dylandreimerink
Copy link
Member

The default builtin timeout in the go test runner is 10 min and the GH runner is also limited to 10 min. In practice the GH runner is quicker to kill tests that exceed this time limit, in which case we abruptly exit without logs on which test caused the timeout.

This change extends the runner timeout to 15 min to give the go runner a chance to kill the bad test and to tell us which test was taking longer so that we can fix the issue.

The default builtin timeout in the go test runner is 10 min and the GH runner is also limited to 10 min. In practice the GH runner is quicker to kill tests that exceed this time limit, in which case we abruptly exit without logs on which test caused the timeout.

This change extends the runner timeout to 15 min to give the go runner a chance to kill the bad test and to tell us which test was taking longer so that we can fix the issue.

Signed-off-by: Dylan Reimerink <[email protected]>
@dylandreimerink dylandreimerink requested a review from a team as a code owner April 5, 2024 12:33
@dylandreimerink dylandreimerink requested a review from lmb April 5, 2024 12:33
@dylandreimerink
Copy link
Member Author

Speak of the devil. This was meant for another PR but this CI run also turned up something:

=== Failed
=== FAIL: perf TestPerfReaderWakeupEvents (unknown)
panic: test timed out after 10m0s
running tests:
	TestPerfReaderWakeupEvents (10m0s)

goroutine 29 [running]:
testing.(*M).startAlarm.func1()
	/opt/hostedtoolcache/go/1.21.8/x64/src/testing/testing.go:2259 +0x3b9
created by time.goFunc
	/opt/hostedtoolcache/go/1.21.8/x64/src/time/sleep.go:176 +0x2d

goroutine 1 [chan receive]:
testing.(*T).Run(0xc0000a2680, {0x71cfe6?, 0x51bbfc?}, 0x72d1a0)
	/opt/hostedtoolcache/go/1.21.8/x64/src/testing/testing.go:1649 +0x3c8
testing.runTests.func1(0x94c4c0?)
	/opt/hostedtoolcache/go/1.21.8/x64/src/testing/testing.go:2054 +0x3e
testing.tRunner(0xc0000a2680, 0xc0000f9bd8)
	/opt/hostedtoolcache/go/1.21.8/x64/src/testing/testing.go:1595 +0xff
testing.runTests(0xc000090320?, {0x945f40, 0xd, 0xd}, {0xc0000f9c90?, 0x4105c9?, 0x94bbe0?})
	/opt/hostedtoolcache/go/1.21.8/x64/src/testing/testing.go:2052 +0x445
testing.(*M).Run(0xc000090320)
	/opt/hostedtoolcache/go/1.21.8/x64/src/testing/testing.go:1925 +0x636
github.com/cilium/ebpf/internal/testutils/fdtrace.TestMain(0xc0000366f0?)
	/home/runner/work/ebpf/ebpf/internal/testutils/fdtrace/fd.go:25 +0x70
github.com/cilium/ebpf/perf.TestMain(...)
	/home/runner/work/ebpf/ebpf/perf/reader_test.go:29
main.main()
	_testmain.go:77 +0x1c7

goroutine 27 [chan receive]:
testing.(*T).Parallel(0xc00026a000)
	/opt/hostedtoolcache/go/1.21.8/x64/src/testing/testing.go:1403 +0x205
github.com/cilium/ebpf/perf.TestPause(0xc00026a000)
	/home/runner/work/ebpf/ebpf/perf/reader_test.go:408 +0x39
testing.tRunner(0xc00026a000, 0x72d168)
	/opt/hostedtoolcache/go/1.21.8/x64/src/testing/testing.go:1595 +0xff
created by testing.(*T).Run in goroutine 1
	/opt/hostedtoolcache/go/1.21.8/x64/src/testing/testing.go:1648 +0x3ad

goroutine 28 [syscall]:
syscall.Syscall6(0x1?, 0x77?, 0x1?, 0x97a120?, 0x77?, 0x0?, 0xc0000fba48?)
	/opt/hostedtoolcache/go/1.21.8/x64/src/syscall/syscall_linux.go:91 +0x30
golang.org/x/sys/unix.EpollWait(0x0?, {0xc0000d2150?, 0x446411?, 0xc00009a4e0?}, 0xc0000fba18?)
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/unix/zsyscall_linux_amd64.go:55 +0x4f
github.com/cilium/ebpf/internal/unix.EpollWait(...)
	/home/runner/work/ebpf/ebpf/internal/unix/types_linux.go:129
github.com/cilium/ebpf/internal/epoll.(*Poller).Wait(0xc0000cc5a0, {0xc0000d2150?, 0x2, 0x2}, {0xc0000fbab0?, 0x1?, 0x0?})
	/home/runner/work/ebpf/ebpf/internal/epoll/poller.go:145 +0x2a5
github.com/cilium/ebpf/perf.(*Reader).ReadInto(0xc0000d46c0, 0x1?)
	/home/runner/work/ebpf/ebpf/perf/reader.go:362 +0x2c5
github.com/cilium/ebpf/perf.(*Reader).Read(...)
	/home/runner/work/ebpf/ebpf/perf/reader.go:336
github.com/cilium/ebpf/perf.checkRecord({0x785618, 0xc00026a1a0}, 0xf?)
	/home/runner/work/ebpf/ebpf/perf/reader_test.go:167 +0x6e
github.com/cilium/ebpf/perf.TestPerfReaderWakeupEvents(0xc00026a1a0)
	/home/runner/work/ebpf/ebpf/perf/reader_test.go:528 +0x527
testing.tRunner(0xc00026a1a0, 0x72d1a0)
	/opt/hostedtoolcache/go/1.21.8/x64/src/testing/testing.go:1595 +0xff
created by testing.(*T).Run in goroutine 1
	/opt/hostedtoolcache/go/1.21.8/x64/src/testing/testing.go:1648 +0x3ad

Copy link
Contributor

@lmb lmb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@lmb lmb merged commit 0ec5f53 into main Apr 8, 2024
@lmb lmb deleted the pr/ci-increase-timeout branch April 8, 2024 09:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants