@thaJeztah Not linked to these changes but looks like we have an issue with criu repo: https://ci-next.docker.com/public/blue/organizations/jenkins/moby/detail/PR-43682/1/pipeline#step-163-log-428

[2022-06-02T17:20:49.208Z] #51 ERROR: executor failed running [/bin/sh -c echo 'deb https://download.opensuse.org/repositories/devel:/tools:/criu/Debian_11/ /' > /etc/apt/sources.list.d/criu.list         && apt-get update         && apt-get install -y --no-install-recommends criu         && install -D /usr/sbin/criu /build/criu]: exit code: 100

Hmm that repo has had issues a couple of times; maybe it's temporary, or they're fixing it?

Forced push and looks worse now 😅:

[2022-06-02T17:57:17.080Z] #51 19.97 E: Failed to fetch https://mirrorcache-us.opensuse.org/repositories/devel:/tools:/criu/Debian_11/amd64/criu_3.17-1_amd64.deb  Could not connect to mirrorcache-us.opensuse.org:443 (91.193.113.65). - connect (111: Connection refused) Could not connect to mirrorcache-us.opensuse.org:443 (2a07:de40:401::65). - connect (101: Network is unreachable)

Very interesting! I like the idea, but I'm concerned about how repetitive it is (both in the code and in the configuration) -- for example, maintaining that long list of tiny functions and/or the list of all their names in the YAML seems really error prone. 😬

Very interesting! I like the idea, but I'm concerned about how repetitive it is (both in the code and in the configuration) -- for example, maintaining that long list of tiny functions and/or the list of all their names in the YAML seems really error prone. 😬

Yes I'm also worried about missing some tests in the future using this pattern. I will check if go tooling allows to have some stats with available tests and suites in a pkg and split them accordingly in the matrix workflow like this.

@thaJeztah Looks like we are not able to update or add new tests in integration-cli: https://ci-next.docker.com/public/blue/organizations/jenkins/moby/detail/PR-43682/7/pipeline#step-256-log-7 😅

[2022-06-03T10:34:23.854Z] The following new tests were added to integration-cli:
[2022-06-03T10:34:23.854Z] 
[2022-06-03T10:34:23.854Z] +func (s *DockerApiSuite) TestGetContainersAttachWebsocket(c *testing.T) {
[2022-06-03T10:34:23.854Z] +func (s *DockerApiSuite) TestPostContainersAttachContainerNotFound(c *testing.T) {
[2022-06-03T10:34:23.854Z] +func (s *DockerApiSuite) TestGetContainersWsAttachContainerNotFound(c *testing.T) {
[2022-06-03T10:34:23.854Z] +func (s *DockerApiSuite) TestPostContainersAttach(c *testing.T) {

To dynamically generate and distribute tests in a matrix I checked go test -list but unfortunately it doesn't list test suites.

I have created a small library called gotestlist to list tests in the given Go packages. Here is what it looks like if I run gotestlist in integration-cli:

$ go install github.com/crazy-max/gotestlist/cmd/gotestlist@latest
$ cd integration-cli/
$ gotestlist -d 4 ./... | jq

[
  "DockerAPISuite|DockerBenchmarkSuite|DockerCliAttachSuite",
  "DockerCliBuildSuite|DockerCliCommitSuite|DockerCliCpSuite|DockerCliCreateSuite|DockerCliEventSuite",
  "DockerCliExecSuite|DockerCliHealthSuite|DockerCliHistorySuite|DockerCliImagesSuite|DockerCliImportSuite|DockerCliInfoSuite|DockerCliInspectSuite|DockerCliLinksSuite|DockerCliLoginSuite|DockerCliLogsSuite|DockerCliNetmodeSuite|DockerCliNetworkSuite|DockerCliPluginLogDriverSuite|DockerCliPluginsSuite|DockerCliPortSuite|DockerCliProxySuite|DockerCliPruneSuite|DockerCliPsSuite|DockerCliPullSuite|DockerCliPushSuite|DockerCliRestartSuite|DockerCliRmiSuite",
  "DockerCliRunSuite|DockerCliSaveLoadSuite|DockerCliSearchSuite|DockerCliSniSuite|DockerCliStartSuite|DockerCliStatsSuite|DockerCliTopSuite|DockerCliUpdateSuite|DockerCliVolumeSuite|DockerDaemonSuite|DockerExternalVolumeSuite|DockerHubPullSuite|DockerNetworkSuite|DockerPluginSuite|DockerRegistryAuthHtpasswdSuite|DockerRegistryAuthTokenSuite|DockerRegistrySuite|DockerSchema1RegistrySuite|DockerSwarmSuite"
]

-d flag dynamically distributes the tests based on the given matrix size (here 4) and number of tests. This JSON output can then be used as matrix input in our workflow. This way no tests are missing.

I don't calculate cyclomatic complexities of tests functions in gotestlist to add some "weight" and distribute a bit better but I think it's ok in the meantime.

@corhere @neersighted @thaJeztah As discussed on Slack, I looked at how we could aggregate data from JSON reports of Go tests. Last commit adds a step for unit and integration tests that will render the output as markdown and create a GitHub summary in our workflow:

There are also some useful stats about slow tests:

Summary also includes failed tests with logs:

@corhere @neersighted @thaJeztah As discussed on Slack, I looked at how we could aggregate data from JSON reports of Go tests. Last commit adds a step for unit and integration tests that will render the output as markdown and create a GitHub summary in our workflow:
...

Nice! Looks like teststat was a solid find -- everything here looks good to me as long as there's no interest in templating/generating the workflows to avoid duplication.

Nice, gotestlist is really clever -- I'm assuming the lists have such dramatically different numbers of suites due to the number of tests in each suite, right? (Just to make sure I'm understanding the implementation correctly 😅)

Nice, gotestlist is really clever -- I'm assuming the lists have such dramatically different numbers of suites due to the number of tests in each suite, right? (Just to make sure I'm understanding the implementation correctly 😅)

Yes exactly, here are the current stats for tests in integration-cli:

{
  "DockerAPISuite": 151,
  "DockerBenchmarkSuite": 2,
  "DockerCliAttachSuite": 7,
  "DockerCliBuildSuite": 249,
  "DockerCliCommitSuite": 9,
  "DockerCliCpSuite": 43,
  "DockerCliCreateSuite": 20,
  "DockerCliEventSuite": 43,
  "DockerCliExecSuite": 27,
  "DockerCliHealthSuite": 2,
  "DockerCliHistorySuite": 6,
  "DockerCliImagesSuite": 17,
  "DockerCliImportSuite": 7,
  "DockerCliInfoSuite": 5,
  "DockerCliInspectSuite": 29,
  "DockerCliLinksSuite": 14,
  "DockerCliLoginSuite": 1,
  "DockerCliLogsSuite": 15,
  "DockerCliNetmodeSuite": 6,
  "DockerCliNetworkSuite": 14,
  "DockerCliPluginLogDriverSuite": 2,
  "DockerCliPluginsSuite": 6,
  "DockerCliPortSuite": 5,
  "DockerCliProxySuite": 1,
  "DockerCliPruneSuite": 4,
  "DockerCliPsSuite": 18,
  "DockerCliPullSuite": 2,
  "DockerCliPushSuite": 2,
  "DockerCliRestartSuite": 13,
  "DockerCliRmiSuite": 15,
  "DockerCliRunSuite": 298,
  "DockerCliSaveLoadSuite": 16,
  "DockerCliSearchSuite": 5,
  "DockerCliSniSuite": 1,
  "DockerCliStartSuite": 9,
  "DockerCliStatsSuite": 6,
  "DockerCliTopSuite": 4,
  "DockerCliUpdateSuite": 13,
  "DockerCliVolumeSuite": 24,
  "DockerDaemonSuite": 123,
  "DockerExternalVolumeSuite": 18,
  "DockerHubPullSuite": 6,
  "DockerNetworkSuite": 48,
  "DockerPluginSuite": 11,
  "DockerRegistryAuthHtpasswdSuite": 8,
  "DockerRegistryAuthTokenSuite": 5,
  "DockerRegistrySuite": 38,
  "DockerSchema1RegistrySuite": 11,
  "DockerSwarmSuite": 121
}

With gotestlist -d 4 each matrix entry has a max allocation of 388 tests so will be distribute accordingly like this:

[
  "DockerAPISuite|DockerBenchmarkSuite|DockerCliAttachSuite",
  "DockerCliBuildSuite|DockerCliCommitSuite|DockerCliCpSuite|DockerCliCreateSuite|DockerCliEventSuite",
  "DockerCliExecSuite|DockerCliHealthSuite|DockerCliHistorySuite|DockerCliImagesSuite|DockerCliImportSuite|DockerCliInfoSuite|DockerCliInspectSuite|DockerCliLinksSuite|DockerCliLoginSuite|DockerCliLogsSuite|DockerCliNetmodeSuite|DockerCliNetworkSuite|DockerCliPluginLogDriverSuite|DockerCliPluginsSuite|DockerCliPortSuite|DockerCliProxySuite|DockerCliPruneSuite|DockerCliPsSuite|DockerCliPullSuite|DockerCliPushSuite|DockerCliRestartSuite|DockerCliRmiSuite",
  "DockerCliRunSuite|DockerCliSaveLoadSuite|DockerCliSearchSuite|DockerCliSniSuite|DockerCliStartSuite|DockerCliStatsSuite|DockerCliTopSuite|DockerCliUpdateSuite|DockerCliVolumeSuite|DockerDaemonSuite|DockerExternalVolumeSuite|DockerHubPullSuite|DockerNetworkSuite|DockerPluginSuite|DockerRegistryAuthHtpasswdSuite|DockerRegistryAuthTokenSuite|DockerRegistrySuite|DockerSchema1RegistrySuite|DockerSwarmSuite"
]

We should also have a look at (finally) renaming all the c *testing.T to t *testing.T, but that's gonna be more fun to do (there's some tests that will have a conflicting variable, so may not be a straight "rename").

Yes let's do that in a follow-up

I'm bringing this one in; there was some discussion on Slack about the large number of jobs (as there's a job limit in GitHub actions); we'll likely run into that limit if we're gonna add Linux as well, but I think this gives the foundation to be flexible, and we can re-slice-and-dice once we run into those limits