Update

#634

What kind of machines and how many do you expect to use (see: https://www.packet.net/bare-metal/)?

• 1x Type 2 as an amd64 head node for running builds.
  • This will run gitlab CI (or jenkins, if necessary)
  • Required large, redundant storage to provide head room for build database (postgres)
• 1x Type 2A will run gitlab-runner for ARM builds

We'll also need a place to back up build database in case of catastrophic failure. We will not require hot standby, as impact of failure will be low.

What OS and networking are you planning to use (see: https://www.packet.net/bare-metal/)?

Ubuntu 17.04

Hello containerd folks!

I am Gianluca and I am involved in the community as Curator and Captain. I am a Sotfware Engineer and I work around delivery and builds from some years now.
I had a chat with @crosbymichael about test suites and continuous integration in order to understand what we have, what we need about these topics. I am going to summarize everything here.

containerd has two main criticizes:

• It needs to support different archs and operating system.
• it's going to be integrated into different tools like Docker, K8s and so on. It means that we need to be sure that our changes are not going to break anything, anywhere unexpectedly.

What we have

At the moment we are using Travis-CI to test all our PRs. Plus CodeCov to check the coverage status.

Testsuites

• unit tests

What we are looking for:

As I said containerd is going to be a core part of different high valuable projects as orchestrator, schedulers and container apps. It means that we need to be careful about codes, stability and performance.

• e2e tests: to test containerd as a standalone project.
• Integration tests with the supported platforms. What do I mean? let's say that containerd is going to be used in Docker (obviously) and k8s (it's just an example). What we can do to add more value at these tree projects is to run the 2e2 test suite provided by k8s and the once provided by Docker with a specific version (new pr or the new master after merge) of containerd.
• stress tests. Large orchestrators and schedulers are going to use containerd capabilities very often it means that speed tests and loading tests are going to be useful to have. We can get these informations from the 2e2 tests probably. We don't need to write more code.
• The multi-arch and OS testing.
• Test integration with services that containerd is using (prometheus).

It's going to be a very big matrix and that's why we need to think about frequency. When we are going to run which test suites. The best answer is probably after every PR but maybe we can think about something better :) .

This discussion starts now because in the roadmap there are some big points that are making this thing more that just a dream: the testing process plays a huge part to mark a project as stable and also one of the goals is to donate containerd to an independent foundation in the end of Q1 it means that it's a good time to make plans!

Please feel free to start a discussion. I will try to make this issue up to date with all the things that we will figure out!