PROPOSAL: New CI structure for building, testing and releasing

Intro

Our current CICD code relies on the following components:

• Several Github action YAML workflows, used to call make or directly call goreleaser for official releases
• A Makefile at the root of the repo which runs commands on everyone local machine (not containerized), like for instance: launch unit tests, run the linter or build the dagger binary used for development
• Yarn to install and spin-up Bats
• Bats to run integration test suites

Problems

The current setup is not reproducible, self-contained or portable.

Here is a non-exhaustive list of problems:

• The number of Github action worklfows and complexity are growing over time:
  • CI issues are opaque to the underlying infrastructure (cannot reproduct an issue locally)
  • Workflow logic is sticky to Github, so switching to another infra will need a full rewrite
• Every command run from the Makefile are executed on the host, so the behavior depends on every developer machine (build dagger binary, run tests, etc...)
• Every single test suite is run for every single change (e.g. the change of line of HTML in the website triggers a full re-run of the engine integration tests)

Proposal

TL;DR: Implement all builds, tests and deploy from a single Dagger Plan.

Objectives

• Portable pipelines: (the behavior on the CI is the same as every developers machine)
• Leverage Dagger's DAG: automatically figures out what test suite needs to be run depending of a specific code change to dagger (a docs change will not re-trigger engine integration tests because the dagger binary did not change)
• Use our CI as a reference implementation for users (a.k.a dogfood)

Main changes

• The Github action worklow will simply run the whole plan, it won't have any logic
• Yarn won't be needed to manage Bats
• Bats will be a another dependency (see Bats integration in stdlib #1213) - I tested and it works to run Dagger integration tests from a Dagger plan
• The parallelization of tests won't be managed by Bats (Bats forking is already causing issues on the CI)
  • Dagger will handle concurrency by declaring one #Bats per test suite

Tiny POC

Snippet of a working configuration building the dagger binary and running a the simple test suite (the whole ./tests/cli.bats suite runs without issues):

package main

import (
    "alpha.dagger.io/dagger"
    "alpha.dagger.io/go"
    "alpha.dagger.io/bats"
)

// docker socket
dockerSocket: dagger.#Stream & dagger.#Input

// build the dagger binary
build: go.#Build & {
    output: "/dagger-dev"
    packages: "./cmd/dagger/"
}

// run the bats test suite once `go build` completed
test: simple: #Bats & {
    env: DAGGER_BINARY: "/bin/dagger/dagger-dev"
    setupCommands: ["apk add --update docker"]
    initScript: "chmod +x $DAGGER_BINARY"
    socket: "/var/run/docker.sock": dockerSocket
    mount: "/bin/dagger": {
        from: build
    }
}

dagger init dir build.source ./git/repos/dagger
dagger init dir test.simple.source ./git/repos/dagger/tests
dagger init socket dockerSocket /var/run/docker.sock

Notes:

• For simplicity, this example uses one bats instance to run the whole suite, we will want one bats instance per test suite to allow dagger to parallelize
• The test source can be taken directly from the repo

Missing features

Although the simple example above works with the latest version of Dagger, we are missing several features in order to implement the full proposal.

In order of importance:

1. Run a part of the DAG → Selectively run parts of the DAG #1396
2. Output a dagger.#Artifact to the host filesystem (replace make to build the binary) already implemented in Europa
3. Control the execution flow (e.g.: allow certain tests to fail): Programatically control the execution flow #989

The first item is open to discussion as several ideas where previously discussed.

The need is really to replace this kind of commands with a dagger up:

• Run a single test: make && ./tests/node_modules/.bin/bats "./tests/<TESTFILE>.bats" -f "<TESTNAME>"
• Lint the code: make lint
• Build the dagger binary: make