What does this PR do?

This PR refactors all the CI infrastructure, having one big workflow file instead of multiple ones. After these changes, the ci.yml file is the single source of truth in terms of CI pipelines, defining multiple what I call "quality gates", which is nothing than different gates in the pipeline that will have more importance depending on when they are run in the pipeline: those to the left in the diagram will run earlier because they have more priority.

In this big workflow we will define dependency relationships, leveraging the needs keyword from GH actions, therefore one "stage" o GH job, won't be triggered until its needed job has finished with success. The proposed pipeline defines:

flowchart TD
    Zero( Start )
    Zero --> Core
    Zero --> ModuleGen
    Core[Core Library] -->|ubuntu - macos\nGo 1.19.x, 1.20.x| B[Go Modules]
    Core[Core Library] -->|ubuntu\nGo 1.19.x, 1.20.x| Ryuk[Ryuk Disabled]
    Core[Core Library] -->|ubuntu\nGo 1.19.x, 1.20.x| Rootless[Docker Rootless]
    B[Core Library] --> |ubuntu - macos\nGo 1.19.x, 1.20.x| Examples[Example Modules]
    ModuleGen[Module Generator]
    ModuleGen --> |ubuntu - macos\nGo 1.19.x, 1.20.x| End( End )
    Ryuk --> |ubuntu\nGo 1.19.x, 1.20.x| End( End )
    Rootless --> |ubuntu\nGo 1.19.x, 1.20.x| End( End )
    Examples --> |ubuntu - macos\nGo 1.19.x, 1.20.x| End( End )

At the same time, we have created a reusable workflow to run the Go tests (see ci-test-go.yml file) to run the Go tests. It will receive certain input parameters to configure the execution:

An example usage of this reusable module:

  test-module-generator:
    strategy:
      matrix:
        go-version: [1.19.x, 1.x]
        platform: [ubuntu-latest, macos-latest]
    uses: ./.github/workflows/ci-test-go.yml
    with:
      go-version: ${{ matrix.go-version }}
      platform: ${{ matrix.platform }}
      project-directory: "modulegen"
      rootless-docker: false
      ryuk-disabled: false

As a result, we will be able to run the tests for the core library, the different Docker configs (e.g. rootless), tests for each Go module, with a single reusable workflow, which will define a proper visual name, based on the input parameters used to call it.

Finally, and because creating a new module using the module generator tool created a workflow for each module, we modified the code generator to generate the new pipeline file, adding a list of modules/examples to the matrix of modules/examples.

Why is it important?

We have experimented many situations where our CI jobs take a lot of time to be processed, mainly after an updates wave from @dependabot. This happens basically because the jobs get queued because the number of allocated CI workers is passing the limit of workers in the entire @testcontainers organisation!! AND THIS IS AN ISSUE AS WE WERE IMPACTING OTHER TC-LANG PROJECTS.

With this new approach, we will reduce the number of workers needed by each "gate", reducing the workers that are needed at the same time. From an initial ramp up of +90 workers, we will have:

• a first gate with 8 workers
• a second gate with 40-50
• a third gate with 40-50

allowing more granularity when new PRs (in any repo of the org) come to grab a free worker.

How to test this PR

The result in the GH actions UI will be: https://github.com/testcontainers/testcontainers-go/actions/runs/5750784979?pr=1414