I've dropped the commit cited below for the following reasons:

• The linked PR comment concern was about running generate-accounts multiple times. This doesn't change that, the matrix feature will run all steps on separate CI instances, thus will happen regardless and is necessary.
• CI: true I assume from your commit message you maybe were wanting pretty output instead? Github has limited support here, it's able to support colour, but not other features/syntax used to manipulate terminal output AFAIK. I had looked into this back in August, and raised an issue at bats-core which explains that --pretty will not work.

If I'm mistaken and you intended something else, make this change in a separate PR as it's unrelated to this one 👍

Dropped commit

Commit message (original commit title differed):

tests(CI): Drop `CI: true` + separate step for `generate-accounts`
===
See https://github.com/docker-mailserver/docker-mailserver/pull/2822#discussion_r996283731

I also removed `CI: true` because I think GH Runners can now properly handle colored TTY output.

Commit Diff:

+      - name: 'Prepare tests'
+        run: make generate-accounts
+
      - name: 'Run tests'
-        run: make generate-accounts tests/${{ matrix.part }}
-        env:
-          CI: true
+        run: make tests/${{ matrix.part }}

Commit history has been revised, here's a breakdown:

1. 3fc8945 - Move all existing tests into their serial / parallel subdirectories.
2. 638eedb - @georglauterbach modified some tests here, presumably as an example of changes / improvements for a future PR to make to all other tests as a convention. I have written a more detailed commit message highlighting the changes covered in the diffs, noting the common change across files and some file specific ones that stood out.
3. 01a498d - Has a bit more going on. I'm not quite following the comment explanation for using explicit / hard-coded container names instead of the ENV var that had been working, apparently that breaks with the parallel feature? (couldn't we just assign different var names though?)
4. 7b00907 - Adds what looks like a reference / example (template.bats) to copy / paste for new tests, or for updating existing tests with to support parallel testing? The additional serial test I haven't looked over yet, and doesn't seem relevant to this PR, but a separate test in itself that could have been added via it's own PR?
5. 369e4b4 - Besides the file renames, the bulk of files changed is just the top line to use an absolute path via REPOSITORY_ROOT (defined in the Makefile), this allows for the nested test folders to properly load test helpers.
6. 5789915 - Finally, all changes to Makefile and the CI testing workflow is bundled into this commit. The commit message is revised based on the content of multiple original commit messages for these file changes from @georglauterbach .

Notes

• I have made a slight error it seems. serial/no_container.bats was renamed to serial/setup.sh.bats thus there is an actual diff between the two to review, it is not a new test file.
• The github review view is not too helpful with some files. This is due to the diff being a merge commit one, that is unable to recognize some files that were renamed and received changes from @georglauterbach are the same file... So you'll want to refer to the actual commit to review the diff there (the commit message should summarize the diff fairly well).

Will answer the rest of the question later - not enough time at the moment :)

So I revised the commit history and did some things in a slightly different way. Had to force-push the branch - hope you don't mind.

In commit 22d358f you reverted some changes you introduced.

The commit message explains why, but I am curious about some of the reverted content. Noting here in case you'd like to add via a follow-up PR at some point:

The file in question probably needs a dedicated follow-up PR indeed. I completely reverted the changes because I was not sure on how much of an impact my initial changes were and why some of the changes lead to test failures. In short: The questions you stated should indeed be picked up later; I cannot give you a comprehensive answer as I currently lack the answers and a proper explanation myself.

@polarathene for some reason I cannot directly answer #2857 (comment), so here it is:

generate-accounts is run 4 times, because I observed some test files alter the accounts files which leads to other tests failing for no apparent reason. There is some sort of dependency here on the order test files that are run. This is not optimal, but currently solved by running the generate-accounts target before every new test set invocation.

In short: The questions you stated should indeed be picked up later; I cannot give you a comprehensive answer as I currently lack the answers and a proper explanation myself.

No worries, that can be looked into after this is merged then 👍

@casperklein @polarathene I pushed the commits addressing the latest PR feedback. Hope all as well now - happy final reviewing :D

Apologies for the delay, finally got around to tackling this! 😀

• Refactored commit history from 24 commits down to 5.
• Commit messages rewritten accordingly.
• There is no difference in output to review.

Upon approval, merge via merge commit or rebase, not squash.

Documentation preview for this PR is ready! 🎉

Built with commit: 75ee0c1

ping @casperklein - when you provide approval for this PR (nothing has changed except commit count and descriptions), I think we can start releasing 11.3.