As I started looking back into #256 and #250 some maintainability issues have become apparent within the test suite. Given the current set of tests, coverage is low since we are only testing the ability to send mail on the mail docker image.

For instance, in #250, the test passes because we only test that the service is not running, however the ability to send an email is never tested, allowing the bug to be introduced.

As I see it, an ideal solution that would avoid the bugs above is to run through the basic functions of the mail server for each option that is configured.

Proposal

Possible implementation would need to be explored

A set of core tests that are common across all configurations

This is a mail server, so always test SMTP, IMAP, log configs, accounts and any other core services are always correctly working no matter what combination of environmental variables are set.

Each environmental variable would have a set of core tests associated with each possible value.

example: SSL_TYPE would have 5 sets of tests associated with it.

1. SSL_TYPE= => verify no ssl is configured
2. SSL_TYPE=letsencrypt => verify Let's Encrypt certificates
3. SSL_TYPE=custom => verify custom certificates
4. SSL_TYPE=manual => verify custom locations of your SSL certificates for non-standard cases
5. SSL_TYPE=self-signed => verify self-signed certificate work

Some kind of preprocessorfor BATS that generates all tests cases

Once this information is defined, it would be possible to generate complete test coverage for every configuration (all combinations of environmental variables) by using the associated set of tests for each option being tested.

Possible Issues

1. Compexity in creating / maintaing a BATS preprocessorfor.
2. The number of combinations grows exponentially, making testing all combinations infeasible.

The more I think about this particular solution, the messier it seems. Ultimately, I still think there are coverage issues that should be solved.

Thoughts?