Well it didn't fail at the same spot, this time it was the healthcheck test:

not ok 312 checking container healthcheck in 87ms
# (from function `assert_output' in file test/test_helper/bats-assert/src/assert_output.bash, line 194,
#  in test file test/tests.bats, line 969)
#   `assert_output "healthy"' failed
#
# -- output differs --
# expected : healthy
# actual   : unhealthy
# --
#

That's fairly far down in the test, not sure how it ended up unhealthy?

• Add basic container healthcheck #2625 (comment)
• https://docs.docker.com/engine/reference/builder/#healthcheck

Defaults:

--interval=DURATION (default: 30s)
--timeout=DURATION (default: 30s)
--start-period=DURATION (default: 0s)
--retries=N (default: 3)

I think retries has no delay between attempts, and all retries are part of the same check in the timeout bounds? So likely a false-positive here due to the check being affected at a time postfix was being restarted due to change detection event.. I'll shift this test up to the start then.

Before:

• addmailuser make an account to postfix-accounts.cf, don't exit until /var/mail has created an associated folder /var/mail/<domain>/<user>.
• changedetector notice update in postfix-accounts.cf:
  • Add account to UserDB and PassDB for Dovecot.
  • Create the folder (addmailuser condition met).
  • Apply check for user/group ownership (mkdir called by root in script, so condition triggers chown) - potential race condition (encountered in PR for Dovecot encrypted storage, I have a fix for that to wait on ownership instead of only checking /var/mail account directory exists).
  • Change detection restarts Postfix and Dovecot.
  • Change detection complete.
  • Update checksum file.
  • Dovecot and Postfix may still be coming back up for a moment.

Now:

• addmailuser make an account to postfix-accounts.cf, exit.
• Tests can use wait_until_account_maildir_exists():
  • Wait until directory exists, just like addmailuser did before.
• Tests can use wait_until_change_detection_event_completes():
  • Wait on "Change detection complete" (after first waiting on a "Change detection start" equivalent event is present in logs).
  • Only confirms a change detection has occurred (may be sufficient, account now exist in Dovecot UserDB and PassDB, doveadm utilities can work without running Dovecot instance I think)
  • Otherwise use additional wait on service needed, or worse sleep for some time. No different to handling addmailuser before.

So mostly, addmailuser will finish much more quickly now, and if you actually need the delay to wait until maildir creation, you use wait_until_account_maildir_exists(), it should be have the same. If you want to instead wait a little bit longer, but avoid some potential race conditions, wait_until_change_detection_event_completes() is more useful.

Neither guarantees Dovecot or Postfix is up, as changedetector only reports when it's done processing, not on async activity of services restarting which can still occur after changedetector is finished it's part. wait_on_service() is available, but it just queries supervisord if the service in RUNNING state, which should be the case when the service is not yet ready during it's restart.

New transient failure (this PR is just the gift that keeps on giving with random failing tests..):

not ok 186 checking setup.sh: setup.sh email del in 429ms
# (from function `assert_failure' in file test/test_helper/bats-assert/src/assert_failure.bash, line 66,
#  in test file test/setup-cli.bats, line 116)
#   `assert_failure' failed
#
# -- command succeeded, but it was expected to fail --
# output :
# --
#

This time it's a test this PR actually touches, and is enabling a check that was previously disabled:

  # Mail storage for account was actually removed by `-y`:
  run docker exec "${TEST_NAME}" ls /var/mail/example.com/user
  assert_failure

Potentially a race condition with the previous setup email update test. Change detection is likely running, and perhaps starts to create all accounts from scratch while setup email del deletes the directory, and that is recreated by the changedetector? Doesn't look like it'd have much of a window to do that within :\

This is perhaps one of those scenarios where proper locking with flock() would avoid the issue as the user management commands and the change detector would be actually aware of each other (present file locking was changed around about 10.2, maybe earlier but mistakenly places a lock on the active script running, not the relevant config).

For now I'll ensure the change detector is done processing.. Opted for repeating the check with a timeout instead which should work just as well and minimize the wait time.