This is great to see @martin-schulze-vireso. I marked this as high priority. I'll have a look at Travis later. Keep on the good work!

EDIT: I removed you @fbartels and @erik-wramner. I don't want force you into looking into everything. Sorry for the noise:)

EDIT2: @martin-schulze-vireso contact / mention me when this is not WIP anymore, I'll review it.

Just a quick update: I am still working on this but the going is rough. The quota tests reliable hang when the whole test.bats is executed but if I skip some tests, then they run fine. @aendeavor @erik-wramner any ideas what could cause the quota script to hang indefinitely? I thought about the read but that should never be reached in these tests.

Just a quick update: I am still working on this but the going is rough. The quota tests reliable hang when the whole test.bats is executed but if I skip some tests, then they run fine. @aendeavor @erik-wramner any ideas what could cause the quota script to hang indefinitely? I thought about the read but that should never be reached in these tests.

I'll investigate and report back. EDIT: I too, when I refactored start-mailserver.sh, had the quota tests freaking out, just FYI.

@erik-wramner I am currently working on detecting when there are still undetected configuration changes to wait for them to be applied. To ease the implementation, I moved the checksum file update after the configuration (see 191fe4c) so we always know that if the checksums match, this configuration is already running.

Do you see any negative consequences? I am not very versed in the interactions of the change detector with other components.

Some more debugging info: It seems one of the docker exec commands gets stuck while returning. It seems to be a problem with the container as I noticed that I could not exit a manual docker exec -it session when the container is in this state. Even weirder, the tests stop the container but it is still marked as running in docker ps:

CONTAINER ID        IMAGE                             COMMAND                  CREATED             STATUS              PORTS                                                                    NAMES
41948c869927        tvial/docker-mailserver:testing   "supervisord -c /etc…"   4 minutes ago       Up 4 minutes        25/tcp, 110/tcp, 143/tcp, 465/tcp, 587/tcp, 993/tcp, 995/tcp, 4190/tcp   mail

However, when trying to docker exec it again I get

OCI runtime exec failed: exec failed: cannot exec a container that has stopped: unknown

I also saw another crash pattern where I'd need to restart docker to be able to start the tests again, because the docker run command would fail otherwise.

@martin-schulze-vireso During my current work I noticed the script hang when issuing the restrict-access file. Maybe this is related in some way to this problem of tests hanging, who knows...

@martin-schulze-vireso I noticed you wrote

• update to newer bats version to use more modern features (like parallelization, native setup/teardown_file)

but I cannot find any diff which shows the bats-submodule being updated to the current master. How shall we go about that? I can update this right now in this repo if this helps you. In the end, this needs to be updated somehow.

@aendeavor

but I cannot find any diff which shows the bats-submodule being updated to the current master.

I am sorry for the confusion. I used these checkboxes to track my progress within the ticket, so there is no change on master yet.

However, we can discuss if we want to do this. I used local changes to improve the test development and will PR them in the bats-core project, once they are polished.

However, we can discuss if we want to do this. I used local changes to improve the test development and will PR them in the bats-core project, once they are polished.

Alright. I saw that you had already contributed on the bats project (latest commit). When I realized the checked-out commit in the submodule is more than three years old, I figured updating would be a good idea. But this was just the first idea when I saw this. Would you like to wait until you pushed everything into the bats project, and then I would merge it?

I'd say we wait for the next release, the current change is not in yet and local development can benefit from it: When the tests hang and you abort via CTRL-C, the last command is printed as aborted/failed.

I'd say we wait for the next release, the current change is not in yet and local development can benefit from it: When the tests hang and you abort via CTRL-C, the last command is printed as aborted/failed.

Alright, then we'll wait. Call me back when this work is to be picked up again.

I will add a review in a few hours.

I think I broke Travis. The failed build fails exactly the things I fixed in the last commit. It is telling, that the failure command looks like before the fix. The triggering commit is also listed wrongly as 1a7f761

Ah, it looks like travis is not running the commit itself but a tentative merge of the previous(?) commit with master.

Travis seems to be getting out of hand for useful testing here. I just looked this up: https://stackoverflow.com/questions/21053657/how-to-run-travis-ci-locally

I'll will try this out and see what real CPU cores can do here. This can also ease the pain of debugging Travis, which is ludacris if you ask me :D

PS: I have seen these rm - permission denied errors locally too when testing a few days ago. I will see that the local Travis will say to that.

Well, no need to run travis for local execution. The whole point of my series of work is geared towards smaller testable chunks for local execution when developing new features and more deterministic test outcomes. So you can locally setup once via sudo make build generate-accounts and then sudo ./test/bats/bin/bats test/<your intended file>.bats to just run one file.

However, there is one problem: your computer is probably too fast. Travis clocks in at ~1000 seconds for all tests. I did not test on my machine but it is definitely below 10 minutes. Many of the problems we see are timing dependent race conditions which are not reproducible on faster machines.

Ah, it looks like travis is not running the commit itself but a tentative merge of the previous(?) commit with master.

I can confirm this. This is what the Docker container (travisci/ci-ubuntu-2004:packer-minimal-1602658599-606a7440) does as well:

Well, no need to run travis for local execution. The whole point of my series of work is geared towards smaller testable chunks for local execution when developing new features and more deterministic test outcomes. So you can locally setup once via sudo make build generate-accounts and then sudo ./test/bats/bin/bats test/<your intended file>.bats to just run one file.

Yeah, I know, but I wanted to test it anyways (since Docker is fun), just to be sure.

However, there is one problem: your computer is probably too fast. Travis clocks in at ~1000 seconds for all tests. I did not test on my machine but it is definitely below 10 minutes. Many of the problems we see are timing dependent race conditions which are not reproducible on faster machines.

I figured this would be a problem as well.

EDIT: I should have thought of this - the systemd limitation of Docker makes DinD
dificult, and I will not mess with this. But we figured out Travis is merging with master somehow.

87 and 106 broken again

I am more concerned of 277. Where does this additional mail come from?

I have looked over every change. Are there any particular lines you want me to check? Since this is a big diff, it's probably better if you tell me where you think I should be careful.

I was referring specifically to Tests 108 and 244, which should now be 106 and 242

@aendeavor I'm afraid that bats-core upstream is not moving fast enough for a new version to get in this PR. I would propose to wait for the next release and get that in but I cannot tell how long until then. The project is currently understaffed and looking for maintainers.

Looks good to me!

@martin-schulze-vireso tell me when you're ready for merging. (Or merge it yourself, if you can - I don't know whether you're a collaborator :D)

I am ready for merging and just a lowly contributor ;) so go ahead please.