Not sure if this counts as breaking. Marked it for v12.1.0, but v13.0.0 is also fine - just change it if you guys think this is breaking.

The fact the test require being removed (failures?) implies a breaking change 🤔 Delay until v13

@jrpear I saw your detailed comment with references to issues and PRs, that is very appreciated and I'll try go through that tomorrow 👍

IIRC we have cron sending mail internally for some things, amavis is also a system user involved IIRC? (from address only I think).

I am aware of a case on the docs that uses a devnull system alias similar to the root, to get the mail.example.com FQDN appended, although that may very well be a workaround to the current behaviour from the looks of it 😅

https://docker-mailserver.github.io/docker-mailserver/latest/faq/#how-can-i-configure-a-catch-all

Do the tests pass if you only dropped the main.cf? It seems like they would?

default: $myhostname, localhost.$mydomain, localhost) - Postfix Docs

Scripts default add the root system alias to Postmaster address:

Which in tests was being caught by the regexp alias for postmaster to redirect to the user mail account:

I don't think we have sufficient test coverage on this particular change to have as much confidence as I'd like, but that's not been an easy goal to reach. I'll go over the detailed comment in the associated issue tomorrow to review the history.

IIRC we have cron sending mail internally for some things, amavis is also a system user involved IIRC? (from address only I think).

Alright, I'll see if I can find that cron mail and whether it's broken by this. I'll take a look at Amavis too. But as long as people don't need to be able to reply to amavis (IDK how it works at all), that shouldn't matter.

I am aware of a case on the docs that uses a devnull system alias similar to the root, to get the mail.example.com FQDN appended, although that may very well be a workaround to the current behaviour from the looks of it sweat_smile

Ah, good spot. I don't think that trick is working around the current config, I think the trick would no longer work with this change because the trick needs to send mail to a system user. There is another way to discard mail sent to a particular address.

I haven't decided whether I think it's best to allow any mail to be sent to system users. I'd prefer totally disallowing it because it'd make our postfix config easier to understand; we would know that all email was being delivered by virtual (8) rather than local (8). But there may be no way to get everything working that way.

Do the tests pass if you only dropped the main.cf? It seems like they would?

I think you meant "if you only dropped the $myhostname"? Then the same three tests still fail. I think this is because mail to root is being rewritten to mail to root@$myorigin (myorigin = $myhostname by default) per append_at_myorigin. If I set myorigin = localhost too, then the three tests pass.

Not the simplest solution, but probably the easiest solution (I'm referencing simple made easy).

Scripts default add the root system alias to Postmaster address:

Ah yes, if we do remove delivery to local users then this would need to be removed too.

I don't think we have sufficient test coverage on this particular change to have as much confidence as I'd like, but that's not been an easy goal to reach. I'll go over the detailed comment in the associated issue tomorrow to review the history.

There's never enough tests 😄

If there's a relevant test I could write that would help check the safety of this change, let me know.

I've read your feedback here and it seems pretty good 👍

I didn't manage to find time to look further into this for the proper review, I'll try again tomorrow 😅

If I get too swamped to participate in further review, I am happy with your feedback and effort to document the motivation supporting the change. You have my approval at present 😀

Summary so far:

There's another solution to this problem which probably isn't breaking and makes things a bit more complicated:

< mydestination = $myhostname, localhost.$mydomain, localhost
---
> mydestination = localhost.$mydomain, localhost
> myorigin = localhost

Todo to make my preferred change safer and better:

• Check whether any mail from a cron job or amavis is broken (maybe write a test too if it's easy)
• Figure out other way of discarding mail for a specific address and change FAQ (maybe add a test for this too) (probably will use this Unix SE answer)
• Remove creation of root alias
• Investigate whether aliases are still tested after this change - add a new test if not.

I couldn't find any mail sent by a cron job.

A grep in the whole repo for "root" found no messages whose hard-coded destination was 'root'.

I can't find any great documentation for Amavis, but I think it delivers virus notifications to virus_admin, which is set in Debian to postmaster:

/etc/amavis/conf.d/20-debian_defaults:$virus_admin = "postmaster\@$mydomain"; # due to D_DISCARD default

So that should be fine. I haven't managed to understand how to configure Amavis or how it communicates with Postfix so I'm not going to try to write a test.

This looks already very good to me 👍🏼 Good job!

The only thing I wish for are some tests, even if they are few and small (if writing such tests is feasible and makes sense, of course) ❤️

I made the existing tests drive this change in 3508485 (not the best commit message as no tests are added). After that commit, several tests fail. After applying 5366c90 and 5519536, all the tests pass again.

I can't think of any other tests to write to verify this, but I'm open to suggestions.

Hopefully later today I'll investigate the alias thing and fix that FAQ.

I'm not sure what to make of the test that failed to pass CI. It passes on my computer. The line that fails is the second time the command has been run (the first being on the previous line).

Edit: will look into it more another time.

I can't find any great documentation for Amavis

Yeah this was my experience as well when looking into the recent SA_KILL change and referencing sources for that.

I used the official Amavis docs (although I guess you found that 😅 ) and a few third-party sites that explained things in better detail. There's also the Amavis Gitlab repo which is sometimes helpful. Other times I looked through the config within the container, especially since we have a few modifications of our own and from Debian there.

I haven't managed to understand how to configure Amavis or how it communicates with Postfix

I haven't looked through these Amavis docs on Postfix integration, but assume that our config is based on that.

I am aware of an issue with ports 587 / 465 being subject to Amavis filtering which may not be desirable.

Neither is too concerning though as we're looking at deprecating Amavis in future in favor of Rspamd.

I haven't decided whether I think it's best to allow any mail to be sent to system users.

I'd prefer totally disallowing it because it'd make our postfix config easier to understand; we would know that all email was being delivered by virtual (8) rather than local (8).
But there may be no way to get everything working that way.

Kind of want to agree, but it also kind of makes sense for the system accounts and that kind of mail to be distinct?

It's probably easier to manage and validate with /etc/aliases + local(8) than it is with /etc/postfix/virtual + virtual (8)? (Dovecot /etc/postfix/vmailbox)

It's just the usage of bare-domains that complicates it a bit more 😛 (but it turns out there are cases where users also want the domain-part of an address with their accounts to use the full hostname too 🤷‍♂️ )

Common cron failure for Amavis

Here is an example of a mail delivered from root cron job that failed an Amavis maintenance task related to SpamAssassin:

From amavis@localhost  Sun Apr 23 12:18:05 2023
Return-Path: <amavis@localhost>
X-Original-To: amavis
Delivered-To: amavis@localhost
Received: by mail.destination.test (Postfix, from userid 109)
        id 2FA674238F5A; Sun, 23 Apr 2023 12:18:05 +1200 (NZST)
From: root@localhost (Cron Daemon)
To: amavis@localhost
Subject: Cron <amavis@mail> test -e /usr/sbin/amavisd-new-cronjob && /usr/sbin/amavisd-new-cronjob sa-sync
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/var/lib/amavis>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=amavis>
Message-Id: <[email protected]>
Date: Sun, 23 Apr 2023 12:18:05 +1200 (NZST)

razor2: razor2 check failed: Invalid argument razor2: razor2 had unknown error during get_server_info at /usr/share/perl5/Mail/SpamAssassin/Plugin/Razor2.pm line 186. at /usr/share/perl5/Mail/SpamAssassin/Plugin/Razor2.pm line 329.
internal error, python traceback seen in response: Traceback (most recent call last):\n  File "/usr/bin/pyzor", line 432, in <module>\n    main()\n  File "/usr/bin/pyzor", line 152, in main\n    if not dispatch(client, servers, config):\n  File "/usr/bin/pyzor", line 265, in check\n    send_digest(digested, runner, servers)\n  File "/usr/bin/pyzor", line 286, in send_digest\n    _send_digest(runner, servers[0], digested)\n  File "/usr/bin/pyzor", line 277, in _send_digest\n    runner.run(server, (digested, server))\n  File "/usr/lib/python3/dist-packages/pyzor/client.py", line 258, in run\n    response = self.routine(*args, **kwargs)\n  File "/usr/lib/python3/dist-packages/pyzor/client.py", line 116, in check\n    sock = self.send(msg, address)\n  File "/usr/lib/python3/dist-packages/pyzor/client.py", line 138, in send\n    return self._send(msg, address)\n  File "/usr/lib/python3/dist-packages/pyzor/client.py", line 143, in _send\n    for res in socket.getaddrinfo(addr[0], addr[1], 0, socket.SOCK_DGRAM,\n  File "/usr/lib/python3.9/socket.py", line 953, in getaddrinfo\n    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):\nsocket.gaierror: [Errno -3] Temporary failure in name resolution at /usr/share/perl5/Mail/SpamAssassin/Plugin/Pyzor.pm line 308.

• This failure seems to be due to my offline setup (containers are configured to use DNS from another container that doesn't forward to a public resolver).
• It repeats every 3 hours (our scripts only manage the filename /etc/cron.d/amavisd-new).
• Cron identifies the user and host in the subject line via an address with domain-part as hostname -s (mail).

/etc/cron.d/amavisd-new:

#
#  SpamAssassin maintenance for amavisd-new
#
# m h dom mon dow user  command
18 */3  * * *   amavis  test -e /usr/sbin/amavisd-new-cronjob && /usr/sbin/amavisd-new-cronjob sa-sync
24 1  * * *   amavis  test -e /usr/sbin/amavisd-new-cronjob && /usr/sbin/amavisd-new-cronjob sa-clean

This failure has been reported in the past, where the local mailbox file at /var/mail/amavis was being written to but future deliveries had a permissions error (as noted in my prior review comment).

• Workarounds suggested was an alias to an actual mailbox (not resolving the reported cron failure itself, just the delivery failure).
• It has been reported multiple other times usually with the suggestion to document creating an amavis alias. Others resolves with an amavis alias to root to indirectly redirect mail to the default Postmaster account.
• One user tried to contribute a PR to silence unwanted Amavis mail. Unlike the above failure, it was reporting a success mail (another report of this) but maintainers could not reproduce receiving any similar mail from Amavis:

  config: created user preferences file: /var/lib/amavis/.spamassassin/user_prefs

cron usage in DMS

We have other cron jobs in /etc/cron.d, /etc/cron.daily, /etc/crontab (_makes a reference to anacron which isn't installed, but this seems to just be for compatibility? _) and /var/spool/cron/crontabs/root (contains the virus-wiper task added via our Dockerfile).

For those like myself not too familiar with the cron management, this is useful info:

• There is an ignore pattern on filenames in the cron config folders, we use .disabled where the . prevents cron running it. This seems to be via run-parts which is used in /etc/crontab to run the /etc/cron.* config dirs.
• The /var/spool crontab location is for user crontabs, but not that appropriate for root. We can move that back to /etc/cron.d but must specify a user to run as otherwise it's also ignored apparently.
• We have a mixed usage elsewhere in scripts that add cron jobs into cron.d (crontab format) or cron.daily (scripts run as root, such as SA-KAM).
• Additionally regardless of user the cron job runs as, the ENV is not inherited. So we cannot rely on any ENV being present that were set on the container or during startup scripts. We did introduce a workaround for most of these via /etc/dms-settings.

Local (system) addresses

This comment notes:

System generated emails like cron job emails or other system notifications are sent to usually root@localhost or root@<hostname>.
Those emails may fail to reach their destination and stay in the queue.

As seen above with amavis recipient from cron (via root sender address), mail was sent to the user only, and Postfix appended the myorigin value. But you have identified configuration for Amavis that would send to postmaster@$mydomain, and the subject from cron using @$(hostname -s).

SpamAssassin has been known to fail it's KAM cron task (run as root).

There was also some discussion on setting up default aliases beyond root, which also configures for the clamav user. While during review a maintainer mentions users may have a workaround already in place by having their own amavis or similar aliases added.

If any one has deployed DMS with modifications to run additional software within the container, it may also implicitly attempt to send system mail. One user reported @localhost mail failures for accounts that did not exist in DMS (but was apparently due to running some service on the same host, outside of the container). Their container config used fetchmail, and an existing instance of fetchmail outside of DMS on the host was some how involved, that DMS was logging / processing mail for those external accounts.

We additionally have users adding their own local aliases like devnull for redirecting mail to be discarded (eg: for a [email protected] send-only address):

• How to setup a catch-all mail but block specific ones #1335
• docs: FAQ - Add note for devnull alias gotcha when using a catchall rule #2949 (bad workaround I approved 😬 )

Aliases and hostnames

Catch-all domain patterns:

• Cannot send e-mail from wildcard address #3093 (virtual alias redirects all accounts to single domain)
• [FR] Add the ability to redirect a domain catch-all to some user mailbox #3022 (requests wildcard support at virtual mailbox level instead, to avoid explicit opt-out)

DMS specific behaviour that probably needs to be taken into consideration:

• Workaround that was added for Dovecot Quotas support (default enabled feature). This adds dummy accounts into Dovecot for virtual aliases, and some users have been seeing it as a "feature" for SASL login since.
• Creation of /etc/postfix/vhosts. Sometimes problematic for users when aliases are added that aren't intended as accounts managed by DMS.
  • This file is used for virtual_mailbox_domains for Postfix to send mail via LMTP to Dovecot to handle. Related can be Dovecot dummy mailbox users being created for any aliases.
  • One user created a mailbox for @gmail.com for a service to send mail as. This prevented other accounts sending mail to actual gmail addresses as @gmail.com had been added into /etc/postfix/vhost. Their solution was to modify /etc/postfix/vhost, although that seems like a misconfiguration..
• Some users run a separate mail-server for example.com, and want to run DMS for mail.example.com (and send as [email protected]), so we should be careful about the distinction of $myhostname / ${HOSTNAME} (overwritten system var during setup) and $mydomain / $DOMAINNAME (internal setup var).

We could probably better document the distinction of the hostname for DMS not being related to the mail addresses it manages. Users often confuse this with affecting the domain-part of an address (thinking [email protected] is required due to hostname set), or they get confused when configuring DMS with multiple domains when it comes to TLS certs (only need for DMS hostname).

Additional references

DMS image users + groups

I documented users and groups in our DMS image earlier this year: #3011 (comment)

That provides a better overview of system accounts that are relevant for mydestination / local(8) currently.

Alias script/support improvements tracking issue + flock

There is a tracking issue about improving our support for it. I summarize what needs to be done at the end of that issue, which links to more detail about file lock situation.

Many improvements were made, but I never got to the point to bring back flock and verify NFS support should be compatible with proper config. I have extensive notes locally that I haven't revised for sharing but I can provide them if anyone needs it.

DMS usage of ${HOSTNAME} (in future to be replaced by DMS_FQDN)

Relevant to $myhostname / $mydomain / $myorigin and usage in mydestination and elsewhere.

• ${HOSTNAME} is a system ENV and it's value sourced via other means by software or configs beyond our own explicit usage
• Choosing to use ${HOSTNAME} was a legacy decision, and related logic in scripts has had various issues in the past before the current dns.sh existed. We've been reluctant to risk breakage with a rename to DMS_FQDN due to test coverage not being adequate (and a mess back then to manage / debug). This change will likely be risked in the next major version or so with the option to revert or fix issues should regressions arise. Ideally there is no major changes / refactoring in that release that would complicate troubleshooting those regressions.
• We override ${HOSTNAME} during startup scripts in helpers/dns.sh, however:
  • Outside of startup scripts (any process that runs without inheriting that modified environment) may receive a different value for ${HOSTNAME} (or other methods to source it like: system calls with like gethostname(), or hostname -f / hostname -d, etc).
  • hostname: mail.example.com / --hostname mail.example.com from the DMS container config should be the value used in most cases, but not when:
    • Using OVERRIDE_HOSTNAME (for containers that cannot explicitly configure the hostname). This is only used to override ${HOSTNAME} within the startup scripts ENV. Without that in scope, software may receive a different hostname value.
    • The value is constructed differently instead of a full FQDN (potentially an issue for bare domains):
      • Hostname value/resolution can vary by distro (base image) setting some related /etc data differently, different implementations of hostname command, container network config / CRI manipulating containers /etc/hosts - which can also differ based on the hosts own DNS config (Docker also has special handling when systemd-resolved is used).
      • See this reference table I created from an earlier investigation.
        • The refactoring PR was rather massive in discussion / investigation, a summary with relevant links for more info and concerns + future work can be found here.
        • More recently, related to those changes was the refactor of the mail_hostname.bats tests. In addition to many other tests being updated in the v12 cycle, it has been noticed that tests lack a fair amount of coverage desired for improved confidence via an eventual test matrix on supported configs. Effort to address that is considerable, thus it is unfortunately easier to risk introducing regressions by swapping usage of the $HOSTNAME var to DMS_FQDN, and fixing any regressions reported.

Documentation preview for this PR is ready! 🎉

Built with commit: 75571ca

@polarathene I've finally got to read and think through what you wrote out. It seems that this change is a lot more complicated than I thought it would be.

I've run out of interest in solving this.

I've pushed a revert to the first solution: unsetting mydestination altogether. I'll leave it up to you all whether to merge it or not.

I'm going to move on from DMS, so I won't be able to provide any support or fixes for bugs this might introduce.

This is a great project, you're all so friendly and generous with your time. Good luck!

I'm going to move on from DMS

No worries! :)

If you don't mind did you choose a different open-source project, a service like SendGrid or something else?

@polarathene do you think we should merge this PR then?

do you think we should merge this PR then?

No, but mydestination may be worth some change given the above. I'm just not sure if localhost is ideal either.

That said, it is easier to just recommend mail.example.com over example.com for DMS? There was alot to consider as shown above. May be useful to keep a reference / link to that via some maintainer docs or issue? That way you can close this, but reference it if similar is suggested in future.

I'm going to move on from DMS

If you don't mind did you choose a different open-source project, a service like SendGrid or something else?

I decided to try mox, an attempt at reimplementing the whole Postfix, Dovecot, SpamAssassin, OpenDKIM, etc. suite in one program (maybe even one process?).

It seems like the right way to do a small email server, but it's very young. Hopefully it isn't abandoned.

do you think we should merge this PR then?

No, but mydestination may be worth some change given the above. I'm just not sure if localhost is ideal either.

I am too short on time to work in this as well now. We can keep it in mind for the future though.

That said, it is easier to just recommend mail.example.com over example.com for DMS?

Yes 👍🏼

There was alot to consider as shown above. May be useful to keep a reference / link to that via some maintainer docs or issue? That way you can close this, but reference it if similar is suggested in future.

👍🏼 Can you update the docs with a short (really) note?:)

Can you update the docs with a short (really) note?:)

Linking back to this PR from the bare domain FAQ entry? Or a maintainers / contributors page for awareness?

Can you update the docs with a short (really) note?:)

Linking back to this PR from the bare domain FAQ entry? Or a maintainers / contributors page for awareness?

Whatever you deem to be the better option. Preferably the way that involves less work (so probably option 1: link back per FAQ?).

Superseeded by #3316