Why do Postfix and Dovecot even need certificates for host names other than $MAILCOW_HOSTNAME?

Proxying Postfix will likely break fail2ban because it won't pass through source IP addresses. Proxying port 25 would be really dangerous for that reason too (DNSBLs and trusted networks need the correct IP).

Thanks for your feedback.

Why do Postfix and Dovecot even need certificates for host names other than $MAILCOW_HOSTNAME?

Postfix does not get any other certificates.
Enabling other certificates with SNI allows using white-label domains when configuring the mail client (ie mail.anydomain.com).
While Dovecot does not need them, I added this as default because if someone is already using subdomains as server addresses, this would break it, since the subdomain.* are not longer included in the default server certificate.

Proxying Postfix will likely break fail2ban because it won't pass through source IP addresses. Proxying port 25 would be really dangerous for that reason too (DNSBLs and trusted networks need the correct IP).

It does not break it, as dovecot will do the authentication in place of postfix, see this log from dovecot:

submission-login: Remote closed connection (auth failed, 2 attempts in 108 secs): user=<[email protected]>, method=LOGIN, rip=84.x.x.x, lip=172.22.1.250, TLS, TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)

which is the same format that f2b already recognized from imap / pop3.

Port 25 is not proxied for the reason you mentioned - still directly mapped to postfix and not using SNI, only the server certificate.

Is there any other reason why postfix might cause issues when mails from authenticated users are proxied? Any spam checks or similar stuff on outgoing mails?

What is submission_relay_port in your dovecot.conf used for? If Dovecot delivers everything to that port, Postfix will not know the submitting user name and can't perform sender checks, rate-limiting, etc.

Does TLS (via STARTTLS) on port 587 still work?

It does not break it, as dovecot will do the authentication in place of postfix, see this log from dovecot:

Interesting. I did not know that that is how it works.

Is there any other reason why postfix might cause issues when mails from authenticated users are proxied? Any spam checks or similar stuff on outgoing mails?

I don't think we do rate-limiting per IP, so that would be fine.

What is submission_relay_port in your dovecot.conf used for? If Dovecot delivers everything to that port, Postfix will not know the submitting user name and can't perform sender checks, rate-limiting, etc.

That is correct and probably an issue atm. Thanks for pointing it out. I have researched this and think the XCLIENT commands from dovecot should solve this, which are already enabled by submission_relay_trusted = yes. However I will try to verify this evening what changes are needed in postfix so it trusts these commands and identifies the correct sender (or maybe it already does).

Does TLS (via STARTTLS) on port 587 still work?

Yes, STARTTLS on port 587 and direct SSL on 465 work with no issues in my tests.

After some changes, postfix now gets the correct sasl_sender as authenticated in dovecot.

However it's not applying the sender restrictions, still have to figure out how to apply them when relaying via dovecot.

I have now figured out the correct XCLIENT config, turned out i had to whitelist the dovecot IP instead of hostname.
All sender restrictions are working because dovecot is telling postfix the real client IP and sender login with XCLIENT, see the following log:

---- SUCCESSFUL MAIL ----
Apr 10 22:25:01 mail2 postfix/postscreen[340]: CONNECT from [172.22.1.250]:50142 to [172.22.1.9]:25
Apr 10 22:25:01 mail2 postfix/postscreen[340]: WHITELISTED [172.22.1.250]:50142
Apr 10 22:25:01 mail2 postfix/smtpd[342]: connect from mailcowdockerized_dovecot-mailcow_1.mailcowdockerized_mailcow-network[172.22.1.250]
Apr 10 22:25:01 mail2 postfix/smtpd[342]: 97530166AAB: client=mailcowdockerized_dovecot-mailcow_1.mailcowdockerized_mailcow-network[84.x.x.x], sasl_method=XCLIENT, [email protected]
Apr 10 22:25:01 mail2 postfix/cleanup[346]: 97530166AAB: message-id=<[email protected]>
Apr 10 22:25:01 mail2 postfix/qmgr[337]: 97530166AAB: from=<[email protected]>, size=1680, nrcpt=1 (queue active)
Apr 10 22:25:01 mail2 postfix/smtpd[342]: disconnect from mailcowdockerized_dovecot-mailcow_1.mailcowdockerized_mailcow-network[84.x.x.x] ehlo=1 mail=1 rcpt=1 data=1 quit=1 commands=5
Apr 10 22:25:02 mail2 postfix/lmtp[348]: 97530166AAB: to=<[email protected]>, relay=dovecot[fd4d:6169:6c63:6f77::10]:24, delay=0.59, delays=0.39/0.03/0.05/0.13, dsn=2.0.0, status=sent (250 2.0.0 <[email protected]> tAprAx5RrlyZAAAABx98KA Saved)
Apr 10 22:25:02 mail2 postfix/qmgr[337]: 97530166AAB: removed

---- SENDER RESTRICTIONS ----
Apr 10 22:24:54 mail2 postfix/postscreen[340]: CONNECT from [172.22.1.250]:50100 to [172.22.1.9]:25
Apr 10 22:24:54 mail2 postfix/postscreen[340]: WHITELISTED [172.22.1.250]:50100
Apr 10 22:24:54 mail2 postfix/smtpd[342]: connect from mailcowdockerized_dovecot-mailcow_1.mailcowdockerized_mailcow-network[172.22.1.250]
Apr 10 22:24:54 mail2 postfix/smtpd[342]: NOQUEUE: reject: RCPT from mailcowdockerized_dovecot-mailcow_1.mailcowdockerized_mailcow-network[84.x.x.x]: 550 5.1.0 <[email protected]>: Sender address rejected: User unknown in virtual mailbox table; from=<[email protected]> to=<[email protected]> proto=ESMTP helo=<mail2.domain.local>
Apr 10 22:24:56 mail2 postfix/smtpd[342]: disconnect from mailcowdockerized_dovecot-mailcow_1.mailcowdockerized_mailcow-network[84.x.x.x] ehlo=1 mail=1 rcpt=0/1 quit=1 commands=3/4

---- RATE LIMITED ----
Apr 10 22:25:08 mail2 postfix/postscreen[340]: CONNECT from [172.22.1.250]:50200 to [172.22.1.9]:25
Apr 10 22:25:08 mail2 postfix/postscreen[340]: WHITELISTED [172.22.1.250]:50200
Apr 10 22:25:08 mail2 postfix/smtpd[342]: connect from mailcowdockerized_dovecot-mailcow_1.mailcowdockerized_mailcow-network[172.22.1.250]
Apr 10 22:25:09 mail2 postfix/smtpd[342]: 017B1166AAB: client=mailcowdockerized_dovecot-mailcow_1.mailcowdockerized_mailcow-network[84.x.x.x], sasl_method=XCLIENT, [email protected]
Apr 10 22:25:09 mail2 postfix/cleanup[346]: 017B1166AAB: message-id=<[email protected]>
Apr 10 22:25:09 mail2 postfix/cleanup[346]: 017B1166AAB: milter-reject: END-OF-MESSAGE from mailcowdockerized_dovecot-mailcow_1.mailcowdockerized_mailcow-network[84.x.x.x]: 4.7.1 Ratelimit "mailcow" exceeded; from=<[email protected]> to=<[email protected]> proto=ESMTP helo=<mail2.domain.local>
Apr 10 22:25:10 mail2 postfix/smtpd[342]: disconnect from mailcowdockerized_dovecot-mailcow_1.mailcowdockerized_mailcow-network[84.x.x.x] ehlo=1 mail=1 rcpt=1 data=0/1 quit=1 commands=4/5

Please let me know if you have further questions or concerns - or how I can improve this PR.

I’m not sure I want Dovecot to do the submission. How does that limit our possibilities to configure the service in Postfix?

You relay over port 25 back to postfix. You are always using an unencrypted connection. I know SOGo does the same, but one can avoid SOGo, while this would be active for all submissions. While I’m fine with this over sockets, we should avoid it for submission.

How about failed certificate requests for a single domain? What’s the fall back? Where are additional SANs appended to? How do additional SAN migrate (or where to)? What if someone decided previously to add an additional SAN to connect to his mail server? This name is put into the general certificate of the server, I guess?

I’m on mobile and haven’t checked yet, so I’m not sure if/how that’s added: What is the default host of each individual FQDN? Like “mail” in “mail.example.org”. Can people change that?

If customer A asks for “mymx” and the other asks for “mail01”, it will probably fall back to the general certificate and they will see each other’s name in the certificate, right? I cannot add additional SANs per domain.

Wouldn’t it be easier and less hassle to just place a HAProxy in front of mailcow? :/

I’m not sure I want Dovecot to do the submission. How does that limit our possibilities to configure the service in Postfix?

I don't think it limits the possibilities with Postfix. The only thing i can think of that Postfix does not know about is the client connection encryption. Are there any rules regarding incoming connection encryption?
For authentication I believe Postfix was handing it off to Dovecot already. Now Dovecot will do the authentication directly before acting as a relay (on Port 465 and 587).

You relay over port 25 back to postfix. You are always using an unencrypted connection. I know SOGo does the same, but one can avoid SOGo, while this would be active for all submissions. While I’m fine with this over sockets, we should avoid it for submission.

Yes, I am using Port 25 to avoid an additional tls overhead for each connection. We could change it to SSL or STARTTLS, but why do you think it is important in a container to container connection?

How about failed certificate requests for a single domain? What’s the fall back?

The same as before: The container will retry after 30 minutes instead of 1 day. Other certificates that were successful will still be activated immediately.

Where are additional SANs appended to? How do additional SAN migrate (or where to)? What if someone decided previously to add an additional SAN to connect to his mail server? This name is put into the general certificate of the server, I guess?

Yes. I am still using the same config value and domain variables:

  IFS=',' read -r -a TMP_ARR <<< "${ADDITIONAL_SAN}"
  for i in "${TMP_ARR[@]}" ; do
    if [[ "$i" =~ \.\*$ ]]; then
      ADDITIONAL_WC_ARR+=(${i::-2})
    else
      ADDITIONAL_SAN_ARR+=($i)
    fi
  done
ADDITIONAL_WC_ARR+=('autodiscover')

The only difference is that ADDITIONAL_WC_ARR will be used to create individual certificates, while ADDITIONAL_SAN_ARR will be appended to the default server certificate.

I’m on mobile and haven’t checked yet, so I’m not sure if/how that’s added: What is the default host of each individual FQDN? Like “mail” in “mail.example.org”. Can people change that?

The default host will be the first domain.* available, in the order defined in the existing ADDITIONAL_SAN config. To change it one has to change the order in the ADDITIONAL_SAN config and it will be applied to all domains.
I did not add any more per default, so in the standard configuration it will be autodiscover.example.org (and nothing else).

If customer A asks for “mymx” and the other asks for “mail01”, it will probably fall back to the general certificate and they will see each other’s name in the certificate, right? I cannot add additional SANs per domain.

Yes this is true for this implementation, except if you apply the subdomain to every domain with mymx.* and mail01.*
I did not expect to have individual SAN requirements per customer. So this will still be limited to 99 additional fully qualified SANs. We could probably try to map each one to the respective autodiscover.* certificate if this is desired?

Wouldn’t it be easier and less hassle to just place a HAProxy in front of mailcow? :/

The point of this PR was to make this work out of the box, without needing to configure and maintain an additional reverse proxy / loadbalancer, since all required tools are already included.
In my opinion, these changes are quite simple, but I have also never configured a HAProxy.

Thank you for your feedback!

Are there any rules regarding incoming connection encryption?

Yes, you can require incoming TLS per user. It's not a commonly used feature of Mailcow because some mail servers still don't deliver over TLS, but in theory it's a nice frature that shouldn't be broken.

Does this apply for submission by users or only incoming mail by other providers? If it's only the latter it should not be broken as port 25 is still directly mapped to postfix.

TBH: I've no clue about the real advantages and if this is really necessary. In general I don't like the idea that Postfix is now depending on Dovecot to work.

However. I'm not a fan of the migration over to certbot:

1. certbot is still not capable of issuing ECDSA certificates. Breaking Add ECDSA support #2443 completely.
2. certbot is so much bigger than ACME with so many python libraries and basically does the same thing.

The reason i choose certbot is because it simplifies renewal and cert management, is well documentend, is the recommended acme client and offers a useful plugin and hooks system.

The image size difference with certbot is 19MB and I personally prefer a long term reliable solution over a few megabytes.
While ECDSA is indeed missing, it appears to be in implementation with backwards compatibility in mind. But if this is the only reason preventing an approval of this PR, I'm willing to add the required implementation with a different client.

I'd like to know more about the reservations regarding the Dovecot submission proxy.

• Postfix already depends on Dovecot for authentication.
• Unauthenticated delivery is still directly provided via port 25.
• Dovecot uses the same authentication methods as with IMAP and POP3.
• The XCLIENT extension appears to works well and basically makes Dovecot a transparent proxy.
• And it even adds some possible features like BURL.

For me the Dovecot proxy is the natural choice in favor of additional software.

Personally I'd require this feature for a seamless migration off cPanel and the confidence I don't need to create new instances or manage additional proxies when reaching a certain amount of domains. Looking at the previous issues I'd say there are also other people who would benefit from this.

I'd like to know more about the reservations regarding the Dovecot submission proxy.

I'm actually not too concerned about it. It just needs extensive testing etc. and we need to check whether it breaks any Postfix feature we use or might use in the future. As @andryyy said,

How does that limit our possibilities to configure the service in Postfix?

If XCLIENT takes care of everything, it is probably ok.

One thing I just thought of: you need to make sure that the same peivate key is used for the certificates as before. Otherwise existing TLSA records become invalid. Does your pull request generate a new key for each domain? Does it continue to use the key generated by the old acme client?

OK this is a really good point. My PR will create a new private key for each domain and certbot will create a new one on each renewal (but that can be changed). If all the existing TLSA records must be re-used, we'd have to use the same private key for each domain and keep it forever. This definitely requires some additional changes for this PR. I'm thinking about some compatibility mode which is enabled on upgrades but disabled on new installations, because I don't think it is a good practice to re-use the private key for multiple certs (or even for a longer duration). I'll try to solve that as soon as I can.

Thanks for all your work on this @mhofer117, I hope your changes are able to be merged at some point, I'm one of those people running a virtual server just for hosting auto*.* reverse proxies.

For what it is worth I generally use acme.sh rather than certbot, it is written in Bash and does support ECC certs.

I have reverted to use acme-tiny and using the existing private key for all certificates, so existing TLSA will still be valid for all domains. I also changed the PR description to match the changes.
This should also make it easy to integrate with #2443, if you want I can already merge those changes into this PR?

Let me know if you see any other possible issues, for exemple (as already mentioned) regarding Postfix features we use or might use in the future that could be affected by the dovecort proxy.

This won't break SKIP_LETS_ENCRYPT=y with custom certificates, correct?

Yes it still works as before

Because the ECDSA PR #2443 looks complete and will probably be merged I have merged it into this PR too and made all required changes for SNI with both RSA and ECDSA certificates.

Let me know if something is unclear or needs to be changed.
If everything is OK and you consider mergin I'll add the required documentation changes.

I don't know if this is a merger tbh.

I use custom submission configs quite often. That's to much of a hassle for the benefit in return.

What type of custom submission configs? What would prevent you from still using them?

Any updates on this? I would love to have these features added.

I will not use Dovecot submission.

It looks like this is a fix for the issue of "DNS setup issues for individual domains-> breaking many maildomains".
I host the mail for some domains where the DNS is not under my control. And if things go wrong there, it spoils the CertificateRenewal for the other domains until i find time to look into the logs and disable stuff manually.

This is true but in this case it would be explicitly defined and the admin can remove them ADDITIONAL_SAN to prevent this. Otherwise there is no option to include a domain explicitly in the default certificate.

Think about that: How do I add an FQDN explicitly only for one domain? Example, i got want mail.example.com but don't want expose it via default cert and don't want mail.example2.com.
I think this is the best solution:

Please add a check whether any of the automatically-generated autoconfig.example.com/autodiscover.example.com are present in ADDITIONAL_SAN and remove them from ADDITIONAL_SAN if found. Once that's done, I think this pull request should be merged.

Or did i miss something here that this is not possible?

Yes adding an FQDN only for one domain will always include it in the default certificate, the autogenerated certificates only contain entries for all domains (autoconfig.*, autodiscover.*, mail.*, ...). It would be quite complex to check every ADDITIONAL_SAN for matches against the DB and include them in the respective certificate (which could still be different than what the user actually wants).

I have added the suggested check / fix.

Ok, looks good to me. @andryyy, you can merge if you agree.

We just need to remember that we cannot easily revert this pull request. Files would need to be moved back to their original location etc. So we will need to fix any bugs that may arise. I don‘t expect anything major to come up though (I‘ve tested it and @mhofer117 has done a solid job) and it‘s definitely a worthwhile improvement.

Yes, looks good. I'm fine with it.

I think you also need to rebuild postfix. 0efa5e5 only updates acme and dovecot.

Thanks

Thanks everyone for the feedback and recommendations!

I'll follow the telegram chat and github to assist on any issues.

Awesome, thanks guys! I'll update my installation next weekend.

Looks good!

Only thing I noticed is that this configuration does no longer work. Also tried commenting out the lines in listen_plain.active, but it gets reverted when restarting nginx.

Did you pull the latest changes? There was a fix for this some hours ago.

I also recommend not changing the site.conf but instead create a new file for this, for example: data/conf/nginx/redirect.conf

This prevents git conflicts

Well, if that's broken and everyone needs to apply a change to site.conf to keep the redirect, that's not good.

Your recent change should not break it, right? :)

We should change the docs to redir.conf anyway, no matter if it works now, that's true.

@mhofer117 Yes I did. The redirect.conf file doesn't work either :(

Your recent change should not break it, right? :)

No should not. I think that documentation page is misleading anyway. That config would also redirect acme challenges? And it includes the same domain names as the integrated server directive, so nginx will complain about duplicates. I think all of that was the case also before this PR.

I have updated the docs here: mailcow/mailcow-dockerized-docs#175

@mhofer117 Yes I did. The redirect.conf file doesn't work either :(

Can you try the config from the documentation PR I linked?
Was it working before the TLS-SNI update?

@mhofer117 Just tested it, doesn't work. Pretty sure it worked before the TLS-SNI update, at least a couple of weeks ago it did.

Hmm I think i know why, let me test and provide a fix.

I think the following PR fixes this:
#3071

There is no reason why the TLS-SNI servers should also listen on http-80.
So now any server with listen 80 default_server; will serve all domain name again.

Andre merged already, can you retest @Geitenijs

@mhofer117 Great! It seems to work only for the mail. subdomains etc, but not for the server hostname (mx.example.com)

@andryyy Shouldn't we just disable HTTP altogether by default? Always redirect to HTTPS, unless the server admin prefers to have HTTP support

Check the latest change from andre:
https://mailcow.github.io/mailcow-dockerized-docs/u_e-80_to_443/

(server_name directive)

@mhofer117 Sorry, missed that one. Works perfectly now, thanks!

Has someone tested if the SNI extension works with wilcard domains (external acme client)?

SNI is unrelated to wildcard certificates.
When using an external acme client:

• You should not ENABLE_SSL_SNI, except if your external acme client creates more than one certificate with the same folder structure and files as the mailcow-acme container.
• If your wildcard certificate is at the default certificate location, it will just work as before and be the only certificate (without SNI).

Please elaborate some more if you have a specific setup / question?

Under data/assets/ssl/ a folder is created for each domain (e.g. mail.example.net), in this folder you will find cert.pem, key.pem and a domains file (e.g. mail.example.net).
For the main domain the files data/assets/ssl/cert.pem and data/assets/ssl/key.pem are created in addition.
ENABLE_SSL_SNI does not need to be activated with an external acme client, since this variable is only evaluated by the acme container.

What I currently don't understand is how to name the domain folder and what I write to the domains file when I use a wildcard domain.

According to Postfix instructions you should use .example.net, so I would name the folder "example.net" and write ".example.net" into the domains file.
Whether Dovecot and nginx understand this is not clear to me at the moment, so I simply asked.

You are right, ENABLE_SSL_SNI is not relevant.
The folder name at data/assets/ssl/* is not important, you can name it anything.
The data/assets/ssl/*/domains file is important, however I could not find the correct syntax for postfix.
For dovecot you need example.net .example.net inside the domains file.
nginx has a different syntax and requires *.example.net.
Not sure if it works if you combine it like example.net .example.net *.example.net or if any of the services will complain then...
Can you try it out and report back?

so my test results:

Postfix

echo "Q" | openssl s_client -starttls smtp -crlf -connect mail.example.net:587 -servername mail.example.net|grep subject
-> example.net in domain file -> main domain
-> .example.net in domain file -> sni domain
-> *.example.net in domain file -> main domain
INFO: http://www.postfix.org/postconf.5.html, section tls_server_sni_maps -> use ".example.net" for wildcard domains

Dovecot

echo "Q" | openssl s_client -showcerts -connect mail.example.net:993 -servername mail.example.net |grep subject
-> example.net in domain file -> main domain
-> .example.net in domain file -> main domain
-> *.example.net in domain file -> sni domain
INFO: No wildcard example found

Nginx (test port 8443)

echo "Q" | openssl s_client -connect mail.example.net:8443 -servername mail.example.net |grep subject
-> example.net in domain file -> main domain
-> .example.net in domain file -> sni domain
-> *.example.net in domain file -> sni domain
INFO: https://nginx.org/en/docs/http/server_names.html, section Optimization
.example.net = *.example.net and example.net
*.example.net = .example.net without example.net

so an adjustment in Dovecot script (Dockerfiles/dovecot/docker-entrypoint.sh) would be necessary (rewrite .example.net -> *.example.net) or you use *.example.net and adapt nginx and Postfix.

Whatever you change. Make sure it does not break anything that works now.