Add aliases to Dovecot's userdb & Adjust the passdb / userdb configuration#2248
Add aliases to Dovecot's userdb & Adjust the passdb / userdb configuration#2248polarathene merged 11 commits intomasterfrom
Conversation
georglauterbach
added
priority/medium
service/dovecot
area/scripts
kind/improvement
|
@docker-mailserver/maintainers tests pass, all seems well. Not sure why my local systems reported errors. Do I need extra software to run tests these days (like |
I looked over this PR earlier today, I didn't provide an approval as I don't know the changes well enough to properly review, but nothing stands out that concerns me. I can provide an approval, just giving time for others to review (we're about to hit the weekend, probably no rush). |
|
Alright:) Take your time, no rush to merge this PR. If you feel like giving an approval, do so, if not, don't :) |
|
@casperklein all of your suggestions are valid, I just didn't have the time yet. Will tend to it ASAP. |
|
IIRC there was a bat test in another issue, which could reproduce the problem (cannot find it right now 😞). It might be a good idea to add the test for this? @andrewlow Do you have a working bats test, to reproduce the issue that this PR is going to fix? |
|
@casperklein No - I'm sorry - the sum of my .bat test efforts were captured in the issue: #2091 (comment) Unfortunately my unfamiliarity with the test structure and the alias problem at a technical level doesn't give me confidence that my test is useful. |
I think there's enough info for me to take a stab at it, I might have time to in the weekend. |
|
FWIW - there are some existing tests for aliases in the testing framework. I was trying to create something specific to detect the dovecot error saying "I don't know who this alias is" - then later once the alias is mapped - the mail is delivered. |
That's be great! I will see whether final code cleanup is needed after all the new comments :) |
ReproductionBased on the test example from @andrewlow . Not presently adapted to a bats test. # Clone PR branch
git clone https://github.com/docker-mailserver/docker-mailserver --branch dovecot-aliases --recurse-submodules
# Build image:
cd docker-mailserver
make build
# Create the accounts and duplicate config directory for container to use:
make generate-accounts
mkdir -p test/duplicate_configs/x_alias/
rsync -a test/config/ test/duplicate_configs/x_alias/
# Initialize the test container:
docker run --rm -itd --name "alias_test" \
--hostname "mail.my-domain.com" \
--volume "${PWD}/test/test-files:/tmp/docker-mailserver-test:ro" \
--volume "${PWD}/test/duplicate_configs/x_alias:/tmp/docker-mailserver" \
"mailserver-testing:ci"
# Send mail to local alias `[email protected]` from external `[email protected]`
docker exec alias_test sh -c 'nc 0.0.0.0 25 < /tmp/docker-mailserver-test/email-templates/existing-alias-local.txt'
# Check for failure (result is 1):
docker logs alias_test | grep 'dovecot: auth: passwd-file([email protected]): unknown user' | wc -l
# 1Info
Observations
Would it be right to assume that Postfix delegated to Dovecot for auth to check if the alias had an account/mailbox managed by At least that is the impression I got, that no authentication was being attempted/failed, just checking with Dovecot if the user was managed by the mailserver. So no value in making the above an actual test? I'd need to modify the test method to have a local account for the alias, and possibly send from an IP that isn't considered local to EDIT: Regarding remote/local SMTP client connecting to Postfix, I assume the Dovecot auth part is what these Postfix SASL docs are about? |
Asked myself the same question. I think would be a good thing to do. |
Please see my reference notes comment here, specifically the larger quote block about what the This happens before Postfix hands anything off to Dovecot, and before it accepts mail. It is using the So the quota check is happening at the correct point in time, it's just If you do not do this, quotas can be bypassed via aliases. The easier workaround is to create the dummy Dovecot accounts that share the intended real user account data, but I agree that's not ideal. One approach that can be done that Postfix docs advise is to have the public MTA on port 25 do all the usual filtering to reject bad mail, and then forward that to a 2nd internal instance to receive where IIRC the alias would have been expanded, effectively relaying the mail.
Without quotas, you cannot limit mail storage capacity but I guess the backscatter/blacklisting concern no longer applies. I'm not sure what happens when you run out of actual disk storage space, but I guess that's more of a sysadmin concern to avoid in the first place? Given the ENV name, it sort of implied to me that it was something you'd need to enable explicitly, not disable 😅 I haven't looked at our other similar named ENV, perhaps that's been the convention. I can understand the value in the feature, but as this failure would have been silent and our workaround not ideal, probably best to make it opt-in as a breaking change release note (_at least I consider changing default ENV like this breaking changes, I think we didn't with Docs should be clear about the workaround with alias support though?
No.. that is just adding it to a complimentary ENV file. I don't ever user that myself.
So it's always been default since the introduction in Apr 2020 (v8 release). I'm fine with making it opt-in instead. That should be handled via separate PR however. |
|
|
||
| # alias is assumed to not be a proper e-mail | ||
| # these aliases do not need to be added to Dovecot's userdb | ||
| [[ ! ${ALIAS} == *@* ]] && continue |
There was a problem hiding this comment.
I am wondering, why non "FQUN" aliases like amavis don't need to be present in Dovecots userdb. Could you explain, thx.
Edit:
@polarathene Or maybe you can answer this too
There was a problem hiding this comment.
This problem only applies to inbound mail from external senders AFAIK.
I would need to check if smtpd_recipient_restrictions apply or not when a local service in the container sends mail to Postfix to deliver to Dovecot.
Even if it does, we have some rules earlier in these restrictions such as permit_mynetworks which will approve accepting the mail earlier. This prevents checking the quota policy against Dovecot. That'd probably not be great usually, but when the mail originates from you as the sender, then the backscatter concern of bouncing mail back to yourself probably doesn't apply?
I am not familiar with amavis other than it having something to do with spam handling. If it's not a concern, and the alias maps to an account that Dovecot receives mail for, and that quota is exceeded, then Dovecot bounces the mail (which from descriptions I read, is repeated multiple times 🤷♂️ )
There was a problem hiding this comment.
Thank you for clarification. FYI, I use the amavis alias because of this.
That's why I have something like this: amavis [email protected] in postfix-virtual.cf.
There was a problem hiding this comment.
As long as [email protected] is not a real address (in postfix-accounts.cf), this PR will not change / add anything because (as @polarathene has pointed out) only 1-to-1 aliases (alias -> real user) are expandend. Should all be fine here then?
There was a problem hiding this comment.
I thought the whole point of this PR is to avoid some nasty error messages? So in any case, this PR should not harm in any way. Or do I miss something fundamental at the moment?
[email protected] is a real address (in postfix-accounts.cf). Why it shouldn't?
There was a problem hiding this comment.
[email protected] is a real address (in postfix-accounts.cf). Why it shouldn't?
This was just an assumption (in this case, an entry will be added to Dovecot's userdb). This PR (should) does not do harm in any way, of course.
There was a problem hiding this comment.
Got it. But then, when someone have something like this in postfix-virtual.cf, this would not add an entry to dovecots userdb?
amavis [email protected] in postfix-virtual.cf
PS: I don't know, if it's valid to point an alias to an other alias.
There was a problem hiding this comment.
That's why I have something like this:
amavis [email protected]inpostfix-virtual.cf.
I believe that should be in postfix-aliases.cf (parameter main.cf:alias_maps) like the comment you linked to mentions with aliasing the amavis account to root (which can alias again to postmaster@<FQDN>.
postfix-virtual.cf (parameter main.cf:virtual_alias_map with generated file /etc/postfix/virtual) and postfix-accounts.cf (parameter main.cf:virtual_mailbox_maps with generated file /etc/postfix/vmailbox) provide unique domains for the generated file /etc/postfix/vhost (_parameter main.cf:virtual_mailbox_domains).
Postfix should receive mail from local account without any FQDN, eg from amavis and match it to an alias in hash:/etc/aliases (/etc/aliases is plain-text, hash database generated by postalias is /etc/aliases.db that Postfix uses, values are sourced from postfix-aliases.cf populating /etc/aliases at startup or change event). Mappings in /etc/aliases lack an FQDN, but Postfix will under some conditions receive mail and extract the local part (eg left side of @) from incoming mail recipient and lookup these /etc/aliases if no explicit match was found in virtual aliases.
Since postfix-aliases.cf (unix user aliases) is purely meant for these aliases, while postfix-accounts.cf (virtual mailboxes, storage not tied to unix user) and postfix-virtual.cf (virtual aliases, no unix user) are handled by us differently (contributing to vhost domains), I think you would place amavis and the like in postfix-aliases.cf 😅 (don't worry, I'm probably as confused as you are right now 😛 )
Got it. But then, when someone have something like this in
postfix-virtual.cf, this would not add an entry to dovecots userdb?
Postfix address rewriting docs on virtual aliases shows an example of a 1-to-1 mapping in postfix-virtual.cf that's got no associated FQDN, so even I'm confused about it vs the postfix-alias.cf kind. This is perfectly valid as you point out..
Dovecot AFAIK doesn't care what the user name is, eg amavis or [email protected]? What I believe happens here is Postfix will append main.cf:myorigin or if not available another fallback domain it's aware of, and then deliver that to Dovecot to store in /var/mail.
In the case of Amavis, it should only be mail to and from your server, if Dovecot can't be configured to ignore quota for system user accounts (eg: [email protected]) you could make the quota really high. In regards to the issue this PR is solving, there are "permit" rules that can come before the quota policy check, eg mynetworks and the one for SASL authenticated is probably safe and both of these are configured to come before the quota check, thus Postfix will accept and if Dovecot rejects, it bounces but if the accounts that sent it aren't external, the backscatter blacklisting concern shouldn't apply?
PS: I don't know, if it's valid to point an alias to an other alias.
I have read of an all alias that expands to many users, some which I guess could be aliases. 1-to-1 alias that maps to another alias seems a bit silly, but our docs have an example of working around SPOOF_PROTECTION issue for aliasing to gmail via an alias to alias mapping which then maps the 2nd level alias to the external account.
Probably good to eventually investigate further the above for definitive answers about account/alias handling and document that somewhere, but I don't have time to allocate to that right now 😬
There was a problem hiding this comment.
Just to clarify, the log issue with Postfix asking Dovecot for quota status of an alias that Dovecot doesn't know about is always the RCPT TO (envelope recipient) address the email was originally sent to.
Regardless of nested chains, Postfix does not expand that aliases at this point (which could be N levels deep or expand to multiple addresses to send to). In the docs mention in my previous comment, that means the gmail account with double aliases won't be added to Dovecot, not that it matters since that's aliasing to an external account anyway.
If it ever turns out to be an issue, we can probably regard it as niche and unsupported. Our responsibility would just be to make the caveat with quota support with aliases visible in our docs, and that we only assist with direct aliasing to a single real mailbox account, any fancy aliasing beyond that puts the user as risk of backscatter if they have quotas enabled.
There was a problem hiding this comment.
don't worry, I'm probably as confused as you are right now
😆
Thank you for you detailed answer!
From a user perspective, the complimentary ENV file defines the defaults.
I've run my previous mail server (also postfix/dovecot but not docker-mailserver based) out of storage - things fail, but I'd say it's graceful enough. If you reach in and fix the storage problem mail resumes flowing. For the most part - sending servers simply fail to deliver, and the receiving server doesn't succeed in receiving the email and sends an error code - so if you address the storage issue soon enough no mail is bounced because the sending server will re-try a few times. |
yep, still using it!
Definitly a breaking change imho
But making it opt-in doesn't solve the problem, does it? The people that enable quotas would still face this behavior? Or am i missunderstanding something (admitting not reading all 55 comments and linked issues/pr comments in detail)? |
The "problem" here is extra entries in the log files - where for every alias, you will see an error indicating that dovecot cannot find the alias user when it is trying to check the quota. This problem is invisible, until you enable the optional logwatch reporting https://docker-mailserver.github.io/docker-mailserver/edge/config/environment/#logwatch_interval Logwatch is detecting these log lines about the alias as an error, and reporting. It makes the logwatch report quite difficult to read if you have a lot of traffic to aliases. |
|
Thanks for clarifying! |
@wernerfred this comment should cover everything you need to know, if not no worries let me know and I'll clarify/expand further 👍
No it is not. This affects anyone using aliases and the default quota feature for Dovecot. It risks making the That can get you blacklisted. The quota feature itself is useful to enforce storage limitations for mail, or per user to avoid any abuse that could fill up the mail server storage, and thus no longer having space for others to store received mail. If you don't need that, disabling quota support avoids the backscatter issue with aliases (still requires quota to actually be exceeded though). The log message itself is only because Postfix is checking with Dovecot with the alias account in the
With this PR they would not see the log lines. The workaround is to add the alias account as a dummy account to Dovecot which shares the same mailbox of the real account with a quota. The workaround is probably a bit naive though, in that:
Due to that, this workaround is naive in that it will only resolve simple alias mappings that map an alias to a single real account directly AFAIK. The proper fix would be to not have Dovecot care about these dummy accounts at all, and write a policy service for Postfix to call that resolves the alias fully and checks any mapped accounts to the Dovecot The quota support is also not implemented by Alternatively we could implement quota support for LDAP too, but I don't think anyone here is interested in tackling that 😅 SummaryIf you use quotasThe problem
The workaround fix
If you don't use quotas
TL;DR
|
@andrewlow the complimentary ENV file is it's own set of defaults. Not all users add that. I don't expect users to know about internals, but for public facing ENV defaults, our ENV documentation should be a more reliable source for users to know what is default. I would like to better automate genating that ENV file from our docs to keep them in sync though. The source you provided was about updating that ENV file, while the actual quota support had an implicit default not "documented" in that ENV file for quite some time prior... an entire year.
It becomes a real problem when quota is exceeded. See my previous comment for details.
The issue probably still needs to be documented either way, especially the caveat (only basic alias mappings) for the workaround proposed. A breaking change doesn't have to be introduced straight away, I've asked for a separate PR for any default ENV change. This way it can be merged at a later date as there are other changes inbound unrelated to this PR that are likely to warrant a major version bump. |
Insane, thank you! So what i am not understanding right now is how |
Where are you referring to this from? I recall that being part of the quota feature PR, but not this one. Postfix I've not looked into that personally so I don't quite know what happens there, presumably it relays over LMTP to Dovecot, if so then either it bypasses the quota check and could be an issue, or by relaying via LMTP it may go through the proper checks again on the recipient, however by this time Postfix probably already expanded the alias via UPDATE: This is a docker-mailserver/target/scripts/startup/setup-stack.sh Lines 517 to 527 in fb72f3a
When receiving mail, presumably Postfix has or had some notion of the mailbox size (we may have shifted that to Dovecot, so I'm not sure if that's true anymore), and Postfix could look at the size of the incoming mail and reject it if it would exceed the mailbox limit or individual mail size limit that's been configured. In that sense it's similar to quotas I think, but no issue here in that it has nothing to do with Dovecot (unclear regarding mailbox limit associated to recipient). Some tests and complimentary documentation would probably be good about the above 😅 |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
6 participants
Description
See #2091 (comment).
Fixes #2091
Type of change
Checklist:
docs/)