Bug Report
Context
Firstly thanks for a great container. The email software stack must be one of the most complex to implement !
I notice that when calling addmailuser it becomes very slow when number of mailboxes increases.
I did not notice slowness too much when using non-NFS e.g. local HDD or AWS EBS storage
Now I am using NFS (actually AWS EFS) storage since deploying as docker in AWS ECS.
For example I have 2500 mailboxes and a call to addmailuser takes 60 seconds.
I have removed some old inactive mailboxes and now have 500 mailboxes but a call to addmailuser now still takes 30 seconds. Seems too slow to add a single user.
I think the slowness is in the design of check-for-changes.sh. It detects change in postfix-accounts.cf and then loops through all lines in this file, trying to add each user or update its password for each line. This may give good consistency of user data across dovecot etc... but is very inefficient. If it saved a copy of the last successful processing of file postfix-accounts.cf, then it could diff the new file against that copy and detect only one line is new and then create the new user. That would probably be 0.5 secs or less.
There is a slight risk that the copy of postfix-accounts.cf becomes incorrect, through script crashing. To help fix this the start on the container could process the entire postfix-accounts.cf (current behaviour) to fix any config "drift".
Arguably this is not a Bug but a Feature Request.
What is affected by this bug?
Speed of addmailuser command.
When does this occur?
When number of mailboxes is high (>=500) and storage is NFS.
How do we replicate the issue?
- NFS storage
- create > 500 mailboxes
- time execution of addmailuser
Behavior
Actual Behavior
Expected Behavior
Your Environment
Using latest EDGE docker image from dockerhub.
AWS ECS docker on EC2 nodes.
EFS storage from all volumes e.g. /var/mail and /tmp/docker-mailserver
Environment Variables
From a docker ECS service spec :
{ "name" : "SSL_TYPE", "value" : "manual" },
{ "name" : "SSL_CERT_PATH", "value" : "/tmp/ssl/mail.qa-mail.tradingapps.com.crt" },
{ "name" : "SSL_KEY_PATH", "value" : "/tmp/ssl/mail.qa-mail.tradingapps.com.key" },
{ "name" : "DMS_DEBUG", "value" : "0" },
{ "name" : "ENABLE_CLAMAV", "value" : "0" },
{ "name" : "ONE_DIR", "value" : "1" },
{ "name" : "ENABLE_POP3", "value" : "1" },
{ "name" : "ENABLE_FAIL2BAN", "value" : "0" },
{ "name" : "ENABLE_MANAGESIEVE", "value" : "" },
{ "name" : "OVERRIDE_HOSTNAME", "value" : "mail.qa-mail.tradingapps.com" },
{ "name" : "POSTMASTER_ADDRESS", "value" : "" },
{ "name" : "POSTSCREEN_ACTION", "value" : "enforce" },
{ "name" : "REPORT_RECIPIENT", "value" : "0" },
{ "name" : "REPORT_INTERVAL", "value" : "daily" },
{ "name" : "SMTP_ONLY", "value" : "" },
{ "name" : "TLS_LEVEL", "value" : "" },
{ "name" : "SPOOF_PROTECTION", "value" : "" },
{ "name" : "ENABLE_SRS", "value" : "0" },
{ "name" : "PERMIT_DOCKER", "value" : "network" },
{ "name" : "VIRUSMAILS_DELETE_DELAY", "value" : "" },
{ "name" : "ENABLE_POSTFIX_VIRTUAL_TRANSPORT", "value" : "" },
{ "name" : "POSTFIX_DAGENT", "value" : "" },
{ "name" : "ENABLE_SPAMASSASSIN", "value" : "0" },
{ "name" : "SA_TAG", "value" : "2.0" },
{ "name" : "SA_TAG2", "value" : "6.31" },
{ "name" : "SA_KILL", "value" : "6.31" },
{ "name" : "SA_SPAM_SUBJECT", "value" : "***SPAM*****" },
{ "name" : "ENABLE_FETCHMAIL", "value" : "0" },
{ "name" : "FETCHMAIL_POLL", "value" : "300" },
{ "name" : "ENABLE_LDAP", "value" : "" },
{ "name" : "LDAP_START_TLS", "value" : "" },
{ "name" : "LDAP_SERVER_HOST", "value" : "" },
{ "name" : "LDAP_SEARCH_BASE", "value" : "" },
{ "name" : "LDAP_BIND_DN", "value" : "" },
{ "name" : "LDAP_BIND_PW", "value" : "" },
{ "name" : "LDAP_QUERY_FILTER_USER", "value" : "" },
{ "name" : "LDAP_QUERY_FILTER_GROUP", "value" : "" },
{ "name" : "LDAP_QUERY_FILTER_ALIAS", "value" : "" },
{ "name" : "DOVECOT_TLS", "value" : "" },
{ "name" : "DOVECOT_USER_FILTER", "value" : "" },
{ "name" : "DOVECOT_PASS_FILTER", "value" : "" },
{ "name" : "ENABLE_POSTGREY", "value" : "0" },
{ "name" : "POSTGREY_DELAY", "value" : "300" },
{ "name" : "POSTGREY_MAX_AGE", "value" : "35" },
{ "name" : "POSTGREY_TEXT", "value" : "Delayed by postgrey" },
{ "name" : "ENABLE_SASLAUTHD", "value" : "" },
{ "name" : "SASLAUTHD_MECHANISMS", "value" : "1" },
{ "name" : "SASLAUTHD_MECH_OPTIONS", "value" : "rimap" },
{ "name" : "SASLAUTHD_LDAP_SERVER", "value" : "127.0.0.1" },
{ "name" : "SASLAUTHD_LDAP_SSL", "value" : "" },
{ "name" : "SASLAUTHD_LDAP_BIND_DN", "value" : "" },
{ "name" : "SASLAUTHD_LDAP_PASSWORD", "value" : "" },
{ "name" : "SASLAUTHD_LDAP_SEARCH_BASE", "value" : "" },
{ "name" : "SASLAUTHD_LDAP_FILTER", "value" : "" },
{ "name" : "SASL_PASSWD", "value" : "" },
{ "name" : "SRS_EXCLUDE_DOMAINS", "value" : "" },
{ "name" : "SRS_SECRET", "value" : "" },
{ "name" : "RELAY_HOST", "value" : "" },
{ "name" : "RELAY_PORT", "value" : "" },
{ "name" : "RELAY_USER", "value" : "" },
{ "name" : "RELAY_PASSWORD", "value" : "" }Relevant Stack Traces
None.
Bug Report
Context
Firstly thanks for a great container. The email software stack must be one of the most complex to implement !
I notice that when calling addmailuser it becomes very slow when number of mailboxes increases.
I did not notice slowness too much when using non-NFS e.g. local HDD or AWS EBS storage
Now I am using NFS (actually AWS EFS) storage since deploying as docker in AWS ECS.
For example I have 2500 mailboxes and a call to addmailuser takes 60 seconds.
I have removed some old inactive mailboxes and now have 500 mailboxes but a call to addmailuser now still takes 30 seconds. Seems too slow to add a single user.
I think the slowness is in the design of check-for-changes.sh. It detects change in postfix-accounts.cf and then loops through all lines in this file, trying to add each user or update its password for each line. This may give good consistency of user data across dovecot etc... but is very inefficient. If it saved a copy of the last successful processing of file postfix-accounts.cf, then it could diff the new file against that copy and detect only one line is new and then create the new user. That would probably be 0.5 secs or less.
There is a slight risk that the copy of postfix-accounts.cf becomes incorrect, through script crashing. To help fix this the start on the container could process the entire postfix-accounts.cf (current behaviour) to fix any config "drift".
Arguably this is not a Bug but a Feature Request.
What is affected by this bug?
Speed of addmailuser command.
When does this occur?
When number of mailboxes is high (>=500) and storage is NFS.
How do we replicate the issue?
Behavior
Actual Behavior
Expected Behavior
Your Environment
Using latest EDGE docker image from dockerhub.
AWS ECS docker on EC2 nodes.
EFS storage from all volumes e.g. /var/mail and /tmp/docker-mailserver
Environment Variables
From a docker ECS service spec :
{ "name" : "SSL_TYPE", "value" : "manual" }, { "name" : "SSL_CERT_PATH", "value" : "/tmp/ssl/mail.qa-mail.tradingapps.com.crt" }, { "name" : "SSL_KEY_PATH", "value" : "/tmp/ssl/mail.qa-mail.tradingapps.com.key" }, { "name" : "DMS_DEBUG", "value" : "0" }, { "name" : "ENABLE_CLAMAV", "value" : "0" }, { "name" : "ONE_DIR", "value" : "1" }, { "name" : "ENABLE_POP3", "value" : "1" }, { "name" : "ENABLE_FAIL2BAN", "value" : "0" }, { "name" : "ENABLE_MANAGESIEVE", "value" : "" }, { "name" : "OVERRIDE_HOSTNAME", "value" : "mail.qa-mail.tradingapps.com" }, { "name" : "POSTMASTER_ADDRESS", "value" : "" }, { "name" : "POSTSCREEN_ACTION", "value" : "enforce" }, { "name" : "REPORT_RECIPIENT", "value" : "0" }, { "name" : "REPORT_INTERVAL", "value" : "daily" }, { "name" : "SMTP_ONLY", "value" : "" }, { "name" : "TLS_LEVEL", "value" : "" }, { "name" : "SPOOF_PROTECTION", "value" : "" }, { "name" : "ENABLE_SRS", "value" : "0" }, { "name" : "PERMIT_DOCKER", "value" : "network" }, { "name" : "VIRUSMAILS_DELETE_DELAY", "value" : "" }, { "name" : "ENABLE_POSTFIX_VIRTUAL_TRANSPORT", "value" : "" }, { "name" : "POSTFIX_DAGENT", "value" : "" }, { "name" : "ENABLE_SPAMASSASSIN", "value" : "0" }, { "name" : "SA_TAG", "value" : "2.0" }, { "name" : "SA_TAG2", "value" : "6.31" }, { "name" : "SA_KILL", "value" : "6.31" }, { "name" : "SA_SPAM_SUBJECT", "value" : "***SPAM*****" }, { "name" : "ENABLE_FETCHMAIL", "value" : "0" }, { "name" : "FETCHMAIL_POLL", "value" : "300" }, { "name" : "ENABLE_LDAP", "value" : "" }, { "name" : "LDAP_START_TLS", "value" : "" }, { "name" : "LDAP_SERVER_HOST", "value" : "" }, { "name" : "LDAP_SEARCH_BASE", "value" : "" }, { "name" : "LDAP_BIND_DN", "value" : "" }, { "name" : "LDAP_BIND_PW", "value" : "" }, { "name" : "LDAP_QUERY_FILTER_USER", "value" : "" }, { "name" : "LDAP_QUERY_FILTER_GROUP", "value" : "" }, { "name" : "LDAP_QUERY_FILTER_ALIAS", "value" : "" }, { "name" : "DOVECOT_TLS", "value" : "" }, { "name" : "DOVECOT_USER_FILTER", "value" : "" }, { "name" : "DOVECOT_PASS_FILTER", "value" : "" }, { "name" : "ENABLE_POSTGREY", "value" : "0" }, { "name" : "POSTGREY_DELAY", "value" : "300" }, { "name" : "POSTGREY_MAX_AGE", "value" : "35" }, { "name" : "POSTGREY_TEXT", "value" : "Delayed by postgrey" }, { "name" : "ENABLE_SASLAUTHD", "value" : "" }, { "name" : "SASLAUTHD_MECHANISMS", "value" : "1" }, { "name" : "SASLAUTHD_MECH_OPTIONS", "value" : "rimap" }, { "name" : "SASLAUTHD_LDAP_SERVER", "value" : "127.0.0.1" }, { "name" : "SASLAUTHD_LDAP_SSL", "value" : "" }, { "name" : "SASLAUTHD_LDAP_BIND_DN", "value" : "" }, { "name" : "SASLAUTHD_LDAP_PASSWORD", "value" : "" }, { "name" : "SASLAUTHD_LDAP_SEARCH_BASE", "value" : "" }, { "name" : "SASLAUTHD_LDAP_FILTER", "value" : "" }, { "name" : "SASL_PASSWD", "value" : "" }, { "name" : "SRS_EXCLUDE_DOMAINS", "value" : "" }, { "name" : "SRS_SECRET", "value" : "" }, { "name" : "RELAY_HOST", "value" : "" }, { "name" : "RELAY_PORT", "value" : "" }, { "name" : "RELAY_USER", "value" : "" }, { "name" : "RELAY_PASSWORD", "value" : "" }Relevant Stack Traces
None.