Skip to content

Kubernetes-Wiki: example with PROXY-protocol does not forward client-ip #954

Closed
Closed
Kubernetes-Wiki: example with PROXY-protocol does not forward client-ip#954
@mariusburfey

Description

@mariusburfey
mariusburfey
opened on Apr 18, 2018

Hi guys,
I'm deploying the mailserver into a kubernetes-cluster, but I got some problems: IMAP seems to work, but I can't send or receive emails.
The logs suggest that it might be a problem with PROXY-setting in nginx-ingress.

I used the wiki and some files from https://github.com/yuanying/k8s-env/tree/master/charts/mailserver to achieve the following:

  • IMAP seems to work.
  • I can telnet to ports 25, 141, 487 and 993.

What I did in terms of kubernetes (based on the wiki: https://github.com/tomav/docker-mailserver/wiki/Using-in-Kubernetes#proxy-port-to-service-via-proxy-protocol):

What I also did:

  • Enabled SSL with a self-signed certificate. I added the 5 files mentioned in the wiki to my kubernetes-configmap and deployment.

Now I was able to telnet to the container and to configure Thunderbird to use the mailserver, but I cannot send or receive mails.

My logfile is full of unsuccessful connections. I think they might come from my load balancer, which connects every 5 seconds on all 4 ports. Sadly I cannot deactivate them (Elastic Load Balance by Open Telekom Cloud, which is based on Open Stack).

Here's the startup log:

Initializing setup
  Registering check,setup,fix,misc and start-daemons functions
  * _check_environment_variables() registered
  * _check_hostname() registered
  * _setup_default_vars() registered
  * _setup_dovecot() registered
  * _setup_dovecot_local_user() registered
  * _setup_dkim() registered
  * _setup_ssl() registered
  * _setup_docker_permit() registered
  * _setup_mailname() registered
  * _setup_amavis() registered
  * _setup_dmarc_hostname() registered
  * _setup_postfix_hostname() registered
  * _setup_dovecot_hostname() registered
  * _setup_postfix_sasl() registered
  * _setup_postfix_override_configuration() registered
  * _setup_postfix_sasl_password() registered
  * _setup_security_stack() registered
  * _setup_postfix_aliases() registered
  * _setup_postfix_vhost() registered
  * _setup_postfix_dhparam() registered
  * _setup_postfix_postscreen() registered
  * _setup_postfix_access_control() registered
  * _setup_postfix_relay_hosts() registered
  * _setup_environment() registered
  * _setup_logrotate() registered
  * _fix_var_mail_permissions() registered
  * _fix_var_amavis_permissions() registered
  * _fix_cleanup_clamav() registered
  * _misc_save_states() registered
  * _start_daemons_cron() registered
  * _start_daemons_rsyslog() registered
  * _start_daemons_dovecot() registered
  * _start_daemons_opendkim() registered
  * _start_daemons_opendmarc() registered
  * _start_daemons_postfix() registered
  * _start_changedetector() registered
  * _start_daemons_amavis() registered
Checking configuration
  Check that there are no conflicts with env variables [_check_environment_variables]
  Check that hostname/domainname is provided or overidden (no default docker hostname/kubernetes) [_check_hostname]
  * Domain has been set to my-domain.com
  * Hostname has been set to mail.my-domain.com
Configuring mail server
  Setting up default variables [_setup_default_vars]
  * Set ENABLE_LDAP=0
  * Set OVERRIDE_HOSTNAME=mail.my-domain.com
  * Set DMS_DEBUG=1
  * Set ENABLE_CLAMAV=0
  * Set ENABLE_FAIL2BAN=0
  * Set FETCHMAIL_POLL=300
  * Set TLS_LEVEL=modern
  * Set ENABLE_SPAMASSASSIN=0
  * Set REPORT_RECIPIENT=0
  * Set POSTGREY_DELAY=300
  * Set POSTGREY_TEXT=Delayed by postgrey
  * Set ENABLE_POSTGREY=0
  * Set POSTGREY_MAX_AGE=35
  * Set [email protected]
  * Set ENABLE_FETCHMAIL=0
  * Set REPORT_INTERVAL=daily
  * Set ENABLE_MANAGESIEVE=0
  * Set VIRUSMAILS_DELETE_DELAY=7
  * Set ENABLE_SRS=0
  * Set POSTSCREEN_ACTION=ignore
  * Set ENABLE_POP3=0
  * Set ENABLE_SASLAUTHD=0
  * Set SMTP_ONLY=0
  * Set SPOOF_PROTECTION=0
  Setting up Dovecot
  Setting up Dovecot Local User
  * Checking file line endings
sed: cannot rename /tmp/docker-mailserver/sedD7Vkzc: Device or resource busy
  * Regenerating postfix user list
sed: cannot rename /tmp/docker-mailserver/sed4EUJsc: Device or resource busy
  * user 'mb' for domain 'my-domain.com' with password '********'
  Setting up DKIM
  * DKIM keys added for: my-domain.com-mail.key
  * Changing permissions on /etc/opendkim
  Setting up SSL
  * TLS configured with 'modern' ciphers
  * Adding mail.my-domain.com SSL certificate
  * SSL configured with 'self-signed' certificates
  Setting up PERMIT_DOCKER Option
  * Adding docker network in my networks
  Setting up Mailname
  * Creating /etc/mailname
  Setting up Amavis
  * Applying hostname to /etc/amavis/conf.d/05-node_id
  Setting up dmarc
  * Applying hostname to /etc/opendmarc.conf
  Applying hostname and domainname to Postfix
  * Applying hostname to /etc/postfix/main.cf
  Applying hostname to Dovecot
  * Applying hostname to /etc/dovecot/conf.d/15-lda.conf
  Setting up Postfix Override configuration
  * Loaded 'config/postfix-main.cf'
  * No extra postfix settings loaded because optional '/tmp/docker-mailserver/postfix-master.cf' not provided.
  * set the compatibility level to 2
  Setting up Postfix SASL Password
  * Warning: 'SASL_PASSWD' is not provided. /etc/postfix/sasl_passwd not created.
  Setting up Security Stack
  * Spamassassin is disabled. You can enable it with 'ENABLE_SPAMASSASSIN=1'
  * Clamav is disabled. You can enable it with 'ENABLE_CLAMAV=1'
  Setting up Postfix Aliases
  * Warning 'config/postfix-virtual.cf' is not provided. No mail alias/forward created.
  Setting up Postfix vhost
  Setting up Postfix dhparam
  * Use dhparams that was generated previously
  * Configuring postscreen
  * Configuring user access
  Setting up Postfix Relay Hosts
  * Setting up outgoing email relaying via out-cloud.mms.t-systems-service.com:25
  * No relay auth file found and no default set
chown: cannot access '/etc/postfix/sasl_passwd': No such file or directory
chmod: cannot access '/etc/postfix/sasl_passwd': No such file or directory
  * Adding relay mapping for my-domain.com
  Setting up /etc/environment
  * Setting up logrotate
  * Setting postfix summary interval to daily
  Checking /var/mail permissions
  * Permissions in /var/mail look OK
  Checking $amavis_state_dir permissions
  * Permissions in /var/mail-state/lib-amavis look OK
  Cleaning up disabled Clamav
Starting Misc
  * Consolidating all state onto /var/mail-state
  *   Destination /var/mail-state/spool-postfix exists, linking /var/spool/postfix to it
  *   Destination /var/mail-state/lib-postfix exists, linking /var/lib/postfix to it
  *   Destination /var/mail-state/lib-amavis exists, linking /var/lib/amavis to it
  *   Destination /var/mail-state/lib-clamav exists, linking /var/lib/clamav to it
  *   Destination /var/mail-state/lib-spamassassin exists, linking /var/lib/spamassassin to it
  *   Destination /var/mail-state/lib-fail2ban exists, linking /var/lib/fail2ban to it
  *   Destination /var/mail-state/lib-postgrey exists, linking /var/lib/postgrey to it
  *   Destination /var/mail-state/lib-dovecot exists, linking /var/lib/dovecot to it
  * Fixing /var/mail-state/* permissions
Starting mail server
  Starting cron2018-04-18 14:19:08,008 INFO spawned: 'cron' with pid 173
2018-04-18 14:19:08,008 INFO success: cron entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
cron: started
  Starting rsyslog 2018-04-18 14:19:08,208 INFO spawned: 'rsyslog' with pid 175
2018-04-18 14:19:08,209 INFO success: rsyslog entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
rsyslog: started
  Starting dovecot services2018-04-18 14:19:08,440 INFO spawned: 'dovecot' with pid 179
2018-04-18 14:19:08,441 INFO success: dovecot entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
dovecot: started
  Starting opendkim 2018-04-18 14:19:08,659 INFO spawned: 'opendkim' with pid 187
2018-04-18 14:19:08,660 INFO success: opendkim entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
opendkim: started
  Starting opendmarc 2018-04-18 14:19:08,875 INFO spawned: 'opendmarc' with pid 195
2018-04-18 14:19:08,875 INFO success: opendmarc entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
opendmarc: started
  Starting postfix2018-04-18 14:19:09,081 INFO spawned: 'postfix' with pid 202
2018-04-18 14:19:09,082 INFO success: postfix entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
postfix: started
  Starting changedetector2018-04-18 14:19:09,295 INFO spawned: 'changedetector' with pid 234
2018-04-18 14:19:09,296 INFO success: changedetector entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
changedetector: started
  Starting amavis2018-04-18 14:19:09,514 INFO spawned: 'amavis' with pid 251
2018-04-18 14:19:09,514 INFO success: amavis entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
amavis: started

#
# mail.my-domain.com is up and running
#

Logs when sending a mail:

Apr 18 14:23:52 mailserver-855855534-bfkp3 dovecot: imap-login: Disconnected (no auth attempts in 0 secs): user=<>, rip=172.18.91.5, lip=172.18.91.12, session=<lwhONCBqJqmsElsF>
Apr 18 14:23:52 mailserver-855855534-bfkp3 dovecot: imap-login: haproxy: Client disconnected (rip=172.18.91.5)
Apr 18 14:23:53 mailserver-855855534-bfkp3 postfix/postscreen[1029]: CONNECT from [172.18.91.5]:50701 to [172.18.91.12]:25
Apr 18 14:23:53 mailserver-855855534-bfkp3 postfix/postscreen[1029]: WHITELISTED [172.18.91.5]:50701
Apr 18 14:23:53 mailserver-855855534-bfkp3 postfix/smtpd[1652]: warning: ignoring non-empty smtpd_upstream_proxy_protocol setting behind postscreen
Apr 18 14:23:53 mailserver-855855534-bfkp3 postfix/smtpd[1652]: connect from unknown[172.18.91.5]
Apr 18 14:23:53 mailserver-855855534-bfkp3 opendmarc[195]: ignoring connection from [172.18.91.5]
Apr 18 14:23:53 mailserver-855855534-bfkp3 postfix/smtpd[1652]: lost connection after CONNECT from unknown[172.18.91.5]
Apr 18 14:23:53 mailserver-855855534-bfkp3 postfix/smtpd[1652]: disconnect from unknown[172.18.91.5] commands=0/0
Apr 18 14:23:55 mailserver-855855534-bfkp3 postfix/submission/smtpd[1649]: warning: haproxy read: unexpected EOF
Apr 18 14:23:55 mailserver-855855534-bfkp3 postfix/submission/smtpd[1649]: connect from unknown[unknown]
Apr 18 14:23:55 mailserver-855855534-bfkp3 postfix/submission/smtpd[1649]: disconnect from unknown[unknown] commands=0/0
Apr 18 14:23:56 mailserver-855855534-bfkp3 dovecot: imap-login: haproxy: Client disconnected (rip=172.18.91.5)
Apr 18 14:23:56 mailserver-855855534-bfkp3 dovecot: imap-login: Disconnected (disconnected before auth was ready, waited 0 secs): user=<>, rip=172.18.91.5, lip=172.18.91.12, session=<E2aINCBqMKmsElsF>
Apr 18 14:23:56 mailserver-855855534-bfkp3 postfix/postscreen[1029]: CONNECT from [172.18.91.5]:50707 to [172.18.91.12]:25
Apr 18 14:23:56 mailserver-855855534-bfkp3 postfix/postscreen[1029]: WHITELISTED [172.18.91.5]:50707
Apr 18 14:23:56 mailserver-855855534-bfkp3 postfix/smtpd[1652]: warning: ignoring non-empty smtpd_upstream_proxy_protocol setting behind postscreen
Apr 18 14:23:56 mailserver-855855534-bfkp3 postfix/smtpd[1652]: connect from unknown[172.18.91.5]
Apr 18 14:23:56 mailserver-855855534-bfkp3 opendmarc[195]: ignoring connection from [172.18.91.5]
Apr 18 14:23:56 mailserver-855855534-bfkp3 postfix/smtpd[1652]: lost connection after CONNECT from unknown[172.18.91.5]
Apr 18 14:23:56 mailserver-855855534-bfkp3 postfix/smtpd[1652]: disconnect from unknown[172.18.91.5] commands=0/0
Apr 18 14:23:56 mailserver-855855534-bfkp3 postfix/submission/smtpd[1649]: warning: haproxy read: unexpected EOF
Apr 18 14:23:56 mailserver-855855534-bfkp3 postfix/submission/smtpd[1649]: connect from unknown[unknown]
Apr 18 14:23:56 mailserver-855855534-bfkp3 postfix/submission/smtpd[1649]: disconnect from unknown[unknown] commands=0/0
Apr 18 14:23:57 mailserver-855855534-bfkp3 dovecot: imap-login: Disconnected (disconnected before auth was ready, waited 0 secs): user=<>, rip=172.18.91.5, lip=172.18.91.12, session=<M1aaNCBqNamsElsF>
Apr 18 14:23:57 mailserver-855855534-bfkp3 postfix/submission/smtpd[1950]: warning: haproxy read: timeout error
Apr 18 14:23:57 mailserver-855855534-bfkp3 postfix/submission/smtpd[1950]: connect from unknown[unknown]
Apr 18 14:23:57 mailserver-855855534-bfkp3 postfix/submission/smtpd[1950]: disconnect from unknown[unknown] commands=0/0
Apr 18 14:23:57 mailserver-855855534-bfkp3 dovecot: imap-login: haproxy: Client disconnected (rip=172.18.91.5)

Logs for incoming mail:

Apr 18 14:25:55 mailserver-855855534-bfkp3 postfix/smtpd[2513]: connect from unknown[172.18.91.5]
Apr 18 14:25:55 mailserver-855855534-bfkp3 opendmarc[195]: ignoring connection from [172.18.91.5]
Apr 18 14:25:55 mailserver-855855534-bfkp3 postfix/smtpd[2513]: Anonymous TLS connection established from unknown[172.18.91.5]: TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)
Apr 18 14:25:55 mailserver-855855534-bfkp3 postfix/trivial-rewrite[2945]: warning: inet_protocols: disabling IPv6 name/address support: Address family not supported by protocol
Apr 18 14:25:55 mailserver-855855534-bfkp3 dovecot: imap-login: haproxy: Client disconnected (rip=172.18.91.5)
Apr 18 14:25:55 mailserver-855855534-bfkp3 postfix/cleanup[2947]: warning: inet_protocols: disabling IPv6 name/address support: Address family not supported by protocol
Apr 18 14:25:55 mailserver-855855534-bfkp3 postfix/smtpd[2513]: 3C76A120227: client=unknown[172.18.91.5]
Apr 18 14:25:55 mailserver-855855534-bfkp3 postfix/cleanup[2947]: 3C76A120227: message-id=<[email protected]>
Apr 18 14:25:55 mailserver-855855534-bfkp3 opendkim[189]: 3C76A120227: no signing table match for '[email protected]'
Apr 18 14:25:55 mailserver-855855534-bfkp3 postfix/submission/smtpd[1950]: warning: haproxy read: unexpected EOF
Apr 18 14:25:55 mailserver-855855534-bfkp3 postfix/submission/smtpd[1950]: connect from unknown[unknown]
Apr 18 14:25:55 mailserver-855855534-bfkp3 postfix/submission/smtpd[1950]: disconnect from unknown[unknown] commands=0/0
Apr 18 14:25:55 mailserver-855855534-bfkp3 opendkim[189]: 3C76A120227: DKIM verification successful
Apr 18 14:25:55 mailserver-855855534-bfkp3 opendkim[189]: 3C76A120227: s=ai d=my-custom-domain.com SSL 
Apr 18 14:25:55 mailserver-855855534-bfkp3 postfix/qmgr[1021]: 3C76A120227: from=<[email protected]>, size=1988, nrcpt=1 (queue active)
Apr 18 14:25:55 mailserver-855855534-bfkp3 postfix/smtp[2952]: warning: inet_protocols: disabling IPv6 name/address support: Address family not supported by protocol
Apr 18 14:25:55 mailserver-855855534-bfkp3 postfix/smtp[2952]: error: open database /etc/postfix/sasl_passwd: No such file or directory
Apr 18 14:25:55 mailserver-855855534-bfkp3 postfix/smtpd[2513]: disconnect from unknown[172.18.91.5] ehlo=2 starttls=1 mail=1 rcpt=1 data=1 quit=1 commands=7
Apr 18 14:25:55 mailserver-855855534-bfkp3 postfix/smtpd[2954]: warning: inet_protocols: disabling IPv6 name/address support: Address family not supported by protocol
Apr 18 14:25:56 mailserver-855855534-bfkp3 dovecot: imap-login: haproxy: Client disconnected (rip=172.18.91.5)
Apr 18 14:25:56 mailserver-855855534-bfkp3 dovecot: imap-login: Disconnected (disconnected before auth was ready, waited 0 secs): user=<>, rip=172.18.91.5, lip=172.18.91.12, session=<y8ivOyBqs6usElsF>
Apr 18 14:25:56 mailserver-855855534-bfkp3 postfix/postscreen[1029]: CONNECT from [172.18.91.5]:51350 to [172.18.91.12]:25
Apr 18 14:25:56 mailserver-855855534-bfkp3 postfix/postscreen[1029]: WHITELISTED [172.18.91.5]:51350
Apr 18 14:25:56 mailserver-855855534-bfkp3 postfix/smtpd[2513]: warning: ignoring non-empty smtpd_upstream_proxy_protocol setting behind postscreen
Apr 18 14:25:56 mailserver-855855534-bfkp3 postfix/smtpd[2513]: connect from unknown[172.18.91.5]
Apr 18 14:25:56 mailserver-855855534-bfkp3 opendmarc[195]: ignoring connection from [172.18.91.5]
Apr 18 14:25:56 mailserver-855855534-bfkp3 postfix/smtpd[2513]: lost connection after CONNECT from unknown[172.18.91.5]
Apr 18 14:25:56 mailserver-855855534-bfkp3 postfix/smtpd[2513]: disconnect from unknown[172.18.91.5] commands=0/0
Apr 18 14:25:56 mailserver-855855534-bfkp3 postfix/submission/smtpd[1950]: warning: haproxy read: unexpected EOF
Apr 18 14:25:56 mailserver-855855534-bfkp3 postfix/submission/smtpd[1950]: connect from unknown[unknown]
Apr 18 14:25:56 mailserver-855855534-bfkp3 postfix/submission/smtpd[1950]: disconnect from unknown[unknown] commands=0/0
Apr 18 14:25:57 mailserver-855855534-bfkp3 postfix/submission/smtpd[1950]: warning: haproxy read: unexpected EOF
Apr 18 14:25:57 mailserver-855855534-bfkp3 postfix/submission/smtpd[1950]: connect from unknown[unknown]
Apr 18 14:25:57 mailserver-855855534-bfkp3 postfix/submission/smtpd[1950]: disconnect from unknown[unknown] commands=0/0

Sorry for the healthchecks in the logs. Does anyone have a hint on how to get rid of them?

I would really appreciate some hints on where to search next or how to debug my SMTP-problems. I have no idea whether my kubernetes-provider has a firewall or if the PROXY-header is not added or whatever else.

Thanks
Marius

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions