Existing tests are failing:

not ok 30 [Getmail] debug-getmail works as expected in 565ms
# (from function `assert_line' in file test/test_helper/bats-assert/src/assert_line.bash, line 219,
#  in test file test/tests/parallel/set1/getmail.bats, line 58)
#   `assert_line --regexp 'retriever:.*SimpleIMAPSSLRetriever\(ca_certs="None", certfile="None", getmaildir="\/tmp\/docker-mailserver\/getmail", imap_on_delete="None", imap_search="None", keyfile="None", mailboxes="\(.*INBOX.*\)", move_on_delete="None", ***, password_command="\(\)", port="993", record_mailbox="True", server="imap.remote-service.test", ssl_cert_hostname="None", ssl_ciphers="None", ssl_fingerprints="\(\)", ssl_version="None", timeout="180", use_cram_md5="False", use_kerberos="False", use_peek="True", use_xoauth2="False", username="user3"\)'' failed
#
# -- no output line matches regular expression --
# regexp : retriever:.*SimpleIMAPSSLRetriever\(ca_certs="None", certfile="None", getmaildir="\/tmp\/docker-mailserver\/getmail", imap_on_delete="None", imap_search="None", keyfile="None", mailboxes="\(.*INBOX.*\)", move_on_delete="None", ***, password_command="\(\)", port="993", record_mailbox="True", server="imap.remote-service.test", ssl_cert_hostname="None", ssl_ciphers="None", ssl_fingerprints="\(\)", ssl_version="None", timeout="180", use_cram_md5="False", use_kerberos="False", use_peek="True", use_xoauth2="False", username="user3"\)
# output (22 lines):
#     retriever:  SimpleIMAPSSLRetriever(ca_certs="None", certfile="None", getmaildir="/var/lib/getmail", imap_on_delete="None", imap_search="None", keyfile="None", mailboxes="('INBOX',)", move_on_delete="None", ***, password_command="()", port="993", record_mailbox="True", server="imap.remote-service.test", ssl_cert_hostname="None", ssl_ciphers="None", ssl_fingerprints="()", ssl_version="None", timeout="180", use_cram_md5="False", use_kerberos="False", use_peek="True", use_xoauth2="False", username="user3")
#     destination:  MDA_external(allow_root_commands="True", arguments="('-d', '[email protected]')", command="deliver", group="None", ignore_stderr="False", path="/usr/lib/dovecot/deliver", pipe_stdout="True", unixfrom="False", user="None")
#     options:
#       delete : False
#       delete_after : 0
#       delete_bigger_than : 0
#       delivered_to : False
#       fingerprint : False
#       logfile : logfile(filename="/var/log/mail/getmail-user3.log")
#       max_bytes_per_session : 0
#       max_message_size : 0
#       max_messages_per_session : 500
#       message_log : /var/log/mail/getmail-user3.log
#       message_log_syslog : False
#       message_log_verbose : False
#       netrc_file : None
#       read_all : False
#       received : False
#       skip_imap_fetch_size : False
#       to_oldmail_on_each_mail : False
#       use_netrc : False
#       verbose : 0
# --
#

I have no idea why the EC lint is failing, though..

TL;DR: Seems that getmail should not be configured for /var/mail-state and this got missed during review as it was likened to fetchmail which is setup differently as a service (although it may technically not need a state dir either for all I know 🤷‍♂️ )

Existing tests are failing

This is due to the logic here:

Previously since the internal directory never existed in the first place (except I guess as a symlink to nothing) it would fallback to the config volume and configure for that instead...

I guess we missed that during review 🤷‍♂️ It shouldn't store data at locations for different concerns? (runtime state that's usually discardable vs config volume that is generally used to persist anything else)

The cronjob (for runtime usage) has the /var/lib/getmail location hard coded (whereas the debug setup command above offers the alternative /tmp/docker-mailserver path):

More context

https://getmail6.org/configuration.html#running-commandline-options

To use getmail, simply run the script getmail, which is typically installed in /usr/local/bin/ by default.
getmail will read the default getmail rc file (getmailrc) from the default configuration/data directory (~/.getmail/) and begin operating.

--getmaildir=DIR or -gDIR — use DIR for configuration and data files

--rcfile=FILE or -rFILE — read getmail rc file FILE instead of the default.

• The file path is assumed to be relative to the getmaildir directory unless this value starts with a slash (/).
• This option can be given multiple times to have getmail retrieve mail from multiple accounts.

Unlike fetchmail, there is no daemon service or polling, thus the cronjob:

• https://getmail6.org/faq.html#faq-how-daemon
• https://wiki.archlinux.org/title/Getmail#Fetching_mail_automatically_with_systemd

RC files are provided via the config volume and copied over to internal location in /etc:

These all have a base config (which our docs advise mounting directly over instead of supporting via the config volume) but to be practical with DMS needs to configure to deliver over to Dovecot:

I'm not sure why this even needs to be in DMS beyond convenience, there isn't anything specific about it to integrate from the looks of it. Although that's probably the same case with fetchmail. Although there's this interesting FAQ entry from the original getmail author criticizing fetchmail.

getmaildir in this case is a location to dump the retrieved remote mail if you haven't configured it to deliver to Dovecot (opt-in). We don't have anything that seems to suggest using /tmp/docker-mailserver/getmail other than in the debug script itself (and test case matching the debug output).

So any existing users of the feature either have had mail delivered to Dovecot as per guidance in the docs, or it's been stored in /var/mail-state/lib-getmail (but we don't document this expectation, and the associated volume isn't really intended for it as it's not runtime state related). The getmail command is run as root UID/GID AFAIK? There's really no value for it being handled in /var/mail-state then.

Hi @polarathene,

I'm attempting to switch from fetchmail to getmail and have a question:

Why use /tmp/docker-mailserver/getmail?

This directory is used by getmail to store information about emails that have already been downloaded. This is because it retrieves messages based on the server's message ID rather than the 'unread' flag of the message as fetchmail doing. So, this directory serves a "data" purpose, and I believe it should not be located inside the config directory, which is often stored in a Git repository.

https://getmail6.org/faq.html#faq-about-oldmail

You are asking if this location should instead be in /var/mail-state? We tend to treat that more for run-time throwaway data, although I think @georglauterbach has redis use it for rspamd now too.

Most data there is intended for /var/run IIRC which has data like the Postfix mail queue, you rarely need to persist that as data you'd migrate to another system, but it is useful for crash recovery so that the queue remains available if you had a power loss event or kernel panic for example.

I can understand that the Config volume is probably not appropriate either. DMS hasn't introduced any other volumes for a long time, this one is usually used for persisting data that isn't logs or /var/mail. Most of the time that's config (hence the name), but the location /tmp/docker-mailserver isn't really reflective of it being config only 😅

I've been thinking an overhaul will need to be considered in future. In the meantime, you are able to volume mount a separate host location over the /tmp/docker-mailserver/getmail path and that should work just fine?

Perhaps @casperklein or @georglauterbach can weigh in with their opinion. From what @mazzz1y describes the getmail data here is state, but it's state you'd want to persist I don't think it's intended for /var/run like other services.

So like the redis database, it might be better if it were stored at a separate location intended for persisting data. Presently that purpose overlaps with the /tmp/docker-mailserver container path, we just refer to this as the Config Volume.

Open to suggestions, but such changes would need to be queued for v15 as it'd be a breaking change.

@mazzz1y you may want to open a new issue and link to the discussion here. Go with Other -> Feedback for issue type.

I think it'd be best to store it in /var/mail-state, but we probably need a separate discussion what should go where.

You are asking if this location should instead be in /var/mail-state?
I think it'd be best to store it in /var/mail-state, but we probably need a separate discussion what should go where.

Yes! This is definitely what is needed. Getmail stores its own state there, and I think /var/mail-state is intended for exactly this purpose (at least by name).

In the meantime, you are able to volume mount a separate host location over the /tmp/docker-mailserver/getmail path and that should work just fine?

I'm already doing this, but it doesn't look nice :)

    volumes:
      - ./data/maildata:/var/mail
      - ./data/mailstate:/var/mail-state
      - ./data/getmail:/tmp/docker-mailserver/getmail
      - ./config/docker-mailserver:/tmp/docker-mailserver

Also, this method results in Docker creating an empty ./config/docker-mailserver/getmail directory on the host. This happens because the mounting action inside the Docker container requires the creation of an empty directory at /tmp/docker-mailserver/getmail before mounting. It doesn't look good, but it works, so the issue is not a significant one and is more of an aesthetic concern

I will create a separate issue

I've been thinking an overhaul will need to be considered in future.

I might migrate from fetchmail to getmail in mid september. I will then take a closer look at the getmail implementation and possibly improve it, e.g. making it a service that is handled by supervisord instead of the current cron approach.

Regarding /var/mail-state, I think that makes sense.

e.g. making it a service that is handled by supervisord instead of the current cron approach.

Discussion should probably continue at the new issue for this, so I'll be brief.

I recall getmail lacking a daemon, it functions differently from fetchmail, hence the reliance on a cron timer to mimic the polling behaviour of fetchmail. (EDIT: I covered this in my earlier comment above under "More Context" section)