Skip to content

breaking: Refactor getmail support#4156

Merged
casperklein merged 37 commits intodocker-mailserver:masterfrom
casperklein:getmail
Aug 17, 2024
Merged

breaking: Refactor getmail support#4156
casperklein merged 37 commits intodocker-mailserver:masterfrom
casperklein:getmail

Conversation

@casperklein
Copy link
Copy Markdown
Member

@casperklein casperklein commented Aug 9, 2024

Description

Related discussion: #4149

New:

  • Introducing getmailrc_general config file. This allows to easily change the base options used for every getmail RC file.
  • IMAP/POP3 configs added to config-examples directory

Changed:

  • GETMAIL_POLL variable is not limited to 30 minutes anymore
  • Instead of cron, a new supervisord service was created to make the periodic polls.

Breaking Changes:

  • getmail configurations are now stored in their own directory: /tmp/docker-mailserver/getmail.
  • The message_log option has been removed. No log file for each getmail configuration is created anymore. Instead, like the other services, logging goes to syslog and end up in mail.log.
  • The getmail state-dir is changed from /tmp/docker-mailserver/getmail to /var/mail-state/getmail.

Not covered by this PR / ToDo:

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Improvement (non-breaking change that does improve existing functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

Checklist

  • My code follows the style guidelines of this project
  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation (README.md or the documentation under docs/)
  • If necessary, I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • I have added information about changes made in this PR to CHANGELOG.md

* Verify GETMAIL_POLL
* Switch state-dir from /tmp/docker-mailserver/getmail to /var/lib/getmail
* Add support for custom getmailrc_general.cf file
* remove getmail 'message_log'  option in favor of 'message_log_syslog'
* Change config directory structure:

├── docker-data/dms/config
│   ├── getmail-example.cf

to

├── docker-data/dms/config
│   ├── getmail
│   │   ├── getmailrc_general.cf
│   │   ├── imap-example.cf
│   │   ├── pop3-example.cf
@docker-mailserver docker-mailserver deleted a comment from github-actions Bot Aug 9, 2024
@casperklein casperklein self-assigned this Aug 9, 2024
@casperklein casperklein added area/scripts kind/improvement Improve an existing feature, configuration file or the documentation area/documentation service/getmail labels Aug 9, 2024
@casperklein casperklein marked this pull request as ready for review August 9, 2024 20:36
@polarathene polarathene added this to the v15.0.0 milestone Aug 9, 2024
@polarathene
Copy link
Copy Markdown
Member

polarathene commented Aug 9, 2024

Since this introduces breaking changes and will need to be delayed until v15.

Review is not a high priority for me right now (I've got a lot to juggle as-is). I will come back to this when I can, I still need to tackle an update to LDAP docs for 14.1 release before I can resume the active LDAP PR that brings breaking changes for v15 (and will require revised LDAP docs again).

Presently wrapping up OAuth/OIDC related task for DMS docs. I should be able to provide an initial review here after I've got that out of the way 👍


At a quick glance, while I don't have an issue with introducing a supervisor service, what was the motivation to replace cron with a while sleep loop script? (we use cron for scheduling periodic actions elsewhere... because that's what it's meant for?)

@casperklein
Copy link
Copy Markdown
Member Author

casperklein commented Aug 10, 2024

Since this introduces breaking changes and will need to be delayed until v15.

Is there a reason, not to release v15 next? I don't want this PR to be open for indefinite time (until v15 is eventually released) and handling merge conflicts in the meantime. I think we once discussed that: As soon as breaking changes are merged, the next version should be a major version. I see no Problem in skipping any 14.x releases / I see no value in forcefully releasing 14.x.

what was the motivation to replace cron with a while sleep loop script?

  • Streamlining with all other services and
  • Giving the ability to start/stop/restart the service on demand (e.g. if you want getmail to check for new mails now, you can just restart the service.)
  • Cronjobs run somewhat uncontrollable in the background (and IMO are harder to debug in case of errors.)

… example, which makes more sense compared to the container directories.
@polarathene
Copy link
Copy Markdown
Member

Is there a reason, not to release v15 next?

I don't have any reason. I just want to release with docs update before my LDAP PR is merged for a follow-up release for that.

If you want to go straight to v15 that's fine by me and I'll review within the next few days.


Reasons to replace cron job makes sense, thanks for that 👍

@casperklein
Copy link
Copy Markdown
Member Author

BTW: Anything more long-term that was considered for v15 can go into v16. Or we just wait with releasing v15.

@mazzz1y
Copy link
Copy Markdown
Contributor

mazzz1y commented Aug 12, 2024

Cronjobs run somewhat uncontrollable in the background (and IMO are harder to debug in case of errors.)

Getmail uses cron for notifications. If getmail fails to fetch emails from retriever, cron sends an error notification to the mailbox

Will the supervisor solution send a notification in case of failure?

The Fetchmail daemon does the same

@casperklein
Copy link
Copy Markdown
Member Author

casperklein commented Aug 12, 2024

I thought about the same. However, DMS does not send any error mails to the user.

The Fetchmail daemon does the same

I cannot confirm that. When I change, e.g. the hostname to something not resolvable, the error appears in the mail.log, but no notification mail is sent:

Aug 12 12:37:21 mail fetchmail[14032]: couldn't find canonical DNS name of does.not.exist (does.not.exist): Name or service not known
Aug 12 12:37:21 mail fetchmail[14032]: Query status=11 (DNS)
Aug 12 12:37:21 mail fetchmail[14032]: getaddrinfo("does.not.exist","pop3s") error: Name or service not known
Aug 12 12:37:21 mail fetchmail[14032]: POP3 connection to does.not.exist failed: Success
Aug 12 12:37:21 mail fetchmail[14032]: Query status=2 (SOCKET)

Why mail notification might not be a good idea / why I've decided against implementing something like that:

Given the default poll interval of 5 minutes, you would receive an error mail every 5 minutes until the problem is resolved.

@mazzz1y
Copy link
Copy Markdown
Contributor

mazzz1y commented Aug 12, 2024

I cannot confirm that.

I'm receiving such messages from fetchmail, maybe for authorization only:

Fetchmail could not get mail from xxx.

The attempt to get authorization failed.
Since we have already succeeded in getting authorization for this
connection, this is probably another failure mode (such as busy server)
that fetchmail cannot distinguish because the server didn't send a useful
error message.

However, if you HAVE changed your account details since starting the
fetchmail daemon, you need to stop the daemon, change your configuration
of fetchmail, and then restart the daemon.

The fetchmail daemon will continue running and attempt to connect
at each cycle.  No future notifications will be sent until service
is restored.
-- 
The Fetchmail Daemon

Given the default poll interval of 5 minutes, you would receive an error mail every 5 minutes until the problem is resolved.

Agreed, this is an issue too

@casperklein
Copy link
Copy Markdown
Member Author

maybe for authorization only

Confirmed. When I configure a wrong password, then I also receive the same mail.

Comment thread docs/content/config/advanced/mail-getmail.md Outdated
Copy link
Copy Markdown
Member

@polarathene polarathene left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apparently if I do the review in the diff view but choose to submit as "Comment" type with no top-level message, it silently updates the earlier review feedback without listing them under a review event for better visibility 🤔


So for reference, this is all that really remains:

  • Update inline docs: #4156 (comment)
  • Minor revision to docs (found instead of provides, more explicit reference to location, mention .cf extension expectation_), the file tree above that line basically conveyed all that implicitly but applying the suggestion may still be worthwhile: #4156 (comment)
  • Your call on ENV enablement for debug methods: #4156 (comment)
  • You may want to include the getmail service wrapper as part of the process check tests: #4156 (comment)
  • The _syslog_error() feedback below

Comment thread target/getmail/getmail-service.sh
casperklein and others added 4 commits August 15, 2024 12:33
getmail6 itself logs as "getmail" to syslog.
we align here to "getmail" too.
@casperklein
Copy link
Copy Markdown
Member Author

casperklein commented Aug 15, 2024

I've checked the boxes in your post, that I consider to be done. I've commented the other ones and waiting for your final feedback there 👍


Apparently if I do the review in the diff view but choose to submit as "Comment" type with no top-level message, it silently updates the earlier review feedback without listing them under a review event for better visibility 🤔

I noticed some glitches? too. For example, when I reply in the diff view, I can't see my answer here in the conversation tab. I just see your comment but not my reply here.

logger --id=$$ ... leads to this error message
    logger: send message failed: Operation not permitted

The same issue is mentioned here: jonathanio/update-systemd-resolved#25
@polarathene
Copy link
Copy Markdown
Member

Everyone can look up commands unknown to them :)

While I agree, logger was a little unexpected, as I noted in that comment you responded to 😝

I'm fine with git blame as sufficient for now.

I was going to suggest adding a clarifying comment for the $$ / ${$} bash syntax to acquire the script PID, since for anyone unfamiliar that is not search friendly 😂 Interesting that it caused an error 🤔


I noticed some glitches? too.
For example, when I reply in the diff view, I can't see my answer here in the conversation tab. I just see your comment but not my reply here.

Yeah, that's related to what I was saying... kinda.

If I do a review in the diff view tab and submit the review with feedback added to existing review threads, I think it only shows my new review feedback comment for visibility (sadly with no link to the earlier thread it belongs to).

The actual discussion thread remains attached to the previous review that introduced the review comment. Although it may no longer be visible in the diff view tab if the original lines it referenced were since deleted (or modified too heavily?). So most of the time it's easier to browse the diff view tab for an overview as the conversation tab gets unwieldly (unresolved comment threads also can be hidden under the "load more" button).

When there's too many threads littered around to navigate I make an overview comment with links so it's easier for us both 😅

polarathene
polarathene previously approved these changes Aug 15, 2024
Copy link
Copy Markdown
Member

@polarathene polarathene left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cheers for putting the time into improving this! Great work❤️

@polarathene polarathene changed the title Getmail revision breaking: getmail refactor Aug 15, 2024
@polarathene polarathene changed the title breaking: getmail refactor breaking: Refactor getmail support Aug 16, 2024
polarathene
polarathene previously approved these changes Aug 16, 2024
Copy link
Copy Markdown
Member

@georglauterbach georglauterbach left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just small questions and nitpicks - overall a good PR :)

Comment thread docs/content/config/advanced/mail-getmail.md Outdated
Comment thread target/getmail/getmail-service.sh Outdated
Comment thread target/getmail/getmail-service.sh Outdated
Comment thread target/getmail/getmail-service.sh Outdated
Comment thread target/getmail/getmail-service.sh
Comment thread target/scripts/startup/setup.d/getmail.sh Outdated
casperklein and others added 2 commits August 16, 2024 16:16
@github-actions
Copy link
Copy Markdown
Contributor

Documentation preview for this PR is ready! 🎉

Built with commit: f48aa22

@casperklein casperklein merged commit b2978fd into docker-mailserver:master Aug 17, 2024
@casperklein casperklein deleted the getmail branch August 17, 2024 10:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/documentation area/scripts kind/improvement Improve an existing feature, configuration file or the documentation service/getmail

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants