Review context regarding version changes in Debian 12 packages, plus some related history/notes for my benefit 😅

Major version updates should be included in changelog.

• fetchmail no notable change, 6.4 on both Debian 11 and 12.
• getmail6 (6.18 => 6.18)
  • Debian 11 provided 6.14 in Bullseye repo and 6.18 in Bullseye-backports repo, but we were installing getmail6 via the python package manager pip. Switching to Debian package with Debian 12.
• openssl (1.1.1n => 3.0.11)
  • Tests caught a slight difference in some output. I haven't looked over the changelog for if this major release impacts anything else.
• amavisd-new (2.11 => 2.13)
  • Adds dep of systemd | systemd-standalone-sysusers | systemd-sysusers. Installing systemd-standalone-sysusers is an appropriate non-default choice for our non-systemd requirement, we should add that dependency explicitly.
  • NOTE: Instead of check_policy_service integration to Postfix, amavisd-milter package could be used instead.
  • Suggested deps (which DMS explicitly installs):
    • clamav (0.103.8 => 1.0.3)
    • spamassassin (3.4.6 => 4.0.0)
• postfix (3.5.23 => 3.7.9)
• redis-server (6.0 => 7.0)
• rsyslog (8.2102 => 8.2302)
• supervisor (4.2.2 => 4.2.5)
• dovecot-core (2.3.13 => 2.3.19)
  • The current default in DMS v13.3 is a community edition install (from Dovecot repos instead of Debian) that is presently at 2.3.21 (Sep 2023 release).
  • Debian 12 (Bookworm) is expected to get 2.3.21 or newer (Dovecot devs plan to release 2.4.0 in 2023H2) via the bookworm backports repo later this year. UPDATE: As of Jan 2024, this is still not the case.
  • Downgrading to Dovecot 2.3.19 notably affects support for OAuth2 (with JWT or using client_id + client_secret with POST requests) and Lua HTTP client (DNS related).
• dovecot-fts-xapian (1.4.9 => 1.5.5)
  • libxapian30 dep >= 1.4.19 + dovecot-abi-2.3.abiv19 virtual dep (provided via dovecot-core)
  • Package previously broke compatibility with Debian 11 Bullseye packages when mixed with third-party Dovecot repo for a newer dovecot-core which could no longer satisfy the ABI virtual dependency.
    • Solution was to build package dovecot-fts-xapian 1.5.5 from source via introducing build/compile.sh.
    • dovecot-fts-xapian was originally integrated into DMS in July 2021 via single package and a documented config guide.
  • When dovecot-core 2.4.0 is available, migrating to the flatcurve plugin becomes a viable option and may instead be a better replacement for this feature.
  • Additionally, the suggestion to migrate to the flatcurve plugin was proposed by the contributor that added support for the xapian plugin in DMS (which was due to the existing solr plugin having notable drawbacks).
• postsrsd (1.10 - no notable change from Debian 11)
  • There is a bug within common container environments that have containerd or similar service on the host configured with LimitNOFILE=infinity where fs.nr_open is a value of 2^30 instead of 2^20 (Debian).
    • That bug was fixed in Aug 2022 but only available with PostSRSd v1.12 / v2.
    • Thus users of this feature in DMS may need to explicitly configure the --ulimit to 1024:524288 (systemd implicit default since v240).
    • The fix also requires at least glibc of 2.35, which Debian 12 provides with glibc 2.36. Docker host kernel is required to be >= 5.9.
• postfix-policyd-spf-python (2.9.2 => 3.04)
  • 3.x dropped the python3-spf dep, which supposedly is not needed anymore to provide Postfix with an SPF check_policy_service.
  • NOTE: There also exists an alternative milter-based package pyspf-milter which appears to be part of python3-spf-engine too (available in spf-engine since 2.9.0). See the spf-engine README for advice on configuring Postfix with milter package.
  • References:
    • spf-engine upstream source and Debian source.
    • Some resources online provide nicer docs to browse.
      • policyd-spf Postfix integration advice (states version 2.0, but this seems to match the python-policy-spf 2.0.0 mention in README source (upstream)) notes that when using the check_policy_service it must come after reject_unauth_destination to avoid creating an open relay (DMS handles this correctly at present, but that expectation is not clearly documented inline for maintainers).
      • Similar pyspf-milter docs.

Regarding base image change consideration for Alpine vs Fedora:

• getmail6 - Fedora and Alpine both lack getmail6 and would need to install via pip. At least for Alpine 3.18 there is no getmail6 package, Alpine edge has a package which may make it available in a future release (Alpine 3.19?).
• rspamd - Fedora does not have any official package for this. There is a relatively maintained third-party COPR repo (github repo), and I had some success IIRC building for Fedora via Dockerfile myself in the past. Would be better if upstream rspamd had more official support, but unclear when that'd happen and that's one of the bigger drawbacks for choosing Fedora.
• postfix-policyd-spf-python - Alpine doesn't have this, closest might be a Perl version, Fedora has it as pypolicyd-spf. Probably acceptable if rspamd becomes the only option in future release of DMS.
• spamassassin - Alpine is still on 3.4, even on edge channel (with a July 2023 build).
• postgrey - Alpine doesn't seem to have this package available (nor on edge).

I still have reasons for cautioning against Alpine as the base image for DMS, but lacking time to do a write-up going into detail 😅

EDIT: Presently there is a bug with QEMU that's being triggered by ARM builds that affects images using Alpine musl, stalling due to timeout. Thought I'd mention that as it's affecting quite a few CI users building docker images that use Alpine. As mentioned before, there is a variety of problems you can experience adopting Alpine and I'd strongly discourage that for DMS.

Also as an additional reference, there was a recent discussion about base image change and migrating from bash to a language like rust. While neither is likely to happen (as per the discussion), it does highlight various problems known with Debian + bash.

I will come back to this PR. @polarathene thank you for your elaborate review; I will make sure to go all of it as soon as I have time again. Probably October.

I've resolved both of these for you directly.

• Renamed the amavis process to test against
• Disabled XOAUTH2 test-case (could bring back netcat, but OAUTHBEARER should be equivalent coverage on the functionality we're interested in)

@polarathene can you please have a look at these test failures
_check_if_process_is_running amavisd-new

Should be a fairly obvious one to fix considering the related changes in this PR. Amavis process name changed, it needs to be adjusted in the test accordingly. The test only fails now from the revision because the coverage / specificity improved to catch it 😅

Should just be changing the check from amavisd-new to amavisd.

'__should_login_successfully_with 'XOAUTH2'' failed

dovecot: imap-login: Login: user=<[email protected]>, method=OAUTHBEARER

This one is odd.

The logs found in the output to match have OAUTHBEARER instead of XOAUTH2. However these two methods are split into separate test cases and XOAUTH2 comes first.

Perhaps a bug with the updated curl? (EDIT: Confirmed, curl 7.74.0 => 7.88.1, bug was introduced in curl 7.80.0 release from Nov 2021, still present in the latest Dec 2023 8.5.0 release)

I have applied your latest review comments about the Rspamd installation and will merge this PR now when tests are passing :)

Finally 🚀🚀🚀 Thank you guys for the nice feedback and review!

I noticed, that the timestamp format in mail.log has changed.

Old:

Oct  1 03:10:21 mail dovecot: pop3-login: Login: user=<[email protected]>, method=PLAIN, rip=46.xx.xx.xx, lip=172.xx.xx.xx, mpid=6198, TLS, session=<yWpVUp0GWpMu78L+>                                                             │l
Oct  1 03:10:21 mail dovecot: pop3([email protected])<6198><yWpVUp0GWpMu78L+>: Disconnected: Logged out top=0/0, retr=0/0, del=0/944, size=105872624

New:

2024-01-29T12:40:52.411491+01:00 mail dovecot: pop3-login: Login: user=<[email protected]>, method=PLAIN, rip=37.xx.xx.xx, lip=172.xx.xx.xx, mpid=23610, TLS, session=<XGE0HhQQVRwlyPVQ>                                                     │i
2024-01-29T12:40:52.471401+01:00 mail dovecot: pop3([email protected])<23610><XGE0HhQQVRwlyPVQ>: Disconnected: Logged out top=0/0, retr=0/0, del=0/0, size=0

It's not only dovecot, postfix etc. uses the same new format. Any idea how to revert this back to the old format?

PS: This is a breaking change for someone parsing mail.log entrys.

The change of the time format ist mentioned here. There are other worth mentioning changes, e.g. the files /var/log/mail.{info,warn,err} are no longer created.

Any idea how to revert this back to the old format?

Adding $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat at the top of /etc/rsyslog.conf brings back the old style.

It seems like rsyslog finally adopted ISO8601 as a time stamp format. I actually really appreciate that (I always thought the rsyslog format was weird and most-notable non-standard)!

We should provide an entry in the changelog though, indeed.

It seems like rsyslog finally adopted ISO8601 as a time stamp format.

That's probably the newer syslog RFC format not just the timestamp change, whereas before it was BSD syslog which was less formalized in spec.

I always thought the rsyslog format was weird and most-notable non-standard

I really like the "weird" format, because it's very easily human readable, compared to the high precision format now used. Having lots of servers and a central logging, the new format seems to be the better choice. But for a single server I prefer it simple and easy accessible.
The fix is now in my user-patches.sh.

I understand that :) Can you additionally provide a PR that updates the CHANGELOG @casperklein?