Skip to content

spam: use Sieve for rewriting subject with Rspamd & SA/Amavis#3820

Merged
georglauterbach merged 26 commits intomasterfrom
rspamd/use-sieve-for-subject-rewrite
Jan 29, 2024
Merged

spam: use Sieve for rewriting subject with Rspamd & SA/Amavis#3820
georglauterbach merged 26 commits intomasterfrom
rspamd/use-sieve-for-subject-rewrite

Conversation

@georglauterbach
Copy link
Copy Markdown
Member

@georglauterbach georglauterbach commented Jan 24, 2024

Description

This PR reworks subject-rewriting. It disables the default rewrite_subject action in Rspamd and renames SA_SPAM_SUBJECT to SPAM_SUBJECT. Under the hood, I utilized Dovecot sieve scripts to adjust the Subject header, which seems to work quite nicely.

Review commit by commit.

@polarathene The commit history contains d0eccd3, which is not strictly related to this PR, but the new tests (like 'SPAM_SUBJECT works') make use of it. I hope it's not too far out of the boundaries to justify another PR. I have only adjusted the Rspamd test file to keep the diff minimal and will provide a follow-up to adjust the whole test suite.

Fixes #3804
Fixes #3323

Type of change

  • New feature (non-breaking change which adds functionality)
  • Improvement (non-breaking change that does improve existing functionality)
  • This is a breaking change
  • This change requires a documentation update

Checklist:

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation (README.md or the documentation under docs/)
  • If necessary I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • I have added information about changes made in this PR to CHANGELOG.md

@georglauterbach georglauterbach added kind/update Update an existing feature, configuration file or the documentation service/security/rspamd labels Jan 24, 2024
@georglauterbach georglauterbach added this to the v14.0.0 milestone Jan 24, 2024
@georglauterbach georglauterbach self-assigned this Jan 24, 2024
@georglauterbach georglauterbach force-pushed the rspamd/use-sieve-for-subject-rewrite branch from c4e20a3 to 3f7a492 Compare January 24, 2024 18:29
@georglauterbach georglauterbach changed the title Rspamd/use sieve for subject rewrite Rspamd: use Sieve for re-writing an e-mail's subject Jan 24, 2024
Comment thread target/scripts/startup/setup.d/security/rspamd.sh Outdated
Comment thread docs/content/config/environment.md Outdated
polarathene

This comment was marked as outdated.

This commit is just altering the documentation and gives an impression
on the upcoming changes.
If I am not mistaken, the configuration I am checking for is the one we
should emit a warning about: in case junk mail is moved to the inbox and
no rewriting happens, we should check whether this is actually what the
user wants.
This helper will be used in un upcoming commit that adds testing
functionality. In a follow-up PR, the helper will be applied in all
tests.
@georglauterbach
Copy link
Copy Markdown
Member Author

georglauterbach commented Jan 25, 2024

I have reset the changes and started afresh (hence the force push, and, a need to re-review everything 😆). The new var SPAM_SUBJECT is agnostic of the underlying anti-spam system. This should better reflect maintainer's views (including mine) and help transitioning between anti-spam systems.


PS: @polarathene noted there is no equivalent of SA_KILL in conjunction with SPAMASSASSIN_SPAM_TO_INBOX. This is, IMO, useless with Rspamd: just set reject = 100; in Rspamd's configuration. This way, no e-mail will be rejected if that's what you're after. Just a side note, unrelated to changes made in this PR.

Comment thread CHANGELOG.md Outdated
Comment thread CHANGELOG.md Outdated
Comment thread mailserver.env
Comment thread target/rspamd/local.d/actions.conf Outdated
Comment thread target/scripts/startup/check-stack.sh Outdated
Comment thread target/scripts/startup/setup.d/security/rspamd.sh Outdated
Comment thread target/scripts/startup/setup.d/security/misc.sh Outdated
@polarathene
Copy link
Copy Markdown
Member

polarathene commented Jan 25, 2024

For visibility / discoverability, since the feedback has been hidden, related Sieve insights / advice related to spam:

@polarathene
Copy link
Copy Markdown
Member

there is no equivalent of SA_KILL in conjunction with SPAMASSASSIN_SPAM_TO_INBOX. This is, IMO, useless with Rspamd: just set reject = 100; in Rspamd's configuration. This way, no e-mail will be rejected if that's what you're after

SA_KILL is basically the equivalent of your reject rspamd setting, except it was agnostic to the action taken, it was a score threshold that triggered a different final action, however DMS only supported adjusting that action between D_PASS and D_BOUNCE via SPAMASSASSIN_SPAM_TO_INBOX toggle.

In the past users had done the same workaround approach by setting SA_KILL to 1000. However SA_KILL also triggered some other behaviour such as adding a copy to quarantine. It was basically a marker for what is definitely spam, and the quarantine could be used for spam training, while still allowing the mail through to the junk folder.

I don't know how useful either of the two settings really are for users to configure though, just that we previously defaulted to D_BOUNCE and that was not what users were happy with, but at the time SA_KILL was set to 6.31 like SA_TAG2, which meant if it was considered as flagged for spam it'd be bounced without the user knowing. Legitimate mail was scoring 8 in some cases. SA_KILL was later bumped to 10, which came after changing the default D_BOUNCE to D_PASS (deliver to users junk).

I don't know how well rspamd reject maps to SA scores, but if they're roughly equal I guess that's fine and we won't have any user complaints 🤷‍♂️ Just bringing up that difference in support between the two services in DMS for awareness.

- counterpart to `_file_exists_in_container`
- usage was adjusted in all Rspamd-related tests and in
  Amavis tests
@georglauterbach
Copy link
Copy Markdown
Member Author

@georglauterbach
Copy link
Copy Markdown
Member Author

Few adjustments and it looks good to go 👍

🚀

May want to wait to see if @casperklein has any input regarding this change impact to SpamAssassin / SA_SPAM_SUBJECT?

👍🏼

@polarathene
Copy link
Copy Markdown
Member

@polarathene It seems like #3838 broke our docs preview deployment

Oh damn, I didn't pay attention to the difference in similar parameter names _ => -, whoops! 😬

polarathene
polarathene previously approved these changes Jan 27, 2024
Copy link
Copy Markdown
Member

@polarathene polarathene left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@casperklein should be given at least 48 hours to express an interest in any review feedback.

  • Sorry about all the notification noise 😅
  • See the changelog entry of this PR first, if you don't have any issue with what is proposed there, no further input required beyond an approval 👍

@georglauterbach georglauterbach changed the title Rspamd: use Sieve for re-writing an e-mail's subject spam: use Sieve for rewriting subject with Rspamd & SA/Amavis Jan 28, 2024
Comment thread target/scripts/startup/setup.d/security/misc.sh Outdated
@casperklein
Copy link
Copy Markdown
Member

casperklein commented Jan 28, 2024

I've read #3804. Did I understand it correctly, that rspam is not capable of adding a X-SPAM Header and at the same time changing the subject? That's a pity 😆

I really don't like when features are reinvented (in sieve) that are build-in and working perfectly in SA 😞 IMO this is the job of a spam filter and not dovecots..


Code LGTM. Just one thing, see above.

@github-actions
Copy link
Copy Markdown
Contributor

Documentation preview for this PR is ready! 🎉

Built with commit: 6364495

@georglauterbach
Copy link
Copy Markdown
Member Author

georglauterbach commented Jan 29, 2024

I've read #3804. Did I understand it correctly, that rspam is not capable of adding a X-SPAM Header and at the same time changing the subject? That's a pity 😆

Indeed; I originally assumed this was the case. It really is a shame..

I really don't like when features are reinvented (in sieve) that are build-in and working perfectly in SA 😞 IMO this is the job of a spam filter and not dovecots..

I agree, but here I guess "it is what it is" fits most. With this change, we're one step closer to being agnostic to the underlying anti-spam (a silver lining), though.

@georglauterbach georglauterbach merged commit afb0093 into master Jan 29, 2024
@georglauterbach georglauterbach deleted the rspamd/use-sieve-for-subject-rewrite branch January 29, 2024 12:38
@ghnp5
Copy link
Copy Markdown
Contributor

ghnp5 commented Jan 29, 2024

Hey!

Thanks very much for this.

Is there a summary somewhere of what the breaking changes will be for this?

Many thanks, again! 🥳

@georglauterbach
Copy link
Copy Markdown
Member Author

Yes, there is - have a look at our changelog (CHANGELOG.md) 👍🏼

@williamdes
Copy link
Copy Markdown
Contributor

For reference here is the rspamd issue report I was referring to when I said that I did think that this PR would probably fix my spam rewrote emails not being moved to Spam folder : rspamd/rspamd#3078 (comment)

To be confirmed

AlMaVizca added a commit to AlMaVizca/docker-mailserver-helm that referenced this pull request Jun 19, 2024
AlMaVizca added a commit to AlMaVizca/docker-mailserver-helm that referenced this pull request Jun 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind/update Update an existing feature, configuration file or the documentation service/security/rspamd

Projects

Archived in project

Development

Successfully merging this pull request may close these issues.

bug report: rspamd X-Spam header not applied to spam message with score 9 tracking: equivalent configs of SpamAssassin & Rspamd

6 participants