This will still potentially trigger a change detection right? As it compares to it's before/after checksums in the config volume?

It seems that DKIM keys aren't copied over to an internal location like the certs are? I'm not too concerned about that I guess 🤔

FWIW, the rspamd docs mention rspamadm template for generating configs from jinja templates if you wanted to take advantage of that 🤷‍♂️

UCL can also be JSON (easier to manipulate)

Or.. since the UCL format also supports a JSON representation, you could also just concat your default config base as yaml:

# documentation: https://rspamd.com/doc/modules/dkim_signing.html
enabled: true
sign_authenticated: true
sign_local: false
try_fallback: false
use_domain: header
use_redis: false # don't change unless Redis also provides the DKIM keys
use_esld: true
allow_username_mismatch: true
check_pubkey: true # you want to use this in the beginning

with a loop over domain blocks (which could be generated as separate files if that's better):

domain:
  ${DOMAIN}:
    path: ${PRIVATE_KEY_FILE}
    selector: ${SELECTOR}

eg:

domain:
  # Single selector example:
  example.com:
    path: /dms/dkim/rsa-2048-mail-rsa-example.com.private.txt
    selector: mail-rsa

  # Multiple selectors example:
  example.org:
    selectors:
      - path: /dms/dkim/rsa-2048-mail-rsa-example.org.private.txt
        selector: mail-rsa
      - path: /dms/dkim/ed25519-mail-ec-example.org.private.txt
        selector: mail-ec

Then with yq we can take this YAML and generate a single JSON document:

$ yq -o json <(cat base.yaml domains.yaml)
{
  "enabled": true,
  "sign_authenticated": true,
  "sign_local": false,
  "try_fallback": false,
  "use_domain": "header",
  "use_redis": false,
  "use_esld": true,
  "allow_username_mismatch": true,
  "check_pubkey": true,
  "domain": {
    "example.com": {
      "path": "/dms/dkim/rsa-2048-mail-rsa-example.com.private.txt",
      "selector": "mail-rsa"
    },
    "example.org": {
      "selectors": [
        {
          "path": "/dms/dkim/rsa-2048-mail-rsa-example.org.private.txt",
          "selector": "mail-rsa"
        },
        {
          "path": "/dms/dkim/ed25519-mail-ec-example.org.private.txt",
          "selector": "mail-ec"
        }
      ]
    }
  }
}

Consistent input format transformed to rspamd single vs multi selector

We can be more consistent with the generation too, changing our domains.yaml into:

domain:
  example.com:
    selectors:
      - path: /dms/dkim/rsa-2048-mail-rsa-example.com.private.txt
        selector: mail-rsa

  example.org:
    selectors:
      - path: /dms/dkim/rsa-2048-mail-rsa-example.org.private.txt
        selector: mail-rsa
      - path: /dms/dkim/ed25519-mail-ec-example.org.private.txt
        selector: mail-ec

Then we can produce the same output as above by filtering for selectors with only 1 element, and rewriting that domain entry into path + selector fields:

yq -o json \
  '. | with(.domain[]; . |=  select(.selectors | length == 1) |= .selectors[0])' \
  <(cat base.yaml domains.yaml)

Now you could handle multiple domains like the open-dkim generator supports.

Better unified internal storage layout for dkim keys

Although due to multiple key types and selectors, a better structure might be:

/dms/dkim/keys/${DOMAIN}/${SELECTOR}/{private,public}.pem

The DNS TXT record can be generated from the public.pem, assuming public.pem is actually just the public key. The rspamd generator like opendkim outputs the zone file record, but additionally creates another file that extracts the record value into a single line (which other DNS providers are more compatible with).

With the above structure, both opendkim and rspamd should be compatible to share the source and you'd have fairly predictable/semantic paths to work with.

That is beneficial for the dkim_signing.conf dynamic generation since you can loop through it with yq to create the config:

# export keyword for first 2 examples for `yq` to access ENV:
export DOMAIN="example.com"
export SELECTOR="mail-rsa"
export PRIVATE_KEY_FILE="/dms/dkim/keys/example.com/mail-rsa/private.pem"

# No input, create a new YAML doc:
yq --null-input '{ env(DOMAIN): { "selectors": [{ "path": env(PRIVATE_KEY_FILE), "selector": env(SELECTOR) }] }}'

# Easier to read spread across multiple lines:
yq --null-input '{
  env(DOMAIN): {
    "selectors": [{
      "path": env(PRIVATE_KEY_FILE),
      "selector": env(SELECTOR) }]
  }
}'

# HereDoc a template to use vars as keys/values
yq -P << EOF
{
  "${DOMAIN}": {
    "selectors": [{
      "path": "${PRIVATE_KEY_FILE}",
      "selector": "${SELECTOR}"
    }]
  }
}
EOF

# All output the following YAML:
example.com:
  selectors:
    - path: /dms/dkim/keys/example.com/mail-rsa/private.pem
      selector: mail-rsa

Walkthrough of dynamic dkim_signing.conf generation with yq

If each keypair location had an associated YAML snippet like above, you could then scan the directory tree for the yaml files and merge them all together:

Grab all the YAML snippets:

$ find /dms/dkim/keys -type f -name "*.yaml" -exec cat {} \;

example.com:
  selectors:
    - path: /dms/dkim/keys/example.com/mail-ec/private.pem
      selector: mail-ec
example.com:
  selectors:
    - path: /dms/dkim/keys/example.com/mail-rsa/private.pem
      selector: mail-rsa
example.org:
  selectors:
    - path: /dms/dkim/keys/example.org/mail-rsa/private.pem
      selector: mail-rsa

Now pass that into yq and merge under the domain field/object:

$ find /dms/dkim/keys -type f -name "*.yaml" \
  | xargs yq eval-all '. as $item ireduce ({}; . *+ $item ) | { "domain": . }'

domain:
  example.com:
    selectors:
      - path: /dms/dkim/keys/example.com/mail-ec/private.pem
        selector: mail-ec
      - path: /dms/dkim/keys/example.com/mail-rsa/private.pem
        selector: mail-rsa
  example.org:
    selectors:
      - path: /dms/dkim/keys/example.org/mail-rsa/private.pem
        selector: mail-rsa

If rspamd requires single selectors to avoid arrays, additionally handle that:

$ find /dms/dkim/keys -type f -name "*.yaml" \
  | xargs yq eval-all '. as $item ireduce ({}; . *+ $item ) | { "domain": . }
  | with(.domain[]; . |=  select(.selectors | length == 1) |= .selectors[0])'

domain:
  example.com:
    selectors:
      - path: /dms/dkim/keys/example.com/mail-ec/private.pem
        selector: mail-ec
      - path: /dms/dkim/keys/example.com/mail-rsa/private.pem
        selector: mail-rsa
  example.org:
    path: /dms/dkim/keys/example.org/mail-rsa/private.pem
    selector: mail-rsa

Finally, set JSON as output and prepend the base.yaml config:

$ find /dms/dkim/keys -type f -name "*.yaml" \
  | xargs yq -o json eval-all '. as $item ireduce ({}; . *+ $item ) | { "domain": . }
  | with(.domain[]; . |=  select(.selectors | length == 1) |= .selectors[0])
  |= load("base.yaml") + .'

{
  "enabled": true,
  "sign_authenticated": true,
  "sign_local": false,
  "try_fallback": false,
  "use_domain": "header",
  "use_redis": false,
  "use_esld": true,
  "allow_username_mismatch": true,
  "check_pubkey": true,
  "domain": {
    "example.org": {
      "path": "/dms/dkim/keys/example.org/mail-rsa/private.pem",
      "selector": "mail-rsa"
    },
    "example.test": {
      "selectors": [
        {
          "path": "/dms/dkim/keys/example.test/mail-ec/private.pem",
          "selector": "mail-ec"
        },
        {
          "path": "/dms/dkim/keys/example.test/mail-rsa/private.pem",
          "selector": "mail-rsa"
        }
      ]
    }
  }
}

No comments support/retained due to JSON, but that is not an issue if relying on this as internal config generated at container runtime.

The base.yaml with comments could still exist in the container and optionally be replaced by users config volume override 🤷‍♂️

Alternatively, UCL supports an .include "path/to/file.conf" macro.

It can handle merging of duplicates too, but not logic for the domain being a map of path + selector or an array of such maps under selectors field AFAIK?

Might not be an issue if you can always use the selectors array, including for single elements (I assume you can). Then you can disregard all the yq + YAML dance I guess and just .include the domains config(s) as separate files similar to the above approach? 👍

This will still potentially trigger a change detection right? As it compares to it's before/after checksums in the config volume?

STAGING_FILES are crated by scanning RSPAMD_DMS_D (i.e. /tmp/docker-mailserver/rspamd), so AFAIK it should not trigger another event (since the cp copies to a location not checked and chown does not change SHA sum IIRC.

It seems that DKIM keys aren't copied over to an internal location like the certs are? I'm not too concerned about that I guess 🤔

Yes. This could be changed in the future if we encounter problems, but it seems to be absolutely fine for now.

FWIW, the rspamd docs mention rspamadm template for generating configs from jinja templates if you wanted to take advantage of that 🤷‍♂️

Have not looked intot that yes.

About the Rspamd DKIM proposals: wow, that was very interesting! Funnily enough, I think we should first try the .include ... macros before dancing around with yq (even though I found the yq stuff cool). Thank you for sharing your ideas here. I will make sure to remember this location; the reason I didn't touch DKIM for Rspamd here any more than the --force option is that the PR is already pretty big. The --force option was due to a recent issue I stumbled upon. I know you'd like to merge the OpenDKIM and Rspamd DKIM scripts, and I can tackle this in the future; for now, I'd like this PR to stay as-is to not inflate it any more. For a future PR that tackles DKIM again, we back-reference to your comments here!

I know you'd like to merge the OpenDKIM and Rspamd DKIM scripts, and I can tackle this in the future; for now, I'd like this PR to stay as-is to not inflate it any more.
For a future PR that tackles DKIM again, we back-reference to your comments here!

Oh yeah I didn't mean to update this PR to apply the review comments here, definitely a separate PR 👍

I agree the include macro would be wiser. I'd have suggested that before typing up all the yq stuff if I had read the rest of the UCL readme first.... 😓

UPDATE: I got around to trying KCL out, and while it has some nice benefits this was a scenario that it didn't handle too well.

• While they've migrated from Python to Rust, the syntax for iteration and such isn't likely to shift from its Python heritage which is a big 👎 for me.
• It does seem to have some nice advantages for more basic feature set though.

For example, it might be nice for managing ENV / internal config with doc generation + type validation/constraints, even has linting and formatting tools 😎

KCL is primarily for defining configuration in a more featureful way, and then outputting YAML it seems. It can ingest YAML (kinda) too.

UPDATE2: KCL discussion got some feedback since and they have some fixes in a future release that improves viability of KCL for this: https://github.com/orgs/kcl-lang/discussions/804#discussioncomment-7430551

Regex not necessary?

Also think about the very likely case, when someone mounts something like /tmp/docker-mailserver/rspamd/overrideXd, which would also match 😆

There are other places where normal string comparison is done with =~, I did choose this just as an example.

Regardless of this, the rest of the PR looks good 👍

I think your example wouldn't work. When you match via ==, you'd need an asterisk before as well, because ${CHANGED} is basically dir/file1 file2 dir2/dir3/file3 and matching file2 would require == *file2*, but with regex, you can just use =~ file2, hence the regex IIRC.

Also think about the very likely case, when someone mounts something like /tmp/docker-mailserver/rspamd/overrideXd, which would also match 😆

Can you elaborate please? I don't see why this would be very likely :)

Can you elaborate please? I don't see why this would be very likely :)

It is not. That was just a joke 😆

Haha, I see 😆 I have always had a hard time telling irony in texts :D