Context

Vector for easier and better documented configuration, offering more flexible log management than the current rsyslog package supports (syslog focused only):

• Forwarding logs to external instance of Fail2Ban: [FR] Postscreen attack vectors should be banned by F2B in the default config #3178 (comment)
• Similar brief mention of adopting Vector for better log management with Redis and Rspamd (also briefly touches on syslog support, and some other redis config to consider unrelated to logging):
  • [BUG] rspamd forgets it learned spam and ham when docker-compose down and up is issued #3138 (comment)
  • [BUG] rspamd forgets it learned spam and ham when docker-compose down and up is issued #3138 (comment)
  • rspamd: rename ENABLE_REDIS & add persistence for Redis #3143 (review)
• Process to approach the change discussed here: config: ensure SASL socket file is not inside a volume mount #3131 (comment)
• Overview / Comparision to rsyslog for why to adopt Vector detailed here: fix: Avoid creating an unnecessary syslog socket for Postfix #3134 (comment)

Another maintainer approving interest in adopting Vector: #3134 (comment)

Description

This has been considered for a while. I just haven't properly documented / discussed an issue for going forward.

It's not likely to be a quick and easy change to implement, tests will likely need to be updated and current logging configuration reviewd/revised. Docs should also be added to better communicate the support to users that want to leverage it.

Alternatives

I think there was some improvements that could be done with rsyslog config, but beyond that there were not many other appealing options within a container that I came across.

• A while back I considered remote_syslog2 but had concerns with the lack of regular maintenance.
• Vector looks very nice but needs further investigation to verify as a suitable fit for the project that actually provides real value to ship with the image without increasing burden to maintainers.

Applicable Users

Anyone that needs better control of log management than we currently have. I am particularly interested in it as our current docs are lacking in this area, our log support with DMS is rather limited AFAIK.

IIRC, there was a concern with how supervisord handled logging which @georglauterbach raised upstream but his fix was rejected. I believe this was related to ANSI-based coloured logs and a workaround was implemented at least for docker logs stdout. With Vector the log level can be treated as metadata or streams, similar to what syslog supports, but IIRC syslog was less flexibility on "facilities"/tagging for grouping streams, especially when handling multi-line logs.

Are you going to implement it?

Yes, because I know the probability of someone else doing it is low and I can learn from it.

What are you going to contribute?

If this is ever contributed, it's probably going to be handled by me 😅

This is a low priority task due to effort required and ROI for users + maintainers vs other tasks in my backlog. I'm raising this FR for the benefit of others that may want to try tackle it if I don't get around to it.

For reference this PR adjusted services to centralize logging to /var/log/mail directory (which we support as a volume to persist for logs).

• Later supervisord was integrated and introduced another location that services output log files to (although these often appear empty / minimal as services get configured to log elsewhere instead of stdout?).
  • This is mentioned in a reverted change (also via collapsed comment of PR description) which changed the output of Postfix logs to /dev/stdout instead of syslog (to be handled by the postlogd service in master.cf), which would then have supervisord write the log to a postfix.log file in the supervisord directory in /var/log.
  • However, since tests often relied on /var/log/mail/mail.log to grep for log entries across multiple services this resulted in many test failures.
  • Vector could better support an isolated log stream and the combined log stream, while preserving the metadata for severity and services that is lost after writing syslog output to a file.
• The fail2ban log for services I believe also differed since, with current monitored services enabled here.

There are some concerns (as noted by the linked context resources).

• Syslog Sink support should probably be implemented first (this provides support for Vector to output logs in the syslog format like rsyslog does, making rsyslog easier to replace).
  • Until then you can forward from rsyslog, but I recall some issues with that like this issue.
  • Many existing services in DMS can easily forward logs into Vector via the Syslog Source input, where they can be transformed if necessary, before output/forwarded to a sink(s).
• I recall the binary size for Vector being a bit big (it bundles many features that can be optionally compiled and we probably don't need many of them), depends how much the added weight to image size matters for approval... May require a custom build if that's the case 🤷‍♂️
  • Some basic tunables share some size comparisons here
  • Feature opt-out would need to be experimented with the Cargo.toml? (seems there is some cargo build flag support, and building with vendored (external?) deps)
  • Size was less than 15MB back in 2019, and could be compressed (such as with UPX) for additional reduction.

Feature is better to delay until Dockerfile sees some more housekeeping and the testsuite is converted to the compose.yaml revision (presently still converting tests to setup.bash helper + refactoring tests)