Skip to content

feat(new sink): add dedicated syslog sink for RFC 5424/3164 output#25055

Open
tot19 wants to merge 3 commits intovectordotdev:masterfrom
tot19:syslog
Open

feat(new sink): add dedicated syslog sink for RFC 5424/3164 output#25055
tot19 wants to merge 3 commits intovectordotdev:masterfrom
tot19:syslog

Conversation

@tot19
Copy link
Copy Markdown
Contributor

@tot19 tot19 commented Mar 27, 2026

Summary

Adds a new syslog sink that sends log events in syslog format (RFC 5424 or RFC 3164) over TCP, UDP, TCP with TLS, or Unix stream sockets.

This wraps Vector's existing socket transport utilities (TcpSinkConfig, UdpSinkConfig, UnixSinkConfig) with the syslog serializer from #23777/#24662. Syslog-specific options (rfc, facility, severity, app_name, proc_id, msg_id) are exposed under syslog for a first-class configuration experience.

The sink supports stream framing for TCP and Unix stream transports:

  • newline_delimited for RFC 6587 non-transparent framing.
  • octet_counting for RFC 6587 octet-counting framing.
  • TCP + TLS + RFC 5424 + octet_counting for RFC 5425 syslog over TLS.

This also hardens the syslog serializer for enterprise interoperability:

  • Sanitizes RFC 5424 header fields and uses NILVALUE for empty invalid header fields.
  • Normalizes structured-data names and avoids invalid empty SD names.
  • Accepts common facility/severity aliases and numeric strings used by rsyslog/syslog-ng/syslog decoders.
  • Rejects non-log events defensively instead of emitting an empty octet-counted frame.

Thanks to @vparfonov, @polarathene, and @syedriko for the syslog encoder work in #23777 that made this possible.

Vector configuration

TCP (RFC 5424):

sinks:
  my_syslog:
    type: syslog
    inputs:
      - my_source
    mode: tcp
    address: syslog.example.com:514
    syslog:
      rfc: rfc5424
      app_name: .app_name
      facility: .facility
      severity: .severity

UDP (RFC 3164):

sinks:
  my_syslog:
    type: syslog
    inputs:
      - my_source
    mode: udp
    address: syslog.example.com:514
    syslog:
      rfc: rfc3164

TCP with TLS (RFC 5425):

sinks:
  my_syslog:
    type: syslog
    inputs:
      - my_source
    mode: tcp
    address: syslog.example.com:6514
    tls:
      enabled: true
      ca_file: /path/to/ca.pem
    framing:
      method: octet_counting
    syslog:
      rfc: rfc5424

Unix stream socket:

sinks:
  my_syslog:
    type: syslog
    inputs:
      - my_source
    mode: unix_stream
    path: /path/to/syslog.sock

How did you test this PR?

Unit tests cover all transport modes, both RFC formats, custom PRI fields, TCP/TLS octet-counting, UDP datagram behavior, Unix stream delivery, and component compliance.

Codec tests cover RFC 5424/3164 formatting, PRI alias parsing, numeric PRI parsing, RFC 5424 header field sanitization, structured-data escaping, structured-data name truncation, empty-name handling, and non-log event rejection.

Validation commands run:

  • rustup run 1.92 cargo test -p codecs --features syslog --lib encoding::format::syslog::tests::
  • rustup run 1.92 cargo test --no-default-features --features sinks-syslog -p vector --lib sinks::syslog
  • rustup run 1.92 cargo check --no-default-features --features sinks-syslog -p vector --lib
  • make check-clippy
  • make check-markdown
  • ./scripts/cue.sh vet
  • ./scripts/check_changelog_fragments.sh
  • git diff --check

Change Type

  • New feature
  • Bug fix
  • Dependencies
  • Non-functional (chore, refactoring, docs)
  • Performance

Is this a breaking change?

  • Yes
  • No

Does this PR include user facing changes?

  • Yes. Changelog fragment added.
  • No.

References

Notes

  • Encoding is fixed to syslog. Users who need configurable encoding, field filtering, or timestamp formatting can use the socket sink with encoding.codec = "syslog".
  • Newline-delimited stream framing can split multiline messages. Use framing.method = "octet_counting" when messages can contain embedded newlines, such as stack traces or multiline JSON.
  • RFC 5425 requires octet-counting. For syslog over TLS compliance, use TCP, TLS, RFC 5424, and framing.method = "octet_counting".
  • RFC 3164 timestamps are emitted in UTC. This is documented rather than changed to local time for the initial sink to avoid host-local timezone ambiguity.
  • Unix socket support is stream-only. Unix datagram sockets, including typical Linux /dev/log sockets, are not supported by this sink. Use the socket sink with mode = "unix_datagram" for that case.
  • Message length policy is intentionally out of scope for v1. The sink currently sends the serialized event as produced by the codec and transport. A future max_message_length policy should define truncate/drop behavior, internal events/metrics, and transport-specific guidance.

Intentionally out of scope for this initial implementation

Please do not block this PR on these items unless maintainers want to expand scope:

  • No max-message-length truncate/drop policy yet.
  • No Unix datagram (/dev/log) mode in this dedicated sink.
  • No RELP, batching, compression, disk-spool-specific semantics, or syslog aggregator behavior.
  • No RFC 5424 UTF-8 BOM toggle.
  • No promotion of octet-counting into the shared codec framing layer.
  • No TLS hostname-verification integration test beyond the existing Vector TLS plumbing and the RFC5425 wire-format test.

@tot19 tot19 requested a review from a team as a code owner March 27, 2026 04:37
@tot19 tot19 mentioned this pull request Mar 27, 2026
@github-actions github-actions Bot added the domain: sinks Anything related to the Vector's sinks label Mar 27, 2026
@tot19 tot19 changed the title feat(syslog sink): add dedicated syslog sink for RFC 5424/3164 output feat(new sink): add dedicated syslog sink for RFC 5424/3164 output Mar 27, 2026
@tot19 tot19 requested a review from a team as a code owner May 1, 2026 18:07
@github-actions github-actions Bot added domain: external docs Anything related to Vector's external, public documentation work in progress labels May 1, 2026
@tot19 tot19 force-pushed the syslog branch 5 times, most recently from 993ab45 to 657a300 Compare May 1, 2026 19:22
@tot19
Copy link
Copy Markdown
Contributor Author

tot19 commented May 1, 2026

@pront FYI, this should be ready for maintainer review now.

Main scope:

  • Dedicated syslog sink for RFC 5424/3164 over TCP, UDP, TLS, and Unix stream.
  • RFC 6587 newline and octet-counting stream framing.
  • RFC 5425-compliant path via TCP + TLS + RFC5424 + octet-counting.
  • Serializer hardening for RFC header fields, structured-data names, and syslog facility/severity aliases.

Adds a new `syslog` sink that sends log events in syslog format over TCP,
UDP, or Unix sockets. This wraps the existing socket transport utilities
with the syslog codec hardcoded as the serializer, exposing syslog-specific
options (rfc, facility, severity, app_name, proc_id, msg_id) at the top
level for a first-class configuration experience.

Closes vectordotdev#6863

Co-Authored-By: Claude Opus 4.6 <[email protected]>
@github-actions github-actions Bot added the domain: ci Anything related to Vector's CI environment label May 4, 2026
tot19 added 2 commits May 4, 2026 13:32
Two issues prevented the rsyslog integration tests from running:

1. The `global(workDirectory="/tmp")` directive in the test rsyslog config
   conflicts with the upstream `rsyslog/rsyslog:2026-02` image's base
   config which already sets `workDirectory`. The duplicate parameter
   failed config validation and the container never started. Drop the
   line and let the image default stand.

2. The UDP tests passed wire-level but failed compliance with
   `Missing metric component_sent_bytes_total{protocol}`. The UDP path
   uses `register!(BytesSent::from(Protocol::UDP))` in `UdpSink::new`,
   which records the event name when the sink is constructed. The test
   helper built the sink before calling `run_and_assert_sink_compliance`,
   so `init_test` cleared the recorded event name and `register!` never
   fired again. Move the build inside the compliance future, mirroring
   `socket.rs::test_datagram`. TCP was unaffected because it emits via
   `emit!(SocketBytesSent)` inside the run loop.

Result: 6/6 syslog integration tests pass.
…and syslog-ng

The previous suite verified happy-path correctness against rsyslog over
plaintext UDP and TCP. Several realistic regression vectors weren't
exercised end-to-end. This commit closes the gaps with a mix of unit
and integration tests:

Unit tests (src/sinks/syslog.rs):
- tcp_finalizers_delivered_on_success: verifies BatchNotifier finalizers
  are marked Delivered after a successful TCP send.
- udp_finalizers_delivered_on_success: same for the UDP datagram path,
  guarding `send_and_emit` in src/sinks/util/datagram.rs.
- tcp_reconnect_after_server_close: forces a hard TCP disconnect mid-
  stream and confirms the sink reconnects to a freshly bound listener.
  Mirrors the existing socket sink reconnect test.

Integration tests (src/sinks/syslog/integration_tests.rs):
- tcp_octet_counting_rfc5424_with_proc_id_msg_id_structured_data_reaches_rsyslog:
  routes proc_id, msg_id, and a `structured_data` ObjectMap through the
  config layer to a real rsyslog and verifies the SD-element survives
  encoding intact. Catches regressions in `decant_config` that the
  codec-only unit tests can't see.
- tcp_tls_octet_counting_rfc5424_to_syslog_ng: RFC 5425 (syslog over
  TLS) interop against syslog-ng's TLS network transport. The
  rsyslog/rsyslog:2026-02 image ships only `lmnsd_ptcp.so`, so TLS
  testing lives under syslog-ng which has OpenSSL built in.
- udp_rfc5424_reaches_syslog_ng / tcp_octet_counting_rfc5424_reaches_syslog_ng:
  cross-daemon coverage. Different syslog implementations parse RFC
  3164/5424 differently in practice; running our wire output through a
  second receiver guards against regressions that only one daemon would
  flag.

Test infrastructure:
- New balabit/syslog-ng:4.8.3 service in compose.yaml with TCP TLS
  using the project test certs.
- tests/integration/syslog/data/syslog-ng.conf with `flags(store-raw-message)`
  on each source so `$RAWMSG` captures the wire-format line.
- New SYSLOG_NG_* env vars in test.yaml for runner addressing.

Result: 13/13 unit + 10/10 integration = 23/23 syslog sink tests pass.
@tot19
Copy link
Copy Markdown
Contributor Author

tot19 commented May 4, 2026

Added the following tests:

Unit tests (src/sinks/syslog.rs):

  • tcp_finalizers_delivered_on_success — verifies BatchNotifier finalizers are marked Delivered after a successful TCP send.
  • udp_finalizers_delivered_on_success — same for the UDP datagram path.
  • tcp_reconnect_after_server_close — forces a hard TCP disconnect mid-stream and confirms the sink reconnects to a freshly bound listener.

Integration tests (src/sinks/syslog/integration_tests.rs):

  • tcp_octet_counting_rfc5424_with_proc_id_msg_id_structured_data_reaches_rsyslog — routes proc_id, msg_id, and a structured_data ObjectMap through the config layer to a real rsyslog and verifies the SD-element survives encoding intact.
  • tcp_tls_octet_counting_rfc5424_to_syslog_ng — RFC 5425 (syslog over TLS) interop against syslog-ng's TLS network transport with octet-counted framing.
  • udp_rfc5424_reaches_syslog_ng — cross-daemon UDP coverage.
  • tcp_octet_counting_rfc5424_reaches_syslog_ng — cross-daemon TCP octet-counted coverage.

A new balabit/syslog-ng:4.8.3 service was added to the integration compose file (rsyslog's 2026-02 image only ships plaintext TCP, hence syslog-ng for TLS). All 13 unit + 10 integration tests pass.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

domain: ci Anything related to Vector's CI environment domain: external docs Anything related to Vector's external, public documentation domain: sinks Anything related to the Vector's sinks work in progress

Projects

None yet

Development

Successfully merging this pull request may close these issues.

New syslog sink

1 participant