feat(new sink): add dedicated syslog sink for RFC 5424/3164 output#25055
Open
tot19 wants to merge 3 commits intovectordotdev:masterfrom
Open
feat(new sink): add dedicated syslog sink for RFC 5424/3164 output#25055tot19 wants to merge 3 commits intovectordotdev:masterfrom
tot19 wants to merge 3 commits intovectordotdev:masterfrom
Conversation
993ab45 to
657a300
Compare
Contributor
Author
|
@pront FYI, this should be ready for maintainer review now. Main scope:
|
Adds a new `syslog` sink that sends log events in syslog format over TCP, UDP, or Unix sockets. This wraps the existing socket transport utilities with the syslog codec hardcoded as the serializer, exposing syslog-specific options (rfc, facility, severity, app_name, proc_id, msg_id) at the top level for a first-class configuration experience. Closes vectordotdev#6863 Co-Authored-By: Claude Opus 4.6 <[email protected]>
Two issues prevented the rsyslog integration tests from running:
1. The `global(workDirectory="/tmp")` directive in the test rsyslog config
conflicts with the upstream `rsyslog/rsyslog:2026-02` image's base
config which already sets `workDirectory`. The duplicate parameter
failed config validation and the container never started. Drop the
line and let the image default stand.
2. The UDP tests passed wire-level but failed compliance with
`Missing metric component_sent_bytes_total{protocol}`. The UDP path
uses `register!(BytesSent::from(Protocol::UDP))` in `UdpSink::new`,
which records the event name when the sink is constructed. The test
helper built the sink before calling `run_and_assert_sink_compliance`,
so `init_test` cleared the recorded event name and `register!` never
fired again. Move the build inside the compliance future, mirroring
`socket.rs::test_datagram`. TCP was unaffected because it emits via
`emit!(SocketBytesSent)` inside the run loop.
Result: 6/6 syslog integration tests pass.
…and syslog-ng The previous suite verified happy-path correctness against rsyslog over plaintext UDP and TCP. Several realistic regression vectors weren't exercised end-to-end. This commit closes the gaps with a mix of unit and integration tests: Unit tests (src/sinks/syslog.rs): - tcp_finalizers_delivered_on_success: verifies BatchNotifier finalizers are marked Delivered after a successful TCP send. - udp_finalizers_delivered_on_success: same for the UDP datagram path, guarding `send_and_emit` in src/sinks/util/datagram.rs. - tcp_reconnect_after_server_close: forces a hard TCP disconnect mid- stream and confirms the sink reconnects to a freshly bound listener. Mirrors the existing socket sink reconnect test. Integration tests (src/sinks/syslog/integration_tests.rs): - tcp_octet_counting_rfc5424_with_proc_id_msg_id_structured_data_reaches_rsyslog: routes proc_id, msg_id, and a `structured_data` ObjectMap through the config layer to a real rsyslog and verifies the SD-element survives encoding intact. Catches regressions in `decant_config` that the codec-only unit tests can't see. - tcp_tls_octet_counting_rfc5424_to_syslog_ng: RFC 5425 (syslog over TLS) interop against syslog-ng's TLS network transport. The rsyslog/rsyslog:2026-02 image ships only `lmnsd_ptcp.so`, so TLS testing lives under syslog-ng which has OpenSSL built in. - udp_rfc5424_reaches_syslog_ng / tcp_octet_counting_rfc5424_reaches_syslog_ng: cross-daemon coverage. Different syslog implementations parse RFC 3164/5424 differently in practice; running our wire output through a second receiver guards against regressions that only one daemon would flag. Test infrastructure: - New balabit/syslog-ng:4.8.3 service in compose.yaml with TCP TLS using the project test certs. - tests/integration/syslog/data/syslog-ng.conf with `flags(store-raw-message)` on each source so `$RAWMSG` captures the wire-format line. - New SYSLOG_NG_* env vars in test.yaml for runner addressing. Result: 13/13 unit + 10/10 integration = 23/23 syslog sink tests pass.
Contributor
Author
|
Added the following tests: Unit tests (
Integration tests (
A new |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a new
syslogsink that sends log events in syslog format (RFC 5424 or RFC 3164) over TCP, UDP, TCP with TLS, or Unix stream sockets.This wraps Vector's existing socket transport utilities (
TcpSinkConfig,UdpSinkConfig,UnixSinkConfig) with the syslog serializer from #23777/#24662. Syslog-specific options (rfc,facility,severity,app_name,proc_id,msg_id) are exposed undersyslogfor a first-class configuration experience.The sink supports stream framing for TCP and Unix stream transports:
newline_delimitedfor RFC 6587 non-transparent framing.octet_countingfor RFC 6587 octet-counting framing.octet_countingfor RFC 5425 syslog over TLS.This also hardens the syslog serializer for enterprise interoperability:
Thanks to @vparfonov, @polarathene, and @syedriko for the syslog encoder work in #23777 that made this possible.
Vector configuration
TCP (RFC 5424):
UDP (RFC 3164):
TCP with TLS (RFC 5425):
Unix stream socket:
How did you test this PR?
Unit tests cover all transport modes, both RFC formats, custom PRI fields, TCP/TLS octet-counting, UDP datagram behavior, Unix stream delivery, and component compliance.
Codec tests cover RFC 5424/3164 formatting, PRI alias parsing, numeric PRI parsing, RFC 5424 header field sanitization, structured-data escaping, structured-data name truncation, empty-name handling, and non-log event rejection.
Validation commands run:
rustup run 1.92 cargo test -p codecs --features syslog --lib encoding::format::syslog::tests::rustup run 1.92 cargo test --no-default-features --features sinks-syslog -p vector --lib sinks::syslogrustup run 1.92 cargo check --no-default-features --features sinks-syslog -p vector --libmake check-clippymake check-markdown./scripts/cue.sh vet./scripts/check_changelog_fragments.shgit diff --checkChange Type
Is this a breaking change?
Does this PR include user facing changes?
References
syslogsink #6863Notes
socketsink withencoding.codec = "syslog".framing.method = "octet_counting"when messages can contain embedded newlines, such as stack traces or multiline JSON.framing.method = "octet_counting"./dev/logsockets, are not supported by this sink. Use thesocketsink withmode = "unix_datagram"for that case.max_message_lengthpolicy should define truncate/drop behavior, internal events/metrics, and transport-specific guidance.Intentionally out of scope for this initial implementation
Please do not block this PR on these items unless maintainers want to expand scope:
/dev/log) mode in this dedicated sink.