feat(codecs): Add syslog encoder#23777
Conversation
Original commit from syedriko
This is only a temporary change to make the diffs for future commits easier to follow.
- Introduce a `Pri` struct with fields for severity and facility as enum values. - `Pri` uses `strum` crate to parse string values into their appropriate enum variant. - Handles the responsibility of encoding the two enum values ordinal values into the `PRIVAL` value for the encoder. - As `Facility` and `Severity` enums better represent their ordinal mapping directly - The `Fixed` + `Field` subtyping with custom deserializer isn't necessary. Parsing a string that represents the enum by name or its ordinal representation is much simpler. - Likewise this removes the need for the get methods as the enum can provide both the `String` or `u8` representation as needed.
`SyslogSerializer::encode()` has been simplified. - Only matching `Event::Log` is relevant, an `if let` bind instead of `match` helps remove a redundant level of nesting. - This method only focuses on boilerplate now, delegating the rest to `ConfigDecanter` (_adapt `LogEvent` + encoder config_) and `SyslogMessage` (_encode into syslog message string_). - This removes some complexity during actual encoding logic, which should only be concerned about directly encoding from one representation to another, not complimentary features related to Vector config or it's type system. The new `ConfigDecanter` is where many of the original helper methods that were used by `SyslogSerializer::encode()` now reside. This change better communicates the scope of their usage. - Any interaction with `LogEvent` is now contained within the methods of this new struct. Likewise for the consumption of the encoder configuration (instead of queries to config throughout encoding). - The `decant_config()` method better illustrates an overview of the data we're encoding and where that's being sourced from via the new `SyslogMessage` struct, which splits off the actual encoding responsibility (see next commit).
`SyslogSerializerConfig` has been simplified.
- Facility / Severity deserializer methods aren't needed, as per their prior refactor with `strum`.
- The `app_name` default is set via `decant_config()` when not configured explicitly.
- The other two fields calling a `default_nil_value()` method instead use an option value which encodes `None` into the expected `-` value.
- Everything else does not need a serde attribute to apply a default, the `Default` trait on the struct is sufficient.
- `trim_prefix` was removed as it didn't seem relevant. `tag` was also removed as it's represented by several subfields in RFC 5424 which RFC 3164 can also use.
`SyslogMessage::encode()` refactors the original PR encoding logic:
- Syslog Header fields focused, the PRI and final message value have already been prepared prior. They are only referenced at the end of `encode()` to combine into the final string output.
- While less efficient than `push_str()`, each match variant has a clear structure returned via the array `join(" ")` which minimizes the noise of `SP` from the original PR. Value preparation prior to this is clear and better documented.
- `Tag` is a child struct to keep the main logic easy to grok. `StructuredData` is a similar case.
No changes beyond relocating the code into a single file.
- Drop notes referring to original PR differences + StructuredData adaption references. None of it should be relevant going forward. - Revise some other notes. - Drop `add_log_source` method (introduced from the original PR author) in favor of using `StructuredData` support instead.
This should be simple and lightweight enough to justify for the DRY benefit? This way the method doesn't need to be duplicated redundantly. That was required because there is no trait for `FromRepr` provided via `strum`. That would require a similar amount of lines for the small duplication here. The `akin` macro duplicates the `impl` block for each value in the `&enums` array.
- `ConfigDecanter::get_message()` replaces the fallback method in favor of `to_string_lossy()` (a dedicated equivalent for converting `Value` type to a String type (_technically it is a CoW str, hence the follow-up with `to_string()`_)). - This also encodes the value better, especially for the default `log_namespace: false` as the message value (when `String`) is not quote wrapped, which matches the behaviour of the `text` encoder output. - Additionally uses the `LogEvent` method `get_message()` directly from `lib/vector-core/src/event /log_event.rs`. This can better retrieve the log message regardless of the `log_namespace` setting. - Encoding of RFC 5424 fields has changed to inline the `version` constant directly, instead of via a redundant variable. If there's ever multiple versions that need to be supported, it could be addressed then. - The RFC 5424 timestamp has a max precision of microseconds, thus this should be rounded and `AutoSi` can be used (_or `Micros` if it should have fixed padding instead of truncating trailing `000`_).
- The original PR author appears to have relied on a hard-coded timestamp key here. - `DateTime<Local>` would render the timestamp field with the local timezone offset, but other than that `DateTime<Utc>` would seem more consistent with usage in Vector, especially since any original TZ context is lost by this point? - Notes adjusted accordingly, with added TODO query for each encoding mode to potentially support configurable timezone.
- Move encoder config settings under a single `syslog` config field. This better mirrors configuration options for existing encoders like Avro and CSV. - `ConfigDecanter::value_by_key()` appears to accomplish roughly the same as the existing helper method `to_string_lossy()`. Prefer that instead. This also makes the `StructuredData` helper `value_to_string()` redundant too at a glance? - Added some reference for the priority value `PRIVAL`. - `Pri::from_str_variants()` uses the existing defaults for fallback, communicate that more clearly. Contextual note is no longer useful, removed.
To better communicate the allowed values, these two config fields can change from the `String` type to their appropriate enum type. - This relies on serde to deserialize the config value to the enum which adds a bit more noise to grok. - It does make `Pri::from_str_variants()` redundant, while the `into_variant()` methods are refactored to `deserialize()` with a proper error message emitted to match the what serde would normally emit for failed enum variant deserialization. - A drawback of this change is that these two config fields lost the ability to reference a different value path in the `LogEvent`. That'll be addressed in a future commit.
In a YAML config a string can optionally be wrapped with quotes, while a number that isn't quote wrapped will be treated as a number type. The current support was only for string numbers, this change now supports flexibility for config using ordinal values in YAML regardless of quote usage. The previous `Self::into_variant(&s)` logic could have been used instead of bringing in `serde-aux`, but the external helper attribute approach seems easier to grok/follow as the intermediary container still seems required for a terse implementation. The match statement uses a reference (_which requires a deref for `from_repr`_) to appease the borrow checker for the later borrow needed by `value` in the error message.
This seems redundant given the context? Mostly adds unnecessary noise. Could probably `impl Configurable` or similar to try workaround the requirement. The metadata description could generate the variant list similar to how it's been handled for error message handling?
Not sure if this is worthwhile, but it adopts error message convention elsewhere I've seen by managing them via Snafu.
Signed-off-by: Vitalii Parfonov <[email protected]>
…rity dynamic, payload_key optional Signed-off-by: Vitalii Parfonov <[email protected]>
Signed-off-by: Vitalii Parfonov <[email protected]>
Signed-off-by: Vitalii Parfonov <[email protected]>
Signed-off-by: Vitalii Parfonov <[email protected]>
Signed-off-by: Vitalii Parfonov <[email protected]>
pront
left a comment
There was a problem hiding this comment.
This looks pretty good!
- Please add edge case tests (missing fields, empty values)
- Validate that the documentation improvement I committed are correct
polarathene
left a comment
There was a problem hiding this comment.
Just a quick glance over I recalled some review feedback I received previously that's presumably still relevant for your iteration of the PR.
- Removed the obsolete `payload_key` field from `SyslogSerializerOptions` and simplified the payload retrieval logic. - Applied `#[serde(deny_unknown_fields)]` to the `SyslogSerializerOptions` struct, to enforces failing if configuration errors.
|
Hello @polarathene and @pront, thank you for the review!
|
|
Hey @vparfonov and @pront , does this open up the door to a dedicated syslog sink? |
Not yet, but after merging will be possible to use it to pair with |
|
Sweet! Thank you |
There was a problem hiding this comment.
Hi @vparfonov. I tried to fix the failing checks but it seems like I don't hash push permissions to this branch. I suggest git fetch && git merge origin/master and doing git checkout origin/master -- Cargo.lock && cargo check when you get a conflict to resolve it. Also, once master is merged you'll also need to run make generate-component-docs and cargo vdev build licenses.
@thomasqueirozb, thanks for pointing this out. I've attempted to run the generation commands, but I am hitting environment issues that I can't resolve quickly.
|
This error message has been on my todo list to fix since forever. You're missing It also looks like changes to |
got it now works, thanks
reverted, but it strange why it failed, only this changes was observed |
I have ran into that before with this same file. Not sure what is going on there - might be a difference between how formatting occurs inside the CI and |
Summary
This pull request introduces a new syslog encoder. This work is a continuation of the feature originally started in PR #21307.
The encoder is designed to be lean and performant, expecting users to perform complex data shaping with an upstream remap transform. It correctly handles both
RFC 5424andRFC 3164standards, including specific field length limitations, character sanitization, and security escaping.Key Features
Simple Configuration: The configuration uses standard
Option<ConfigTargetPath>for all fields.Flexible Parsing: facility and severity values read from the event are parsed intelligently, accepting either a string name (e.g., "user") or a number (e.g., 16), with case-insensitive matching for names.
Strict RFC Compliance:
Added logic to truncate
app_name,proc_id, andmsg_idto their specified maximum lengths forRFC 5424.Implemented robust truncation for the
RFC 3164TAGfield to ensure it never exceeds 32 characters.Added a sanitization step for
RFC 3164messages to remove non-printable ASCII characters.Implemented correct character escaping
(\, ", ])for structured data parameter values to prevent log injection.Unit tests: including parsing, truncation, sanitization, and escaping.
Vector configuration
How did you test this PR?
This plan covers the basic functionality of the syslog encoder for both
RFC5424andRFC3164, focusing on dynamic field resolution from a JSON source.Note: All tests assume thestdinsource is configured with decoding.codec = "json" to parse the input. Expected timestamps and hosts are illustrative.Test Case 1: RFC 5424 - field references
Verify that all configured fields are correctly read from a JSON event.
Config:
Input:
{"host": "my-host", "@timestamp": "2025-10-23T19:00:00.123456Z", "message": "hello world", "app": "my_app", "pid": "987", "mid": "REQ-1", "fac": "daemon", "sev": 3}Expected Output:
<27>1 2025-10-23T17:37:08.711556Z my-host my_app 987 REQ-1 - hello worldTest Case 2: RFC 3164 - fields references
Verify that all configured fields are correctly read for the legacy format.
Config:
Input:
{"host": "my-host", "@timestamp": "2025-10-23T19:00:00Z", "message": "hello legacy", "app": "legacy_app", "pid": "456", "fac": "user", "sev": 5}Expected Output:
<13>Oct 23 19:00:00 my-host legacy_app[456]: hello legacyTest Case 3: Field Parsing
Verify facility and severity are parsed from names (case-insensitive) and numbers.
Config:
Input 1 (Name):
{"fac": "local1", "sev": "warning"}Output 1:
<140>1 ...Input 2 (Number):
{"fac": 17, "sev": 4}Output 2:
<140>1 ...(same PRI)Input 3 (Uppercase):
{"fac": "LOCAL1", "sev": "WARNING"}Output 3:
<140>1 ...(same PRI)Input 4 (Mix):
{"fac": "LOCAL1", "sev": "WARNING"}Output 4:
<140>1 ...(same PRI)Test Case 4: Default Fallbacks
Verify the encoder uses defaults.
Config:
Input:
{"host": "my-host", "@timestamp": "2025-10-23T19:00:00Z", "message": "hello default"}Expected Output:
<14>1 2025-10-23T19:00:00.000000Z my-host vector - - - hello defaultChange Type
Is this a breaking change?
Does this PR include user facing changes?
no-changeloglabel to this PR.References
Notes
@vectordotdev/vectorto reach out to us regarding this PR.pre-pushhook, please see this template.make fmtmake check-clippy(if there are failures it's possible some of them can be fixed withmake clippy-fix)make testgit merge origin masterandgit push.Cargo.lock), pleaserun
make build-licensesto regenerate the license inventory and commit the changes (if any). More details here.