Add no-piggyback-errors flag to syslog-parser() and syslog related source drivers.#245
Conversation
|
I'm thinking about retaining even more information in case parsing fails. The current behaviour drops everything, for instance the original sending IP address does seem useful when you want to diagnose parse errors. |
bazsi
force-pushed
the
retain-rawmsg-on-msg-parse-errors
branch
from
592694c to
1d360a7
Compare
bazsi
force-pushed
the
retain-rawmsg-on-msg-parse-errors
branch
from
1d360a7 to
dbd411f
Compare
bazsi
changed the title
bazsi
force-pushed
the
retain-rawmsg-on-msg-parse-errors
branch
from
dbd411f to
9e33b79
Compare
bazsi
changed the title
bazsi
force-pushed
the
retain-rawmsg-on-msg-parse-errors
branch
from
9e33b79 to
4862157
Compare
bazsi
changed the title
|
I am finished with the branch now and mostly happy how it shaped up. @mbonsack would be great to get your input. |
|
LGTM based on our review this past weekend. |
MrAnno
left a comment
MrAnno
left a comment
There was a problem hiding this comment.
LGTM otherwise.
Another idea that may probably be a separate PR but could be attached to this new flag:
We should allow a syslog-parser() mode where we don't fall back to parsing bsdsyslog when we fail to parse the message as IETF syslog. This could be useful for writing complex parser logic (I saw something like this assumed in @jszigetvari 's filterx codes).
it is already possible, there's a flag called 'no-rfc3164-fallback' |
Signed-off-by: Balazs Scheidler <[email protected]>
Signed-off-by: Balazs Scheidler <[email protected]>
Signed-off-by: Balazs Scheidler <[email protected]>
Just like rfc3164, make it possible to report rfc5424 style parsing issues as tags. Signed-off-by: Balazs Scheidler <[email protected]>
This tells syslog-ng not to wipe out the input message in syslog-parser() and
source encapsulated syslog parsers when there's a parse issue.
For example:
parser { syslog-parser(flags(syslog-protocol, no-piggyback-errors)); };
With that in place, $MSG retains its original value and various tags (e.g.
the syslog.* ones) report what kind of issues we detected with the messsage.
When the message is sent to a syslog-like destination, we will prepend
a header.
Basically this makes 5424 and 3164 parsing more similar. Previously 5424 parsing
reported its errors by completely wiping out the the existing log message
fields and adding a message attributed to "syslog-ng" itself. But this
message was coming from the same source and not internal().
Signed-off-by: Balazs Scheidler <[email protected]>
bazsi
force-pushed
the
retain-rawmsg-on-msg-parse-errors
branch
from
4862157 to
2ab3cd1
Compare
|
if you can re-approve @MrAnno we could merge this so the work on the dependant PR can continue. |
3 participants
Problem:
In case syslog-parser() fails, the entire log message is cleared: all name-value pairs, tags, along with partially parsed values already extracted from the message when the parsing failed. This also clears anything that was set prior to syslog-parser() and unrelated to syslog parsing (e.g. $SOURCEIP or $SOURCE or even $TRANSPORT).
Then, syslog-ng adds some error information to the message and then attributes it to itself, as if it was coming from syslog-ng itself, but not via the internal() driver, rather by the driver which actually receives the message.
This behaviour makes sense in cases where the syslog parser is embedded in source drivers, as in that case the message should not have any other things yet.
It is not intuitive with syslog-parser() though, as in this case, we might have requested store-raw-message or even associated various name-value pairs to the message already.
Solution
This PR adds a no-piggyback-errors flag to syslog-parser (both the embedded case and the syslog-parser case). This flag is not enabled by default, as that would be an incompatible change with syslog-parser().
With
no-piggyback-errorsthen the message will not be attributed to syslog-ng in case of errors. Actually it retains everything that was present at the time of the parse error, potentially things that were already extracted.So $MSG remains that was set (potentially the raw message), $HOST may or may not be extracted, likewise for $PROGRAM, $PID, $MSGID, etc.
The error is still indicated via $MSGFORMAT set to "syslog-error" (but note #250, which will change this to syslog:error).
Others:
This branch also: