Skip to content

[HAPROXY] use $SOURCEIP instead of $PROXIED_SRCIP#5305

Merged
HofiOne merged 23 commits intosyslog-ng:developfrom
HofiOne:haproxy-use-sourceip-instead-of-proxied-srcip
Apr 23, 2025
Merged

[HAPROXY] use $SOURCEIP instead of $PROXIED_SRCIP#5305
HofiOne merged 23 commits intosyslog-ng:developfrom
HofiOne:haproxy-use-sourceip-instead-of-proxied-srcip

Conversation

@HofiOne
Copy link
Collaborator

@HofiOne HofiOne commented Mar 28, 2025 •
edited
Loading

This is a large refactor of the HAProxy support, and preparations for protocol auto detection. It also changes the HAProxy support to use the standard source/destination addresses in LogMessage, instead of a proxy specific values e.g. $SOURCEIP instead of $PROXIED_SRCIP.

Short summary of the patches:

  • small refactors and cleanups (first 5 patches)
  • simplify TLS compression setup code, previously this crossed a lot of layers, whereas it can be a lot simpler
  • LogTransportTLS is becoming an adapter, e.g. instead of going directly to file descriptors, go through another layer of the LogTransport interface. This prepares for protocol auto detection.
  • introduces LogTransportStack level aux data, making it possible to enrich messages from the any of the Transports
  • adds the $SOURCEPORT macro
  • LogTransportHAProxy is changed so it invokes a log_transport_stack_switch() instead of doing this internally
  • has a big refactor of TransportMapperInet, as its various settings and options were becoming too difficult to read. This patch also adds a large light testcase to cover all possible transport() cases
  • contains an alternative fix for 482 (although pretty similar)
  • and few smaller patches

Backport of 361 by @bazsi

Depend on #5303

@HofiOne HofiOne force-pushed the haproxy-use-sourceip-instead-of-proxied-srcip branch 6 times, most recently from 03f19e9 to 9f2db58 Compare March 28, 2025 16:41
@kira-syslogng
Copy link
Contributor

kira-syslogng commented Mar 28, 2025

Build FAILURE

@HofiOne HofiOne force-pushed the haproxy-use-sourceip-instead-of-proxied-srcip branch 2 times, most recently from d975302 to f27454d Compare April 4, 2025 10:52
@HofiOne HofiOne mentioned this pull request Apr 4, 2025
Merged
@kira-syslogng
Copy link
Contributor

kira-syslogng commented Apr 4, 2025

Build FAILURE

@HofiOne
Copy link
Collaborator Author

HofiOne commented Apr 7, 2025

@kira-syslogng test this please test=functions/source-drivers/file-source-driver/multiple-lines;

@kira-syslogng
Copy link
Contributor

kira-syslogng commented Apr 7, 2025

Build FAILURE

@HofiOne HofiOne marked this pull request as ready for review April 11, 2025 10:07
@folti
Copy link
Contributor

folti commented Apr 12, 2025

@kira-syslogng retest this please

@kira-syslogng
Copy link
Contributor

kira-syslogng commented Apr 12, 2025

Build FAILURE

@HofiOne HofiOne force-pushed the haproxy-use-sourceip-instead-of-proxied-srcip branch from f27454d to 32a900e Compare April 17, 2025 13:28
@kira-syslogng
Copy link
Contributor

kira-syslogng commented Apr 17, 2025

Build FAILURE

@syslog-ng syslog-ng deleted a comment from github-actions bot Apr 22, 2025
bazsi added 11 commits April 23, 2025 14:10
@bazsi @HofiOne
light: allow the use of "auto" transport for syslog() source drivers
e5affed 
Signed-off-by: Balazs Scheidler <[email protected]>
Signed-off-by: Hofi <[email protected]>
@bazsi @HofiOne
logproto: rename prepare() methods to poll_prepare
5259267 
Signed-off-by: Balazs Scheidler <[email protected]>
Signed-off-by: Hofi <[email protected]>
@bazsi @HofiOne
transport-stack: add poll_prepare() methods for LogTransport and LogT…
8310e71 
…ransportStack

Signed-off-by: Balazs Scheidler <[email protected]>
Signed-off-by: Hofi <[email protected]>
@bazsi @HofiOne
transport-stack: add LogTransportStack level read/write/writev methods
5962341 
Signed-off-by: Balazs Scheidler <[email protected]>
Signed-off-by: Hofi <[email protected]>
@bazsi @HofiOne
transport-adapter: add free_method
5180d10 
Signed-off-by: Balazs Scheidler <[email protected]>
Signed-off-by: Hofi <[email protected]>
@bazsi @HofiOne
transport/tls-context: simplify compression setup
5ff0dec 
Previously TLS compression was enabled using an overly complicated mechanism
crossing a number of layers (TransportMapperInet -> TransportFactoryTLS ->
TLSSession -> SSL). This can be a lot simpler, which this patch
implements.

NOTE: compression will not work in most cases due to OpenSSL security
levels and this patch adds a warning about it.

Signed-off-by: Balazs Scheidler <[email protected]>
Signed-off-by: Hofi <[email protected]>
@bazsi @HofiOne
transport-tls: convert to a LogTransportAdapter
dbe9fb3 
Instead of going to the fd directly, wrap the lower-level LogTransport
instance into a BIO and use that. This implements proper stacking
for LogTransportTLS.

This adds the use of OpenSSL BIOs to wrap the lower level LogTransport
instance.

Signed-off-by: Balazs Scheidler <[email protected]>
Signed-off-by: Hofi <[email protected]>
@bazsi @HofiOne
transport-aux-data: add log_transport_aux_data_move()
d616f3f 
Signed-off-by: Balazs Scheidler <[email protected]>
Signed-off-by: Hofi <[email protected]>
@bazsi @HofiOne
transport-stack: make it possible to add aux data from the LogTranspo…
ce3a7e2 
…rtStack level

Signed-off-by: Balazs Scheidler <[email protected]>
Signed-off-by: Hofi <[email protected]>
@bazsi @HofiOne
logproto: call transport-stack level I/O methods
7676c39 
Signed-off-by: Balazs Scheidler <[email protected]>
Signed-off-by: Hofi <[email protected]>
@bazsi @HofiOne
logproto/logproto-auto-server: remove RFC6587 references from log mes…
26c555c 
…sages

The "auto" protocol can be applied to both syslog() and network(), so
it's not strictly RFC6587 related and it does not add too much information
anyway.

Signed-off-by: Balazs Scheidler <[email protected]>
Signed-off-by: Hofi <[email protected]>
bazsi added 12 commits April 23, 2025 14:10
@bazsi @HofiOne
templates: add $SOURCEPORT macro
1398e69 
Signed-off-by: Balazs Scheidler <[email protected]>
Signed-off-by: Hofi <[email protected]>
@bazsi @HofiOne
transport-haproxy: use the normal $SOURCEIP, $DESTIP, $DESTPORT macros
11d2605 
Instead of using proxy protocol specific name value pairs, set the
addresses in the message's saddr/daddr members.

This should be a lot faster and a lot easier to use.

Signed-off-by: Balazs Scheidler <[email protected]>
Signed-off-by: Hofi <[email protected]>
@bazsi @HofiOne
transport-haproxy: use the LogTransportStack->aux to save src/dst add…
23bfdcd 
…resses

Signed-off-by: Balazs Scheidler <[email protected]>
Signed-off-by: Hofi <[email protected]>
@bazsi @HofiOne
transport-haproxy: implement transport switch instead of changing bas…
9ec5c3d 
…e_index

Signed-off-by: Balazs Scheidler <[email protected]>
Signed-off-by: Hofi <[email protected]>
@bazsi @HofiOne
transport-haproxy: add more information to the proxy detection failur…
d0f24a8 
…e message

Signed-off-by: Balazs Scheidler <[email protected]>
Signed-off-by: Hofi <[email protected]>
@bazsi @HofiOne
afsocket: refactor TransportMapperInet transport setup logic
c90d1dc 
This reworks the various boolean members in TransportMapperInet that
control which logproto/transport we apply to a specific connection.

With these renames, it's much easier to follow what happens and why.

NOTE: there's a followup bugfix that fixes the same bug as syslog-ng#482.

Signed-off-by: Balazs Scheidler <[email protected]>
Signed-off-by: Hofi <[email protected]>
@bazsi @HofiOne
afsocket: transport(auto) with tls() should immediately start using TLS
87e8dfa 
"auto" has originally been planned to auto-detect TLS as well as framing
format, but at this point it does not do TLS auto-detection.

But this means that transport(auto) with tls() options set will start reading
data without SSL, e.g. the encrypted stuff will make it into the
messages received.

This patch fixes that for both the syslog() and the network() driver. The
only change is that delegate_tls_start_to_logproto is FALSE for the "auto"
case. This will be changed once the TLS auto detection feature is also
in.

Signed-off-by: Balazs Scheidler <[email protected]>
Signed-off-by: Hofi <[email protected]>
@bazsi @HofiOne
transport/transport-factory-haproxy: use factory for creating HAProxy…
116ec31 
… transports

Signed-off-by: Balazs Scheidler <[email protected]>
Signed-off-by: Hofi <[email protected]>
@bazsi @HofiOne
transport: free LogTransportBIO BIO_meth instance at shutdown
c4a1f20 
This was a one-off allocation, but it's better if it is freed.

Signed-off-by: Balazs Scheidler <[email protected]>
Signed-off-by: Hofi <[email protected]>
@bazsi @HofiOne
light: generalize test_pp_network and test_pp_syslog test cases
77d93c4 
Instead of just exercising the proxyprotocol try all valid transports, including
the "auto" variants.

Signed-off-by: Balazs Scheidler <[email protected]>
Signed-off-by: Hofi <[email protected]>
@bazsi @HofiOne
light: fix up files included in the dist
59c10b9 
Signed-off-by: Balazs Scheidler <[email protected]>
Signed-off-by: Hofi <[email protected]>
@bazsi @HofiOne
news: added news file
5b2e723 
Signed-off-by: Balazs Scheidler <[email protected]>
Signed-off-by: Hofi <[email protected]>
@HofiOne HofiOne force-pushed the haproxy-use-sourceip-instead-of-proxied-srcip branch from 32a900e to 5b2e723 Compare April 23, 2025 12:11
@kira-syslogng
Copy link
Contributor

kira-syslogng commented Apr 23, 2025

Build FAILURE

@HofiOne HofiOne merged commit f5ca76e into syslog-ng:develop Apr 23, 2025
29 of 30 checks passed
@HofiOne HofiOne deleted the haproxy-use-sourceip-instead-of-proxied-srcip branch April 23, 2025 13:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@kovgeri01 kovgeri01 kovgeri01 approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@HofiOne @kira-syslogng @folti @kovgeri01 @bazsi