Add kafka backend #1
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* added required configuration definition to allow setting up a kafka logger * added a logger backend to send the agents logs to kafka * updated the docker-compose to allow to run different backends
saryani
commented
Oct 4, 2024
| const ( | ||
| // DBString to format connection string to database for postgres | ||
| DBString = "host=%s port=%s dbname=%s user=%s password=%s sslmode=disable" | ||
| DBString = "host=%s port=%s dbname=%s user=%s password=%s sslmode=require" |
Owner
Author
There was a problem hiding this comment.
i will revert this line before creating PR against upstream, it's needed for our tests
Owner
Author
There was a problem hiding this comment.
I'm making some changes to make this configurable!
There was a problem hiding this comment.
We don't need this db change as the author already added them
This reverts commit f58a5f4.
zhuoyuan-liu
reviewed
Oct 4, 2024
|
|
||
| OSCTRL_TLS_LOGGER=db | ||
|
|
||
| LOG_KAFKA_BOOTSTRAP_SERVERS=my-kafka-bootstrap-server |
There was a problem hiding this comment.
Need be disabled by default for these config and add more comments for it
| #### Redis settings #### | ||
| - REDIS_HOST=osctrl-redis | ||
| #### Kafka settings #### | ||
| - LOG_KAFKA_BOOTSTRAP_SERVERS=${LOG_KAFKA_BOOTSTRAP_SERVERS} |
There was a problem hiding this comment.
Same here, it should be disabled by default.
| "failed to produce message to kafka topic '%s'. details: %s", l.config.Topic, err) | ||
| } | ||
| if debug { | ||
| log.Printf("message with key '%s' was sent to topic '%s' successfully\n%s", key, l.config.Topic, string(data)) |
There was a problem hiding this comment.
I think they updated the log for the application, try to merge with the upstream
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR will make it possible for osctrl to send the osquery-agent logs to a kafka topic. this means that these logs will no longer be pushed to the database, you need to set kafka as the log backend at init time otherwise it will log into database.