Skip to content

scl: add elasticsearch-datastream destination#178

Merged
MrAnno merged 2 commits intomainfrom
scl-elasticsearch-datastream
Jul 10, 2024
Merged

scl: add elasticsearch-datastream destination#178
MrAnno merged 2 commits intomainfrom
scl-elasticsearch-datastream

Conversation

@OverOrion
Copy link
Contributor

@OverOrion OverOrion commented Jun 25, 2024 •
edited
Loading

See https://www.elastic.co/guide/en/elasticsearch/reference/current/use-a-data-stream.html#add-documents-to-a-data-stream for details.

Sample trace log:

[2024-06-25T15:28:30.157427] cURL debug; worker='0', type='data_out', data='{"create":{ }}.{"PROGRAM":"prg00000","PRIORITY":"info","PID":"1234","MESSAGE":"seq: 0000000000, thread: 0000, runid: 1719322110, stamp: 2024-06-25T15:28:30 PADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADD","ISODATE":"2024-06-25T15:28:30+02:00","HOST":"localhost","FACILITY":"auth","@timestamp":"2024-06-25T15:28:30+02:00"}.'
[2024-06-25T15:28:30.157427] cURL debug; worker='0', type='ssl_data_in', data='.....'
[2024-06-25T15:28:30.157427] cURL debug; worker='0', type='ssl_data_in', data='.'
[2024-06-25T15:28:30.157427] cURL debug; worker='0', type='text', data='TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):.'
[2024-06-25T15:28:30.157427] cURL debug; worker='0', type='ssl_data_in', data='.'
[2024-06-25T15:28:30.157427] cURL debug; worker='0', type='ssl_data_in', data='.....'
[2024-06-25T15:28:30.157427] cURL debug; worker='0', type='ssl_data_in', data='.'
[2024-06-25T15:28:30.157427] cURL debug; worker='0', type='header_in', data='HTTP/1.1 200 OK..'
[2024-06-25T15:28:30.157427] cURL debug; worker='0', type='header_in', data='X-elastic-product: Elasticsearch..'
[2024-06-25T15:28:30.157427] cURL debug; worker='0', type='header_in', data='content-type: application/json..'
[2024-06-25T15:28:30.157427] cURL debug; worker='0', type='header_in', data='Transfer-Encoding: chunked..'
[2024-06-25T15:28:30.157427] cURL debug; worker='0', type='header_in', data='..'
[2024-06-25T15:28:30.157427] cURL debug; worker='0', type='data_in', data='190..{.  "errors" : false,.  "took" : 2,.  "items" : [.    {.      "create" : {.        "_index" : "ds-lo",.        "_id" : "kO2VT5ABNjqgUaS3aGHT",.        "_version" : 1,.        "result" : "created",.        "_shards" : {.          "total" : 2,.          "successful" : 1,.          "failed" : 0.        },.        "_seq_no" : 56,.        "_primary_term" : 1,.        "status" : 201.      }.    }.  ].}...0....'
[2024-06-25T15:28:30.157427] cURL debug; worker='0', type='text', data='Connection #0 to host localhost left intact.'
[2024-06-25T15:28:30.157427] http: HTTP response received; url='https://localhost:9200/ds-lo/_bulk?pretty', status_code='200', body_size='403', batch_size='1', redirected='0', total_time='0.009', worker_index='0', driver='output_quickstart_elasticsearch#0', location='syslog-ng.conf:45:5'

@OverOrion OverOrion force-pushed the scl-elasticsearch-datastream branch from 078fca0 to e56317f Compare June 25, 2024 13:40
@OverOrion OverOrion mentioned this pull request Jul 8, 2024
Merged
@OverOrion OverOrion marked this pull request as ready for review July 10, 2024 06:56
@MrAnno MrAnno merged commit dfbcfe0 into main Jul 10, 2024
@MrAnno MrAnno deleted the scl-elasticsearch-datastream branch July 10, 2024 09:35
@HofiOne HofiOne mentioned this pull request Aug 26, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@MrAnno MrAnno MrAnno approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@OverOrion @MrAnno