add ELK support by castorinop · Pull Request #331 · docker-mailserver/docker-mailserver

castorinop · 2016-09-23T19:06:23Z

* https://github.com/whyscream/postfix-grok-patterns * custom imput * override syslog filter.

tomav

I had a first try and here are my feedbacks.

Also, when I go to Kibana, I see No default index pattern. You must select or create one to continue.
How can we I get it configured?

tomav · 2016-09-23T21:48:38Z

+    cap_add:
+    - NET_ADMIN
+  elk:
+     build elk


Should be build: elk.
Got this error without :

ERROR: yaml.scanner.ScannerError: mapping values are not allowed here

tomav · 2016-09-23T21:49:36Z

+RUN mkdir /etc/logstash/patterns.d
+RUN curl -L https://raw.githubusercontent.com/whyscream/postfix-grok-patterns/master/postfix.grok > /etc/logstash/patterns.d/postfix.grok
+RUN curl -L https://raw.githubusercontent.com/whyscream/postfix-grok-patterns/master/50-filter-postfix.conf > /etc/logstash/conf.d/15-filter-postfix.conf
+


These include amavis grok?

added Amavis and Dovecot, but crash geoip plugin fail. working on..

tomav · 2016-09-23T21:50:37Z

    fi
  done
 fi
+if [ "$ENABLE_ELK" = 1 ]; then


I would name it ENABLE_ELK_FORWARDER.
Without forwarder, we can think that it launches ELK stack.

tomav · 2016-09-23T21:53:49Z

+ELK_PORT=${ELK_PORT:="10514"}
+ELK_HOST=${ELK_HOST:="elk"}
+echo "forward logs to ELK ($ELK_HOST:$ELK_PORT)"
+echo " *.* @$ELK_HOST:$ELK_PORT " > /etc/rsyslog.d/60-elk.conf


No filebeat client? (I'm not an ELK expert but it seemed the recommended way to forward logs last time I checked)

i try use the resources that have mailserver image, but i will try add firebeat client

It's an open question, don't worry. If it works without, let's try like that.

tomav · 2016-09-23T21:56:23Z

+if [ "$ENABLE_ELK" = 1 ]; then
+ELK_PORT=${ELK_PORT:="10514"}
+ELK_HOST=${ELK_HOST:="elk"}
+echo "forward logs to ELK ($ELK_HOST:$ELK_PORT)"


Message should be more explicit like:
Enabling log forwarding to ELK ($ELK_HOST:$ELK_PORT)

tomav · 2016-09-25T13:55:49Z

@castorinop Great work!
Let me know how I can run it (cf. comments above).
This is a huge addition to the stack 👍

tomav · 2016-09-26T14:15:04Z

Let me know regarding the first comment:

Also, when I go to Kibana, I see No default index pattern. You must select or create one to continue.
How can we I get it configured?

Thanks.

castorinop · 2016-09-26T14:36:20Z

Steps:

Create Index pattern
- Index name or pattern *
- Select Time-field name (refresh fields): received_at (appears until process some log)
- Create
Go to Discover and filter by fields.

TODO: a way to configure at startup.

add dovecot grok add geoip db add logstash geoip plugin

tomav · 2016-09-27T12:57:53Z

Strange build error. Restarted it.

tomav · 2016-09-27T13:58:50Z

Thanks, just tried and amavis entries get tag _grok_amavis_nomatch (and no separate field)

tomav · 2016-09-27T23:21:54Z

I modified the amavis pattern like the following:

AMAVIS \(%{DATA:amavis_id}\) %{DATA:amavis_action} %{DATA:amavis_status} {%{DATA:amavis_relaytype}},( %{GREEDYDATA:amavis_policybank})? \[%{IP:remote_ip}\]:%{POSINT:remote_port} \[%{IP:amavis_ip}\] <%{DATA:from}> -> <%{DATA:to}>(, quarantine: %{DATA:quarantine_id})?, Queue-ID: %{DATA:queue_id}(, Message-ID: <%{DATA:message_id}>)?(, mail_id: %{DATA:mail_id})?, Hits: %{NUMBER:amavis_hits}, size: %{POSINT:amavis_size}(, queued_as: %{DATA:amavis_queue_id})?(, dkim_sd=%{DATA:amavis_dkim})?, %{NUMBER:amavis_duration} ms

Can you try?

tomav · 2016-09-28T17:04:40Z

I tested. Amavis logs are now caught by Logstash.
But I get data twice in some fields (first time I use logstash, perhaps you will find the reason)

{
  "_index": "%{[@metadata][beat]}-2016.09.28",
  "_type": "%{[@metadata][type]}",
  "_id": "AVdxvR3qQzU_1fj9zO1-",
  "_score": null,
  "_source": {
    "message": [
      "<21>Sep 28 17:00:00 mail amavis[1361]: (01361-02-4) Passed CLEAN {RelayedOutbound}, LOCAL [127.0.0.1]:49712 [127.0.0.1] <[email protected]> -> <[email protected]>, Queue-ID: 5BC55D3F, Message-ID: <[email protected]>, mail_id: 9-BRNeg_CfFe, Hits: 1.459, size: 416, queued_as: 78B1DD43, 812 ms",
      "(01361-02-4) Passed CLEAN {RelayedOutbound}, LOCAL [127.0.0.1]:49712 [127.0.0.1] <[email protected]> -> <[email protected]>, Queue-ID: 5BC55D3F, Message-ID: <[email protected]>, mail_id: 9-BRNeg_CfFe, Hits: 1.459, size: 416, queued_as: 78B1DD43, 812 ms"
    ],
    "@version": "1",
    "@timestamp": "2016-09-28T17:00:00.000Z",
    "type": "syslog",
    "host": "172.20.0.3",
    "timestamp": "Sep 28 17:00:00",
    "hostname": "mail",
    "program": "amavis",
    "pid": "1361",
    "received_at": "2016-09-28T17:00:00.550Z",
    "received_from": "172.20.0.3",
    "syslog_severity_code": 5,
    "syslog_facility_code": 1,
    "syslog_facility": "user-level",
    "syslog_severity": "notice",
    "amavis_id": [
      "01361-02-4",
      "01361-02-4"
    ],
    "amavis_action": [
      "Passed",
      "Passed"
    ],
    "amavis_status": [
      "CLEAN",
      "CLEAN"
    ],
    "amavis_relaytype": [
      "RelayedOutbound",
      "RelayedOutbound"
    ],
    "amavis_policybank": [
      "LOCAL",
      "LOCAL"
    ],
    "remote_ip": [
      "127.0.0.1",
      "127.0.0.1"
    ],
    "remote_port": [
      "49712",
      "49712"
    ],
    "amavis_ip": [
      "127.0.0.1",
      "127.0.0.1"
    ],
    "from": [
      "[email protected]",
      "[email protected]"
    ],
    "to": [
      "[email protected]",
      "[email protected]"
    ],
    "queue_id": [
      "5BC55D3F",
      "5BC55D3F"
    ],
    "message_id": [
      "[email protected]",
      "[email protected]"
    ],
    "mail_id": [
      "9-BRNeg_CfFe",
      "9-BRNeg_CfFe"
    ],
    "amavis_hits": [
      1.459,
      1.459
    ],
    "amavis_size": [
      416,
      416
    ],
    "amavis_queue_id": [
      "78B1DD43",
      "78B1DD43"
    ],
    "amavis_duration": [
      "812",
      "812"
    ],
    "tags": [
      "_grok_amavis_success"
    ]
  },
  "fields": {
    "received_at": [
      1475082000550
    ],
    "@timestamp": [
      1475082000000
    ]
  },
  "highlight": {
    "program": [
      "@kibana-highlighted-field@amavis@/kibana-highlighted-field@"
    ],
    "message": [
      "<21>Sep 28 17:00:00 mail @kibana-highlighted-field@amavis@/kibana-highlighted-field@[1361]: (01361-02-4) Passed CLEAN {RelayedOutbound}, LOCAL [127.0.0.1]:49712 [127.0.0.1] <[email protected]> -> <[email protected]>, Queue-ID: 5BC55D3F, Message-ID: <[email protected]>, mail_id: 9-BRNeg_CfFe, Hits: 1.459, size: 416, queued_as: 78B1DD43, 812 ms"
    ]
  },
  "sort": [
    1475082000550
  ]
}

Other point in the elk log, I see sometimes this error:

elk_1   | [2016-09-28 17:02:04,602][DEBUG][action.fieldstats        ] [Count Abyss] [.kibana][0], node[CagZdtjcRlWvnRnc64peow], [P], v[2], s[STARTED], a[id=zRwMmOC8Tk20F0Y5i0MmQg]: failed to execute [org.elasticsearch.action.fieldstats.FieldStatsRequest@26ac6c92]
elk_1   | RemoteTransportException[[Count Abyss][172.20.0.2:9300][indices:data/read/field_stats[s]]]; nested: IllegalArgumentException[field [received_at] doesn't exist];
elk_1   | Caused by: java.lang.IllegalArgumentException: field [received_at] doesn't exist
elk_1   |   at org.elasticsearch.action.fieldstats.TransportFieldStatsTransportAction.shardOperation(TransportFieldStatsTransportAction.java:166)
elk_1   |   at org.elasticsearch.action.fieldstats.TransportFieldStatsTransportAction.shardOperation(TransportFieldStatsTransportAction.java:54)
elk_1   |   at org.elasticsearch.action.support.broadcast.TransportBroadcastAction$ShardTransportHandler.messageReceived(TransportBroadcastAction.java:282)
elk_1   |   at org.elasticsearch.action.support.broadcast.TransportBroadcastAction$ShardTransportHandler.messageReceived(TransportBroadcastAction.java:278)
elk_1   |   at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77)
elk_1   |   at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:376)
elk_1   |   at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
elk_1   |   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
elk_1   |   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
elk_1   |   at java.lang.Thread.run(Thread.java:745)

tomav · 2016-09-28T21:27:45Z

Same duplicates with postfix grok. (dovecot doesn't seem to be affected):

{
  "_index": "%{[@metadata][beat]}-2016.09.28",
  "_type": "%{[@metadata][type]}",
  "_id": "AVdyqW2mU4UDR6NXbF_X",
  "_score": null,
  "_source": {
    "message": [
      "<22>Sep 28 21:18:08 mail postfix/qmgr[1375]: EEA76D52: from=<[email protected]>, size=684, nrcpt=1 (queue active)",
      "EEA76D52: from=<[email protected]>, size=684, nrcpt=1 (queue active)"
    ],
    "@version": "1",
    "@timestamp": "2016-09-28T21:18:08.000Z",
    "type": "syslog",
    "host": "172.20.0.3",
    "timestamp": "Sep 28 21:18:08",
    "hostname": "mail",
    "program": "postfix/qmgr",
    "pid": "1375",
    "received_at": "2016-09-28T21:18:08.092Z",
    "received_from": "172.20.0.3",
    "syslog_severity_code": 5,
    "syslog_facility_code": 1,
    "syslog_facility": "user-level",
    "syslog_severity": "notice",
    "postfix_queueid": [
      "EEA76D52",
      "EEA76D52"
    ],
    "tags": [
      "_grok_postfix_success"
    ],
    "postfix_from": [
      "[email protected]",
      "[email protected]"
    ],
    "postfix_size": [
      684,
      684
    ],
    "postfix_nrcpt": [
      1,
      1
    ]
  },
  "fields": {
    "received_at": [
      1475097488092
    ],
    "@timestamp": [
      1475097488000
    ]
  },
  "sort": [
    1475097488092
  ]
}

add template config replace rsyslog with filebeat.

castorinop · 2016-09-29T20:17:29Z

I guess that duplicate logs problem is syslog related.
I try with filebeat client, and work very well.

tomav · 2016-09-29T20:28:01Z

Restarted timed out build.

tomav · 2016-09-29T20:48:36Z

Oh yeah, works great! Congrats @castorinop.
There's still the error:

RemoteTransportException[[Set][172.20.0.2:9300][indices:data/read/field_stats[s]]]; nested: IllegalArgumentException[field [received_at] doesn't exist];
Caused by: java.lang.IllegalArgumentException: field [received_at] doesn't exist
    at org.elasticsearch.action.fieldstats.TransportFieldStatsTransportAction.shardOperation(TransportFieldStatsTransportAction.java:166)
    at org.elasticsearch.action.fieldstats.TransportFieldStatsTransportAction.shardOperation(TransportFieldStatsTransportAction.java:54)
    at org.elasticsearch.action.support.broadcast.TransportBroadcastAction$ShardTransportHandler.messageReceived(TransportBroadcastAction.java:282)
    at org.elasticsearch.action.support.broadcast.TransportBroadcastAction$ShardTransportHandler.messageReceived(TransportBroadcastAction.java:278)
    at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77)
    at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:376)
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

This is not blocking, so I'll merge this PR.
If you can have a look in the next few days. I use the make fixtures to generate emails end log error show quickly.

The last missing thing, if a Wiki entry regarding ELK.

I'll create 2 new issues to log.

Thanks again for your work!

tomav · 2016-09-29T20:50:56Z

#336 & #337

@tomav

* add support to forward logs to ELK stack. * from docker elk customize image with * https://github.com/whyscream/postfix-grok-patterns * custom imput * override syslog filter. * fix typo. * Explicit forwarder vars and messages. * add amavis grok * add dovecot grok * add geoip db * add logstash geoip plugin * add custom amavis grok from @tomav. * switch to filebeats input * refactor syslog filter * add filebeat * add template config * replace rsyslog with filebeat.

castorinop added 2 commits September 23, 2016 16:02

add support to forward logs to ELK stack.

e08d7c2

from docker elk customize image with

0380938

* https://github.com/whyscream/postfix-grok-patterns * custom imput * override syslog filter.

castorinop mentioned this pull request Sep 23, 2016

Monitoring tools #266

Closed

tomav reviewed Sep 23, 2016

View reviewed changes

tomav added the wip - waiting for contributor action label Sep 24, 2016

castorinop added 2 commits September 26, 2016 10:20

fix typo.

c2ffca1

Explicit forwarder vars and messages.

c59c422

add amavis grok

64a9c18

add dovecot grok add geoip db add logstash geoip plugin

tomav assigned castorinop Sep 28, 2016

add custom amavis grok from @tomav.

8392234

castorinop added 3 commits September 29, 2016 16:14

switch to filebeats input

e7b034b

refactor syslog filter

68c4233

add filebeat

1dc08e4

add template config replace rsyslog with filebeat.

tomav removed the wip - waiting for contributor action label Sep 29, 2016

tomav merged commit e4bab5b into docker-mailserver:master Sep 29, 2016

tomav mentioned this pull request Oct 11, 2016

Feature: Add (daily/weekly/...) summary of mailserver by parsing logs #349

Closed

Uh oh!

Conversation

castorinop commented Sep 23, 2016 • edited by tomav Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tomav left a comment

Choose a reason for hiding this comment

Uh oh!

tomav Sep 23, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tomav Sep 23, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tomav commented Sep 25, 2016

Uh oh!

tomav commented Sep 26, 2016

Uh oh!

castorinop commented Sep 26, 2016

Uh oh!

tomav commented Sep 27, 2016

Uh oh!

tomav commented Sep 27, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tomav commented Sep 27, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tomav commented Sep 28, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tomav commented Sep 28, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

castorinop commented Sep 29, 2016

Uh oh!

tomav commented Sep 29, 2016

Uh oh!

tomav commented Sep 29, 2016

Uh oh!

tomav commented Sep 29, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

castorinop commented Sep 23, 2016 •

edited by tomav

Loading

tomav Sep 23, 2016 •

edited

Loading

tomav Sep 23, 2016 •

edited

Loading

tomav commented Sep 27, 2016 •

edited

Loading

tomav commented Sep 27, 2016 •

edited

Loading

tomav commented Sep 28, 2016 •

edited

Loading

tomav commented Sep 28, 2016 •

edited

Loading