{"id":99664,"date":"2019-11-01T07:00:51","date_gmt":"2019-11-01T05:00:51","guid":{"rendered":"https:\/\/www.javacodegeeks.com\/?p=99664"},"modified":"2019-10-30T10:16:32","modified_gmt":"2019-10-30T08:16:32","slug":"elastic-stack-introduction","status":"publish","type":"post","link":"https:\/\/www.javacodegeeks.com\/2019\/11\/elastic-stack-introduction.html","title":{"rendered":"Elastic Stack Introduction"},"content":{"rendered":"<p>When you are running software that is critical for your company you can&#8217;t have logs just for analysing things that happened some time ago, having the client tell you that your application is broken while you didn&#8217;t even know what was happening is a real problem. One of the ways to deal with that is using monitoring and logging.<\/p>\n<p>Most applications will have logging, even if it is just a text file that will require the use of tools like grep or tail to see what&#8217;s going on. But that&#8217;s a very passive way to see the logs, as you have to take action, and probably you will only look at the logs when there&#8217;s something wrong. What you need is to have your logs tell you what is happening, and today we are going to build the first step for that.<\/p>\n<p>By the title you already probably know that we are going to talk about the <code>Elasticstack<\/code>. My intention with this post is to provide the basics way to start collecting and analysing your logs without you having to deal with all the hardships that I went through.<\/p>\n<h3 class=\"wp-block-heading\">The Elastic Stack<\/h3>\n<p>Before known as ELK, the Elastic Stack is a set of tools that help you to <code>collect<\/code>, <code>structure<\/code>, <code>store<\/code>, <code>analyse<\/code> and also helps to create <code>actions<\/code> for certain situations.<\/p>\n<p>The Elastic Stack is composed of 4 components:<\/p>\n<ul class=\"wp-block-list\">\n<li><code>Kibana<\/code> is the visualisation tool that reads data from <code>ElasticSearch<\/code>. You can create dashboards or make queries to <code>ElasticSearch<\/code> manually.<\/li>\n<li><code>ElasticSearch<\/code> is the store for the logs. You can send logs from <code>Beats<\/code> or <code>Logstash<\/code> and it&#8217;s stored in an index. You can have multiple indexes to store data from multiple sources.<\/li>\n<li><code>Logstash<\/code> is the app that takes care of the logs, you can parse the logs into more useful data and send it to <code>ElasticSearch<\/code>.<\/li>\n<li><code>Beats<\/code> or <code>Filebeat<\/code> is a lightweight tool that reads the logs and sends them to <code>ElasticSearch<\/code> or <code>Logstash<\/code>. The only purpose of this tool is to read the log files, it can&#8217;t do any complex operation with it. If you need to do a complex operation then you can send that log to <code>Logstash<\/code> for it to parse it into the desired information.<\/li>\n<\/ul>\n<h2 class=\"wp-block-heading\">Logstash<\/h2>\n<p>We are starting with <code>Logstash<\/code> since it&#8217;s how you collect and parse your logs (Yes, I mentioned <code>Beats<\/code> but you don&#8217;t need that to start).<\/p>\n<p><code>Logstash<\/code> is a log processor and retriever. The main feature of <code>Logstash<\/code> is to give structure to unstructured log files, there are three steps for processing a log:<\/p>\n<ul class=\"wp-block-list\">\n<li>Input &#8211; Receiving or Fetching the log.<\/li>\n<li>Filter &#8211; Processing or filtering.<\/li>\n<li>Output &#8211; Sending the log to a permanent store.<\/li>\n<\/ul>\n<h3 class=\"wp-block-heading\">Input<\/h3>\n<p><code>Logstash<\/code> can retrieve data not only from log files, but it can get data from multiple sources like:<\/p>\n<ul class=\"wp-block-list\">\n<li>Text files<\/li>\n<li>Databases<\/li>\n<li>Shell commands<\/li>\n<li>Queues<\/li>\n<li>Http Requests<\/li>\n<\/ul>\n<p>If you want to see all the input plugins that <code>Logstash<\/code> supports check the <a href=\"https:\/\/www.elastic.co\/guide\/en\/logstash\/current\/input-plugins.html\">docs<\/a>.<\/p>\n<p>Right now, the inputs that I&#8217;ve been using and I will explain how to use are:<\/p>\n<ul class=\"wp-block-list\">\n<li>Text files<\/li>\n<li>Databases<\/li>\n<li>Shell commands<\/li>\n<\/ul>\n<h4 class=\"wp-block-heading\">Text File Input<\/h4>\n<p>One of the most common ways to store logs is in text files that sit somewhere in the machine, usually <code>\/var\/log<\/code>. <code>Logstash<\/code> has a plugin that reads the file and keeps watching for new lines just like <code>tail -f<\/code>.<\/p>\n<p>To use the plugin it&#8217;s very straightforward, you just have to add the path of the file and Logstash will take care of the rest, don&#8217;t worry about rotating the files, the plugin knows how to deal with that too.<\/p>\n<p>The usage of the <code>file<\/code> plugin is like this:<\/p>\n<pre class=\"wp-block-preformatted brush:java\">input {\n  file {\n    path =&gt; \/var\/log\/app\/realworld.log\n    id =&gt; realworld\n  }\n}<\/pre>\n<p>In the way that we configured Logstash above, every line will be a log entry. But sometimes our logs are not that simple and we have things like Stack traces or we write JSON into the logs. In this case we need them to be together to make sense and that&#8217;s why Logstash provides <code>codecs<\/code>, which are a way to extend the input plugins. One of those codecs is <code>Multiline<\/code>.<\/p>\n<p>For example, below we have the logs for an application that uses Spring and when we have a Stack Trace we want to group it just in one line.<\/p>\n<pre class=\"wp-block-preformatted brush:java\">2019-08-18 18:31:53.845 INFO 6724 --- [ main] o.s.t.web.servlet.TestDispatcherServlet : FrameworkServlet '': initialization completed in 17 ms\ncom.andre2w.transaction.TransactionTooOldException\n  at com.andre2w.transaction.TransactionService.validateTimestamp(TransactionService.java:46)\n  at com.andre2w.transaction.TransactionService.add(TransactionService.java:29)\n  at com.andre2w.controllers.TransactionController.create(TransactionController.java:42)\n  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n  at java.lang.reflect.Method.invoke(Method.java:498) \n2019-08-18 18:31:53.891 INFO 6724 --- [ main] o.s.b.t.m.w.SpringBootMockServletContext : Initializing Spring FrameworkServlet ''\n2019-08-18 18:31:53.892 INFO 6724 --- [ main] o.s.t.web.servlet.TestDispatcherServlet : FrameworkServlet '': initialization started<\/pre>\n<p>So if we want to capture the Stack Trace from our logs we can do a simple regex to group everything starting with space.<\/p>\n<pre class=\"wp-block-preformatted brush:java\">input {\n  file {\n    path =&gt; \/var\/log\/app\/realworld.log\n    id =&gt; realworld\n    codec =&gt; multiline {\n      pattern =&gt; \"^\\s\"\n      what =&gt; \"previous\"\n    }\n  }\n}<\/pre>\n<p>But in this case we can go even further. As these logs are from an application using Spring and all the logs they follow a certain format, we can group everything that doesn&#8217;t match that format which will include stack traces, JSON and objects.<\/p>\n<pre class=\"wp-block-preformatted brush:java\">input {\n  file {\n    path =&gt; \/var\/log\/app\/realworld.log\n    id =&gt; realworld\n\n    codec =&gt; multiline {\n      pattern =&gt; \"\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2}\\.\\d{3} \\w+ \\d+ --- \\[\\s+\\w+]\"\n      negate =&gt; true\n      what =&gt; \"next\"\n    }\n  }\n}<\/pre>\n<p>What does <code>negate<\/code> and <code>what<\/code> mean?<\/p>\n<p>Negate is the option to define if you want to group lines where the regex does or doesn&#8217;t match. In the first example the field <code>negate<\/code> is omitted, so it has the default value <code>False<\/code>. <code>negate =&gt; false<\/code> means that lines will be grouped together when the regex <strong>IS<\/strong> matched. <code>negate =&gt; true<\/code> means that lines will be grouped together when the regex <strong>IS NOT<\/strong> matched.<\/p>\n<p>The <code>what<\/code> is how Logstash will group the requests. <code>previous<\/code> means that the matched line will group with the previous line and <code>next<\/code> it&#8217;s pretty much obvious what it does.<\/p>\n<h4 class=\"wp-block-heading\">Database<\/h4>\n<p>Since Logstash is running using Java, you have access to any database supported by <code>JDBC<\/code>. For Postgres, you can see the number of locks querying the table <code>pg_locks<\/code>.<\/p>\n<p>The configuration for something like that would be:<\/p>\n<pre class=\"wp-block-preformatted brush:java\">input {\n  jdbc {\n    jdbc_connection_string =&gt; \"jdbc:postgres:\/\/localhost:5432\/main\"\n    jdbc_user =&gt; \"postgres\"\n    jdbc_password =&gt; \"postgres\"\n    jdbc_driver_class =&gt; \"org.postgresql.Driver\"\n    statement_filepath =&gt; \"\/etc\/logstash\/queries\/query-locks.sql\"\n    schedule =&gt; \"* * * * *\"\n    tags =&gt; [\"jdbc\",\"staging-1\",\"query-locks\", \"main\"]\n  }\n}<\/pre>\n<p>The fields starting <code>jdbc_<\/code> hold the information to connect to the database. Then we have <code>statement_filepath<\/code>, this points to the file with the query that you want <code>Logstash<\/code> to run, you can also use the <code>statement<\/code> option for more ad-hoc queries. Finally, we have the <code>schedule<\/code> option, this is the frequency that you want to run the query, the value is based on the Linux crontab, in this example it would be running every minute.<\/p>\n<p><code>tags<\/code> help you to identify the query that you are running. It&#8217;s just an array of strings, you can add anything.<\/p>\n<p><code>Logstash<\/code> will parse all the fields and send it to the datastore defined in the output.<\/p>\n<p>Something to keep an eye on when using the JDBC input is that you need the jar in the Logstash classpath. In the Logstash install folder, you go to <code>libs\/jars<\/code> and add the jar for the database driver.<\/p>\n<h4 class=\"wp-block-heading\">Exec<\/h4>\n<p>Sometimes you might want to get a piece of data from a source that <code>Logstash<\/code> doesn&#8217;t support very well. In this case you can use the <code>exec<\/code> input plugin that executes a command-line application and gets the result as the logline.<\/p>\n<pre class=\"wp-block-preformatted brush:java\">input { \n  exec {\n    command =&gt; \"\/usr\/bin\/retrieve-data.sh\"\n    schedule =&gt; \"* * * * *\"\n    tags =&gt; [\"retrieve-data\",\"staging\"]\n  }\n}<\/pre>\n<p>This input is very straightforward to use, you have the options:<\/p>\n<ul class=\"wp-block-list\">\n<li><code>command<\/code>: The shell command that you want to use.<\/li>\n<li><code>schedule<\/code>: Same as the <code>jdbc-input<\/code> schedule is the frequency you want to run the command.<\/li>\n<li><code>tags<\/code>: Information to identify the result later.<\/li>\n<\/ul>\n<p>With this plugin you don&#8217;t have to worry about the result being multi-lined, <code>Logstash<\/code> will group everything in one line.<\/p>\n<h3 class=\"wp-block-heading\">Filter<\/h3>\n<p>Lets say we are receiving data from all the inputs, but we need to transform that data in something useful. We don&#8217;t just want to have log lines but we want statistics to be able to see what is happening in real-time and transform the logs that we receive into properly structured data.<\/p>\n<h4 class=\"wp-block-heading\">Grok Parser<\/h4>\n<p>In most cases the Grok Parser plugin is used. Most of the time the data from the logs is not in a structured format but we still need to parse this unstructured data and give some meaning to it. We can&#8217;t have numbers being strings if we want to sum something. Grok doesn&#8217;t structure the data by using regexes, <code>Logstash<\/code> has a set of built-in regexes that cover most cases.<\/p>\n<p>Here&#8217;s an example of a filter.<\/p>\n<p>We are receiving HTTP requests from the internet and we want to track which endpoints have the most requests, our log is structured in this following way:<\/p>\n<p>192.168.0.1 GET \/index<\/p>\n<p>So we add a Grok filter to structure that:<\/p>\n<pre class=\"wp-block-preformatted brush:java\">filter {\n  grok {\n    match =&gt; { \n      \"message\" =&gt; \"%{IP:client} %{WORD:method} %{URIPATHPARAM:request}\"\n    }\n  }\n}<\/pre>\n<p>What is happening in that filter?<\/p>\n<p><code>match<\/code> is saying to check the message field of the logs that we receive and transform them into structured data.<\/p>\n<p>So we have the request from the webserver:<\/p>\n<p>I want <code>Logstash<\/code> to get the log and transform it into structured data (it&#8217;s a JSON, like everything nowadays). We make the matcher like:<\/p>\n<pre class=\"wp-block-preformatted brush:java\">\"%{IP:client} %{WORD:method} %{URIPATHPARAM:request}\"<\/pre>\n<p>The syntax for grok is <code>%{PATTERN:field}<\/code><\/p>\n<p>The pattern is how you want to match the field and what kind of regex will be used in the data and the field will be the field where that data is going to be stored (bit obvious). You can see all the patterns available <a href=\"https:\/\/github.com\/elastic\/logstash\/blob\/v1.4.2\/patterns\/grok-patterns\">here<\/a>.<\/p>\n<p>In the case that <code>Logstash<\/code> is unable to parse the pattern properly, it will add a tag called <code>_grokparsefailure<\/code>.<\/p>\n<p>Since it&#8217;s just a bunch of regexes parsing data you can create your own patterns. For that, you need to create a folder named <code>patterns<\/code> inside the <code>Logstash<\/code> folder (you will need to check where is installed). The filename can be named anything you want, <code>Logstash<\/code> will read everything inside.<div style=\"display:inline-block; margin: 15px 0;\"> <div id=\"adngin-JavaCodeGeeks_incontent_video-0\" style=\"display:inline-block;\"><\/div> <\/div><\/p>\n<pre class=\"wp-block-preformatted brush:java\">grok {\n  match =&gt; {\n    \"path\" =&gt; \"\/%{FILE_VERSION:version}_%{BU_ID:id}\\.csv$\"\n  }\n  patterns_dir =&gt; [\"\/elasticsearch\/logstash\/example\/patterns\"]\n}<\/pre>\n<p>There&#8217;s also a way of having the files in a different folder and then declare this in the config file.<\/p>\n<p>The contents of the file have to follow the pattern of <code>PATTERN_NAME regex<\/code>, so you would end with something like:<\/p>\n<pre class=\"wp-block-preformatted brush:java\">FILE_VERSION \\w+\\d{6}(?=_)\nBU_ID \\d{3}(?=\\.)<\/pre>\n<p>If you are not planning to reuse the regex and want to inline it, you can do that too:<\/p>\n<pre class=\"wp-block-preformatted brush:java\">%{DATESTAMP:timestamp} (?&lt;mti_before&gt;\\w{46})(?&lt;mti&gt;\\w{4})%{GREEDYDATA}<\/pre>\n<p>Like a regex group you use parenthesis to specify what you want to group, then you start with a question mark to inform that you are going to use a regex. Then you add the name of the field that you are going to parse the data to and finally, you can add the regex.<\/p>\n<p>Grok will also allow conversions to <code>int<\/code> and <code>float<\/code>. You just have to add as an extra parameter in the <code>match<\/code>. E.g.: <code>%{IP:client:int}<\/code><\/p>\n<p>When using the grok plugin take care to not duplicate the message that you are sending by adding the pattern:<\/p>\n<pre class=\"wp-block-preformatted brush:java\">grok {\n  match { message =&gt; \"%{GREEDYDATA:message}\" }\n}<\/pre>\n<p>This would add a new message to the message field instead of replacing.<\/p>\n<h3 class=\"wp-block-heading\">JSON<\/h3>\n<p>You might be in a better situation and have structured your logs in a format like JSON. For that <code>Logstash<\/code> will give you free parsing.<\/p>\n<pre class=\"wp-block-preformatted brush:java\">json {\n  source =&gt; \"message\"\n  target =&gt; \"payload\"\n}<\/pre>\n<p>That&#8217;s everything you have to do for <code>Logstash<\/code> to parse all the JSON messages and set the proper data type for the fields.<\/p>\n<h4 class=\"wp-block-heading\">Key-Value<\/h4>\n<p>Another filter that might be useful is Key-Value or <code>kv<\/code>. It is used to split data based on two keys. So if we have a logline that looks like:<\/p>\n<pre class=\"wp-block-preformatted brush:java\">timestamp=10\/09\/2019 10:10:50, level=INFO, message=Something wrong might not be right<\/pre>\n<p>We can use the kv filter like this:<\/p>\n<pre class=\"wp-block-preformatted brush:java\">filter {\n  kv {\n    source =&gt; \"message\"\n    field_split =&gt; \",\"\n    value_split =&gt; \"=\"\n    target =&gt; \"summary\"\n  }\n}<\/pre>\n<p>So we can parse the data from the message using the <code>kv<\/code> filter. The only problem with this filter is that you can&#8217;t set the data type during the filter step.<\/p>\n<h4 class=\"wp-block-heading\">Mutate<\/h4>\n<p>You might want to change the log that you are receiving, I&#8217;m not talking about full parsing of a logline but small changes. There&#8217;s the <code>mutate<\/code> filter to do that and there are multiple commands that can be used to change your log.<\/p>\n<p>Some examples of what you can do with the mutate filter:<\/p>\n<ul class=\"wp-block-list\">\n<li><code>convert<\/code>: You might have parsed a field but you need that field to be more than just a string. The <code>convert<\/code> command allows you to convert to <code>integer<\/code>, <code>float<\/code>, <code>string<\/code>, or <code>boolean<\/code>.<\/li>\n<\/ul>\n<pre class=\"wp-block-preformatted brush:java\">filter {\n  mutate {\n    convert =&gt; {\n      \"quantity\" =&gt; \"integer\"\n      \"is_paid\" =&gt; \"boolean\"\n    }\n  }\n}<\/pre>\n<ul class=\"wp-block-list\">\n<li><code>remove_field<\/code>: You might want to drop some sensitive data from your logs, so you can use this command to remove it.<\/li>\n<\/ul>\n<pre class=\"wp-block-preformatted brush:java\">filter {\n  mutate {\n    remove_field =&gt; [\"sensitive_data\"]\n  }\n}<\/pre>\n<ul class=\"wp-block-list\">\n<li><code>gsub<\/code>: This is an option to replace values using a regex, you might want to obfuscate some data that isn&#8217;t relevant, you can use this option for that.<\/li>\n<\/ul>\n<pre class=\"wp-block-preformatted brush:java\">filter {\n  mutate {\n    gsub =&gt; [\n      # field                   regex                      result\n      \"transaction_reference\", \"\\d{4}-\\d{4}-\\d{4}-\\d{4}\", \"XXXX-XXXX-XXXX-XXXX\"\n    ]\n  }\n}<\/pre>\n<p>This will replace all transaction references with a masked version.<\/p>\n<h3 class=\"wp-block-heading\">Output<\/h3>\n<p>This is the part where you can direct the log that you just parsed to an output or datastore. In our case we are going to use <code>Elasticsearch<\/code> which is a NoSQL document store, but you can also send to other places like <code>CSV<\/code>, <code>HTTP<\/code> or even <code>email<\/code>.<\/p>\n<p>You can check the documentation for <code>Logstash<\/code> to see all the output plugins.<\/p>\n<pre class=\"wp-block-preformatted brush:java\">output {\n  elasticsearch {\n    hosts =&gt; [\"192.168.0.15\"]\n    user =&gt; \"elastic_user\"\n    password =&gt; \"elastic_password\"\n    index =&gt; \"application-log\"\n  }\n}<\/pre>\n<p>In the example here we are sending our logs to <code>Elasticsearch<\/code> hosted in another machine.<\/p>\n<h2 class=\"wp-block-heading\">Elasticsearch<\/h2>\n<p><code>Elasticsearch<\/code> is a search analytics engine that does the job of collecting and aggregating the stored data. It also provides a real-time search for all kinds of data be that structured or unstructured text, or numerical data.<\/p>\n<p>All the data in <code>Elasticsearch<\/code> is stored in the JSON format and then indexed which enables you to immediately search it. Each stored document is a collection of key-value pairs that contain the data and it&#8217;s stored in an optimised data structure that helps searching it later.<\/p>\n<h3 class=\"wp-block-heading\">Basic constructs<\/h3>\n<h4 class=\"wp-block-heading\">Nodes<\/h4>\n<p>Nodes are one of the most basic constructs that stores and indexes the data. There are multiple types of node.<\/p>\n<ul class=\"wp-block-list\">\n<li>Master node: This controls the cluster<\/li>\n<li>Data node: This holds the data and performs CRUD operations, aggregations and searches.<\/li>\n<li>Ingest node: This transforms and enriches the data before indexing.<\/li>\n<\/ul>\n<h4 class=\"wp-block-heading\">Index<\/h4>\n<p>An index is a collection of documents with similar characteristics, they are like tables in a relational database.<\/p>\n<p>The Indexes are more flexible than a relational database, since they are lightweight you can create multiple indexes without much difficulty. In logging for example, you can create an index for each day and have the type to be the kind of log that you have.<\/p>\n<p>Every day a new index will be created, you wouldn&#8217;t do that for a relational DB.<\/p>\n<h3 class=\"wp-block-heading\">Using Elasticsearch<\/h3>\n<p>There are two main things that we have to pay attention to when working with <code>Elasticsearch<\/code>. They are <code>templates<\/code> and <code>policies<\/code>.<\/p>\n<h4 class=\"wp-block-heading\">Templates<\/h4>\n<p>Templates could be considered the schema of your index, <code>Elasticsearch<\/code> can set a default schema, but you need more control of it if you want do make aggregations and calculations in the data that you have.<\/p>\n<p>Which types does <code>Elasticsearch<\/code> support? The main data types supported are:<\/p>\n<ul class=\"wp-block-list\">\n<li>String<\/li>\n<li>Numeric (long, int, short, double, float)<\/li>\n<li>Date<\/li>\n<\/ul>\n<h4 class=\"wp-block-heading\">Building templates<\/h4>\n<p>How do I set up my template? Well, <code>Elasticsearch<\/code> has a REST-like API that you can easily interact with.<\/p>\n<p>We are adding logs for an application for a Fintech company and we want to monitor the funds transfers that we are making. The payload that we have for transfers is:<\/p>\n<pre class=\"wp-block-preformatted brush:java\">{\n  \"paymentDate\": \"2019-09-14 11:25:32.321\",\n  \"amount\": 100.00,\n  \"message\": \"Payment message\",\n  \"origin\": {\n    \"sortCode\": \"11-11-11\",\n    \"account\": \"838383\"\n  },\n  \"destination\": {\n    \"sortCode\": \"11-11-11\",\n    \"account\": \"1313123\"\n  }\n}<\/pre>\n<p>We start building our template with the <code>payment_date<\/code> field and we can set the type as date and give the format for the field:<\/p>\n<pre class=\"wp-block-preformatted brush:java\">\"payment_date\": {\n  \"type\": \"date\",\n  \"format\": \"yyyy-MM-dd HH:mm:ss.SSS\"\n}<\/pre>\n<p>Then we have the <code>amount<\/code>, we add this field with the type <code>scaled_float<\/code> and the <code>scaling_factor<\/code> to be 100 so <code>Elasticsearch<\/code> can handle two digits in the decimal part making life a little easier for our searches later:<\/p>\n<pre class=\"wp-block-preformatted brush:java\">\"amount\": {\n  \"type\": \"scaled_float\",\n  \"scaling_factor\": \"100\"\n}<\/pre>\n<p>Then we have the field <code>message<\/code> which is just a string, so we are going to use the <code>text<\/code> type which creates a field that is indexed for full text search:<\/p>\n<pre class=\"wp-block-preformatted brush:java\">\"message\": {\n  \"type\": \"text\"\n}<\/pre>\n<p>The <code>origin<\/code> and <code>destination<\/code> fields are virtually the same and they always have the same format, so we can use the <code>keyword<\/code> type. This type is good for small amounts of semi-structured data like postal code, addresses, emails, sort codes and account numbers:<\/p>\n<pre class=\"wp-block-preformatted brush:java\">\"origin\": {\n  \"properties\": {\n    \"body\": {\n      \"properties\": {\n        \"sort_code\": {\n          \"type\": \"keyword\"\n        },\n        \"account\": {\n          \"type\": \"keyword\"\n        }\n      }\n    }\n  }\n}<\/pre>\n<p>Now we have the full mapping for the index we can insert that to <code>Elasticsearch<\/code>. We just make a <code>PUT<\/code> request to it.<\/p>\n<pre class=\"wp-block-preformatted brush:java\">curl -X PUT \"http:\/\/elasticsearch.com\/_template\/transfers_template\" -d @transfers_template.json<\/pre>\n<pre class=\"wp-block-preformatted brush:java\">{\n  \"index_patterns\": [\n    \"transfers-*\"\n  ],\n  \"mappings\": {\n    \"_meta\": {\n      \"beat\": \"transfers\",\n      \"version\": \"7.0.1\"\n    },\n    \"date_detection\": false,\n    \"dynamic_templates\": [\n      {\n        \"labels\": {\n          \"mapping\": {\n            \"type\": \"keyword\"\n          },\n          \"match_mapping_type\": \"string\",\n          \"path_match\": \"labels.*\"\n        }\n      },\n      {\n        \"container.labels\": {\n          \"mapping\": {\n            \"type\": \"keyword\"\n          },\n          \"match_mapping_type\": \"string\",\n          \"path_match\": \"container.labels.*\"\n        }\n      },\n      {\n        \"fields\": {\n          \"mapping\": {\n            \"type\": \"keyword\"\n          },\n          \"match_mapping_type\": \"string\",\n          \"path_match\": \"fields.*\"\n        }\n      },\n      {\n        \"docker.container.labels\": {\n          \"mapping\": {\n            \"type\": \"keyword\"\n          },\n          \"match_mapping_type\": \"string\",\n          \"path_match\": \"docker.container.labels.*\"\n        }\n      },\n      {\n        \"kibana.log.meta\": {\n          \"mapping\": {\n            \"type\": \"keyword\"\n          },\n          \"match_mapping_type\": \"string\",\n          \"path_match\": \"kibana.log.meta.*\"\n        }\n      },\n      {\n        \"strings_as_keyword\": {\n          \"mapping\": {\n            \"ignore_above\": 1024,\n            \"type\": \"keyword\"\n          },\n          \"match_mapping_type\": \"string\"\n        }\n      }\n    ],\n    \"properties\": {\n      \"@timestamp\": {\n        \"type\": \"date\"\n      },\n      \"payment_date\": {\n        \"type\": \"date\",\n        \"format\": \"yyyy-MM-ddTHH:mm:ss.SSSSSS\"\n      },\n      \"amount\": {\n        \"type\": \"scaled_float\",\n        \"scaling_factor\": \"100\"\n      },\n      \"message\": {\n        \"type\": \"text\"\n      },\n      \"origin\": {\n        \"properties\": {\n          \"body\": {\n            \"properties\": {\n              \"sort_code\": {\n                \"type\": \"keyword\"\n              },\n              \"account\": {\n                \"type\": \"keyword\"\n              }\n            }\n          }\n        }\n      },\n      \"destination\": {\n        \"properties\": {\n          \"body\": {\n            \"properties\": {\n              \"sort_code\": {\n                \"type\": \"keyword\"\n              },\n              \"account\": {\n                \"type\": \"keyword\"\n              }\n            }\n          }\n        }\n      }\n    }\n  }\n}<\/pre>\n<h4 class=\"wp-block-heading\">Policies<\/h4>\n<p>This feature is only available in the premium versions of <code>Elasticsearch<\/code>.<\/p>\n<p>The indexes are going to be bombarded with data the entire time and just like log files, we need a rollover policy to not get our disks full. In the premium version of <code>Elasticsearch<\/code>, we have the Index Policies tools to help us manage that.<\/p>\n<p>The first thing to know is what are the states that an Index can be.<\/p>\n<ul class=\"wp-block-list\">\n<li><code>hot<\/code>: Is the index that we are writing in.<\/li>\n<li><code>warm<\/code>: Is an index that we are querying frequently, but not writing in.<\/li>\n<li><code>cold<\/code>: Is an index that we don&#8217;t write to anymore and we also don&#8217;t query the data very often.<\/li>\n<li><code>delete<\/code>: Is an index that is no longer needed and can be deleted.<\/li>\n<\/ul>\n<p>An index starts at the <code>hot<\/code> state and we can say to <code>Elasticsearch<\/code> when we don&#8217;t want to keep writing in an index anymore. We tell it to start to use another index using the <code>max_age<\/code> and the <code>max_size<\/code> options. In the example below, we are making a new index every day or when it reaches <code>5GB<\/code> (the number was arbitrarily chosen).<\/p>\n<pre class=\"wp-block-preformatted brush:java\">{\n  \"policy\": {\n    \"phases\": {\n      \"hot\": {\n        \"actions\": {\n          \"rollover\": {\n            \"max_age\": \"1d\",\n            \"max_size\": \"5GB\"\n          }\n        }\n      }\n    }\n  }\n}<\/pre>\n<p>We don&#8217;t want to keep all the indexes <code>hot<\/code>, so we can start to change the state of our older indexes and make them <code>warm<\/code>. The policy to define what is going to the <code>warm<\/code> state starts with the <code>min_age<\/code> parameter, which sounds very obvious what it does. In our case we are setting the <code>min_age<\/code> to <code>7d<\/code>, so all the <code>hot<\/code> indexes that are seven days or older are going to be converted to a <code>warm<\/code> index.<\/p>\n<p>For <code>warm<\/code> indexes we have some options that weren&#8217;t previously available in the <code>hot<\/code> one, the <code>actions<\/code> section allows us to do some changes when changing the state of the index.<\/p>\n<p>The first one that we can see is <code>forcemerge<\/code> this option, when set to <code>1<\/code>, tells <code>Elasticsearch<\/code> to merge all the indexes that are going from <code>hot<\/code> to <code>warm<\/code>. This is helpful because in <code>Elastisearch<\/code> when you delete a document, that document isn&#8217;t really deleted, but only marked has deleted. During the merge the documents marked as deleted are going to be properly deleted, like you would send the files to the <code>Trash bin<\/code> and then delete them from your system later.<\/p>\n<p>Then we have <code>shrink<\/code> which is used to reduce the number of shards of an index. Since we are not writing in that index anymore we don&#8217;t need all shards that we allocated previously.<\/p>\n<p>And finally we have <code>allocate<\/code>. Here we can set the <code>number_of_replicas<\/code>, in case we need the data to be highly available, also it&#8217;s more secure than having only one shard.<\/p>\n<pre class=\"wp-block-preformatted brush:java\">\"warm\": {\n  \"min_age\": \"7d\",\n  \"actions\": {\n    \"forcemerge\": {\n      \"max_num_segments\": 1\n    },\n    \"shrink\": {\n      \"number_of_shards\": 1\n    },\n    \"allocate\": {\n      \"number_of_replicas\": 2,\n      \"require\": {\n        \"data\": \"warm\"\n      }\n    }\n  }\n}<\/pre>\n<p>For data that is even older than what we set for the <code>warm<\/code> state and that we are just storing for security reasons, we can turn their indexes to <code>cold<\/code>. We set to every index older than 6 months to be set to <code>cold<\/code> and we freeze them. The advantage of having the freeze in this step is that <code>Elastisearch<\/code> don&#8217;t use extra memory for frozen indexes.<\/p>\n<pre class=\"wp-block-preformatted brush:java\">\"cold\": {\n  \"min_age\": \"180d\",\n  \"actions\": {\n    \"freeze\": {}\n  }\n}<\/pre>\n<p>Finally, there is the deletion of the index that is quite straight forward.<\/p>\n<pre class=\"wp-block-preformatted brush:java\">\"delete\": {\n  \"min_age\": \"365d\",\n  \"actions\": {\n    \"delete\": {}\n  }\n}<\/pre>\n<p>The final policy that we have would look like this:<\/p>\n<pre class=\"wp-block-preformatted brush:java\">{\n  \"policy\": {\n    \"phases\": {\n      \"hot\": {\n        \"actions\": {\n          \"rollover\": {\n            \"max_age\": \"1d\",\n            \"max_size\": \"5GB\"\n          }\n        }\n      },\n      \"warm\": {\n        \"min_age\": \"7d\",\n        \"actions\": {\n          \"forcemerge\": {\n            \"max_num_segments\": 1\n          },\n          \"shrink\": {\n            \"number_of_shards\": 1\n          },\n          \"allocate\": {\n            \"number_of_replicas\": 2,\n            \"require\": {\n              \"data\": \"warm\"\n            }\n          }\n        }\n      },\n      \"cold\": {\n        \"min_age\": \"100d\",\n        \"actions\": {\n          \"freeze\": {}\n        }\n      },\n      \"delete\": {\n        \"min_age\": \"365d\",\n        \"actions\": {\n          \"delete\": {}\n        }\n      }\n    }\n  }\n}<\/pre>\n<h2 class=\"wp-block-heading\">Beats<\/h2>\n<h3 class=\"wp-block-heading\">FileBeat<\/h3>\n<p><code>Filebeat<\/code> is a lightweight tool that reads the logs and sends them to <code>ElasticSearch<\/code> or <code>Logstash<\/code>. The only purpose of this tool is to read the log files, it can&#8217;t do any complex operation with it. If you want to perform complex operations then you can send the log to <code>Logstash<\/code> and parse into the desired information.<\/p>\n<p>When you have multiple servers and you don&#8217;t want to install <code>Logstash<\/code> in all the machines because it&#8217;s a heavy application, you can use <code>Filebeat<\/code> as it&#8217;s written in Go, is natively compiled and it&#8217;s very lightweight.<\/p>\n<p>It&#8217;s not too hard to configure <code>Filebeat<\/code> since it doesn&#8217;t do much. You have the <code>filebeat.yml<\/code> that contains the configuration:<\/p>\n<pre class=\"wp-block-preformatted brush:java\">filebeat.config.inputs:\n filebeat:\n inputs:\n  - type: log\n paths:\n  - \"\/var\/log\/applog\/*.log\"\n\n output:\n logstash:\n hosts: [\"logstash:5043\"]<\/pre>\n<p>We can see the <code>inputs<\/code> part that reads the logs from a folder and sends to an <code>output<\/code> in <code>Logstash<\/code> that is hosted elsewhere. In this configuration file, we are reading logs from the folder <code>applog<\/code>. You can also use the glob patterns in the paths to get multiple files or to capture differences like upper or lower case letters.<\/p>\n<h4 class=\"wp-block-heading\">Multiline string<\/h4>\n<p>When working with <code>Filebeat<\/code> you might need to read multiline strings. Just like <code>Logstash<\/code> you can use the <code>multiline<\/code> options in the configuration to read the file, all the same fields for <code>Logstash<\/code> are applied for <code>Filebeat<\/code>.<\/p>\n<pre class=\"wp-block-preformatted brush:java\">filebeat.config.inputs:\n  - type: log\n    paths:\n      - \"\/var\/log\/applog\/application.log\"\n    multiline:\n      pattern: \"\\\\[\\\\d{2}\\\\\/\\\\d{2}\\\\\/\\\\d{4} \\\\d{2}:\\\\d{2}:\\\\d{2}]\\\\[\\\\w{2}\\\\]\"\n      negate: True\n      match: after<\/pre>\n<p><a href=\"https:\/\/www.elastic.co\/guide\/en\/beats\/filebeat\/current\/how-filebeat-works.html\">How Filebeat works | Filebeat Reference [7.1] | Elastic<\/a><\/p>\n<h4 class=\"wp-block-heading\">Fields<\/h4>\n<p>You can also add extra information to the logs that you are reading so you can identify the files that the data has been obtained from for use at a later date.<\/p>\n<pre class=\"wp-block-preformatted brush:java\">filebeat.config.inputs:\n  - type: log\n    paths:\n      - \"\/var\/log\/applog\/application.log\"\n    fields:\n      file_name: \"application_log\"\n    multiline:\n      pattern: \"\\\\[\\\\d{2}\\\\\/\\\\d{2}\\\\\/\\\\d{4} \\\\d{2}:\\\\d{2}:\\\\d{2}]\\\\[\\\\w{2}\\\\]\"\n      negate: True\n      match: after<\/pre>\n<h4 class=\"wp-block-heading\">Autoreload<\/h4>\n<p>You can also split the input list from the configuration file and by doing that you can update the file with the inputs without restarting <code>Filebeat<\/code>. So instead of adding the <code>inputs<\/code> straight away in the main config file we provide the to the file with the inputs configuration.<\/p>\n<pre class=\"wp-block-preformatted brush:java\">filebeat.config.inputs:\n  enabled: true\n  path: inputs.yml\n  reload.enabled: true\n  reload.period: 10s<\/pre>\n<p>The <code>inputs.yml<\/code> that <code>Filebeat<\/code> is going to load:<\/p>\n<pre class=\"wp-block-preformatted brush:java\">- type: log\n  paths:\n    - \"\/var\/log\/haproxy.log\"\n  fields:\n    file_name: \"Haproxy\"<\/pre>\n<h2 class=\"wp-block-heading\">Kibana<\/h2>\n<p>Up to this point we are collecting, structuring and storing all those logs. Now we need to get some value from them. Just having them stored isn&#8217;t the best option, we need to visualise them to have some feedback.<\/p>\n<p>For visualising all the data that was stored to <code>Elasticsearch<\/code> you can use <code>Kibana<\/code>. It&#8217;s an application that allows you to query the data from <code>Elasticsearch<\/code> and create visualisations based on that.<\/p>\n<p>In the <code>Kibana<\/code> home we opted to connect to an Index in <code>Elastisearch<\/code> using a pattern to specify the name of the index like <code>logs-*<\/code> so we can search all indexes starting with <code>logs-<\/code> since we might have grouped our logs by day and not with everything in one index.<\/p>\n<h3 class=\"wp-block-heading\">Discovery<\/h3>\n<p>The <code>Discovery<\/code> area allows you to visualise and search data stored in <code>Elastichsearch<\/code>.<\/p>\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter is-resized\"><img decoding=\"async\" src=\"https:\/\/www.javacodegeeks.com\/wp-content\/uploads\/2019\/10\/Discovery-1-1024x585.png\" alt=\"\" class=\"wp-image-99667\" width=\"768\" height=\"439\" srcset=\"https:\/\/www.javacodegeeks.com\/wp-content\/uploads\/2019\/10\/Discovery-1-1024x585.png 1024w, https:\/\/www.javacodegeeks.com\/wp-content\/uploads\/2019\/10\/Discovery-1-300x171.png 300w, https:\/\/www.javacodegeeks.com\/wp-content\/uploads\/2019\/10\/Discovery-1-768x439.png 768w\" sizes=\"(max-width: 768px) 100vw, 768px\" \/><\/figure>\n<\/div>\n<h4 class=\"wp-block-heading\">1 &#8211; Filters<\/h4>\n<p>You have a bar where you can write your queries using <a href=\"https:\/\/www.elastic.co\/guide\/en\/kibana\/7.1\/kuery-query.html\">KQL<\/a> which is a custom query language that is quite easy to use. <code>Kibana<\/code> will help you to auto-complete the query too.<\/p>\n<p>So if we want to search you can just type:<\/p>\n<pre class=\"wp-block-preformatted brush:java\">tags: \"retrieve-data\" and stats1 &gt;= 10000<\/pre>\n<p>Or you can use the <code>filters<\/code> section where you have a more UI oriented way to make your searches.<\/p>\n<p>You can also save your searches for later so you don&#8217;t have to rewrite every time. The saved searches can be used in other parts of the Kibana UI.<\/p>\n<h4 class=\"wp-block-heading\">2 &#8211; Date filters<\/h4>\n<p>In case you want to filter the results by a specific period, the filter allows you to use two options:<\/p>\n<ul class=\"wp-block-list\">\n<li>Absolute: You can set the exact date and time that you want.<\/li>\n<li>Relative: You set a date and a timestamp that you want, like <code>10 minutes ago<\/code>. With this option, you can also set to get the latest data from <code>Elasticsearch<\/code>.<\/li>\n<\/ul>\n<h4 class=\"wp-block-heading\">3 &#8211; Records<\/h4>\n<p>You can expand and see records in a key-value way and it also shows the type of the field, which can be three:<\/p>\n<ul class=\"wp-block-list\">\n<li><code>t<\/code> &#8211; Text field<\/li>\n<li><code>#<\/code> &#8211; Numeric field<\/li>\n<li>Clock Symbol &#8211; Date time<\/li>\n<\/ul>\n<p>You can also filter which fields you want to see by selecting them in the <code>Available Fields<\/code> menu on the left<\/p>\n<h3 class=\"wp-block-heading\">Visualise<\/h3>\n<p>We need a way to visualise all that data that is stored nicely, and Visualise allows us to create multiple kinds of graphs.<\/p>\n<h4 class=\"wp-block-heading\">Y-Axis<\/h4>\n<p>In this example we are creating a <code>bar graph<\/code>. In the left menu you can set the options to create a graph.<\/p>\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter is-resized\"><img decoding=\"async\" src=\"https:\/\/www.javacodegeeks.com\/wp-content\/uploads\/2019\/10\/Visualisation-1-1024x583.png\" alt=\"\" class=\"wp-image-99668\" width=\"768\" height=\"437\" srcset=\"https:\/\/www.javacodegeeks.com\/wp-content\/uploads\/2019\/10\/Visualisation-1-1024x583.png 1024w, https:\/\/www.javacodegeeks.com\/wp-content\/uploads\/2019\/10\/Visualisation-1-300x171.png 300w, https:\/\/www.javacodegeeks.com\/wp-content\/uploads\/2019\/10\/Visualisation-1-768x437.png 768w\" sizes=\"(max-width: 768px) 100vw, 768px\" \/><\/figure>\n<\/div>\n<ol class=\"wp-block-list\">\n<li><code>Aggregation<\/code>: Is the kind of operation that you want to perform, which can be operations <code>count<\/code>, <code>sum<\/code> and <code>average<\/code>. There are also more complex operations like <code>Standard Deviation<\/code> and operations using other values.<\/li>\n<li><code>Field<\/code>: Is the field that you want to use to make the calculation and the value to be displayed. For simple aggregations you can select the field straightway from the dropdown and for more complex queries you can use <code>QueryDSL<\/code> and <code>JSON<\/code> to find the values.<\/li>\n<li><code>Custom Label<\/code>: You probably don&#8217;t want to display your data without meaning, so here you can add a nice label for it.<\/li>\n<\/ol>\n<h4 class=\"wp-block-heading\">Buckets<\/h4>\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter is-resized\"><img decoding=\"async\" src=\"https:\/\/www.javacodegeeks.com\/wp-content\/uploads\/2019\/10\/Visualisation-2-1024x508.png\" alt=\"\" class=\"wp-image-99669\" width=\"768\" height=\"381\" srcset=\"https:\/\/www.javacodegeeks.com\/wp-content\/uploads\/2019\/10\/Visualisation-2-1024x508.png 1024w, https:\/\/www.javacodegeeks.com\/wp-content\/uploads\/2019\/10\/Visualisation-2-300x149.png 300w, https:\/\/www.javacodegeeks.com\/wp-content\/uploads\/2019\/10\/Visualisation-2-768x381.png 768w, https:\/\/www.javacodegeeks.com\/wp-content\/uploads\/2019\/10\/Visualisation-2.png 1914w\" sizes=\"(max-width: 768px) 100vw, 768px\" \/><\/figure>\n<\/div>\n<ol class=\"wp-block-list\">\n<li><code>Aggregation<\/code>: Is how your are going to group your data. You might want a time using the <code>Date Histogram<\/code> or something to group the endpoints that you have using a specific field.<\/li>\n<li><code>Field<\/code>: Is the field that you are going to use to split the data. In case you need a more complex aggregation, you will be able to use the <code>QueryDSL<\/code> as an input.<\/li>\n<li><code>Interval<\/code>: In case you want to group by the date you can specify the time period that you want to split the data. For other aggregations like ranges you get different interval fields.<\/li>\n<\/ol>\n<h3 class=\"wp-block-heading\">Dashboards<\/h3>\n<p>Now we can put together the visualisation that we built into <code>Dashboards<\/code>, so we can aggregate all the data that we want to see into one place. You can add the visualisations that you created and resize them to fit the screen the way you want. Also you can change the search for them and even have them in full screen to be displayed really nicely.<\/p>\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter is-resized\"><img decoding=\"async\" src=\"https:\/\/www.javacodegeeks.com\/wp-content\/uploads\/2019\/10\/Dashboard-1-1024x640.png\" alt=\"\" class=\"wp-image-99670\" width=\"768\" height=\"480\" srcset=\"https:\/\/www.javacodegeeks.com\/wp-content\/uploads\/2019\/10\/Dashboard-1-1024x640.png 1024w, https:\/\/www.javacodegeeks.com\/wp-content\/uploads\/2019\/10\/Dashboard-1-300x188.png 300w, https:\/\/www.javacodegeeks.com\/wp-content\/uploads\/2019\/10\/Dashboard-1-768x480.png 768w\" sizes=\"(max-width: 768px) 100vw, 768px\" \/><\/figure>\n<\/div>\n<h3 class=\"wp-block-heading\">Dev Tools<\/h3>\n<p>Kibana also provides you with the <code>Dev Tools<\/code> sections that have some tools to help your work.<\/p>\n<h4 class=\"wp-block-heading\">Console<\/h4>\n<p>We saw that <code>Elasticsearch<\/code> has a REST-like API, well the console provides an easy way to interact with the API. Providing auto-complete and connecting straight away to <code>Elasticsearch<\/code> so you don&#8217;t have to build the entire request by hand. You can make a request just by having the method and the path, you can also write <code>JSON<\/code> with verification and completion.<\/p>\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter is-resized\"><img decoding=\"async\" src=\"https:\/\/www.javacodegeeks.com\/wp-content\/uploads\/2019\/10\/Console-1-1024x512.png\" alt=\"\" class=\"wp-image-99671\" width=\"768\" height=\"384\" srcset=\"https:\/\/www.javacodegeeks.com\/wp-content\/uploads\/2019\/10\/Console-1-1024x512.png 1024w, https:\/\/www.javacodegeeks.com\/wp-content\/uploads\/2019\/10\/Console-1-300x150.png 300w, https:\/\/www.javacodegeeks.com\/wp-content\/uploads\/2019\/10\/Console-1-768x384.png 768w, https:\/\/www.javacodegeeks.com\/wp-content\/uploads\/2019\/10\/Console-1.png 1912w\" sizes=\"(max-width: 768px) 100vw, 768px\" \/><\/figure>\n<\/div>\n<pre class=\"wp-block-preformatted brush:java\">GET _template\/transfers_template<\/pre>\n<pre class=\"wp-block-preformatted brush:java\">PUT \/test-template-000001\n{\n  \"aliases\": {\n    \"test-template-alias\": {\n      \"is_write_index\": true\n    }\n  }\n}<\/pre>\n<h4 class=\"wp-block-heading\">Grok Debugger<\/h4>\n<p>When we were talking about <code>Logstash<\/code> we saw the <code>Grok<\/code> plugin and how powerful it can be, but <code>Grok<\/code> works using regexes, and everyone knows that regexes are complete madness and very easy to break, so the debugger helps us to build the patterns to parse our log lines.<\/p>\n<p>We have the message getting the stats from our server, there are four columns, but we have to split into proper fields so we use the <code>Grok<\/code> parser for that.<\/p>\n<pre class=\"wp-block-preformatted brush:java\">16765 | 10475 | 26017 | 27583<\/pre>\n<pre class=\"wp-block-preformatted brush:java\">%{NUMBER:stats1} \\| %{NUMBER:stats2} \\| %{NUMBER:stats3} \\| %{NUMBER:stats4}<\/pre>\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter is-resized\"><img decoding=\"async\" src=\"https:\/\/www.javacodegeeks.com\/wp-content\/uploads\/2019\/10\/Console-2-1024x516.png\" alt=\"\" class=\"wp-image-99672\" width=\"768\" height=\"387\" srcset=\"https:\/\/www.javacodegeeks.com\/wp-content\/uploads\/2019\/10\/Console-2-1024x516.png 1024w, https:\/\/www.javacodegeeks.com\/wp-content\/uploads\/2019\/10\/Console-2-300x151.png 300w, https:\/\/www.javacodegeeks.com\/wp-content\/uploads\/2019\/10\/Console-2-768x387.png 768w, https:\/\/www.javacodegeeks.com\/wp-content\/uploads\/2019\/10\/Console-2.png 1892w\" sizes=\"(max-width: 768px) 100vw, 768px\" \/><\/figure>\n<\/div>\n<h2 class=\"wp-block-heading\">That&#8217;s all for today<\/h2>\n<p>In this post we saw the components of the Elastic Stack, how to start using them and the importance of having your logs organised. Also you can always resort to the Elastic Stack documentation <a href=\"https:\/\/www.elastic.co\/guide\/index.html\">here<\/a> to see what other functionality is supported that isn&#8217;t mentioned in my post.<\/p>\n<div class=\"attribution\">\n<table>\n<tbody>\n<tr>\n<td>Published on Java Code Geeks with permission by Andr\u00e9 Guelfi Torres, partner at our <a href=\"\/\/www.javacodegeeks.com\/join-us\/jcg\/\" target=\"_blank\" rel=\"noopener noreferrer\">JCG program<\/a>. See the original article here: <a href=\"https:\/\/codurance.com\/2019\/10\/24\/elastic-stack-introduction\/\" target=\"_blank\" rel=\"noopener noreferrer\">Elastic Stack Introduction<\/a><\/p>\n<p>Opinions expressed by Java Code Geeks contributors are their own.<\/p>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n","protected":false},"excerpt":{"rendered":"<p>When you are running software that is critical for your company you can&#8217;t have logs just for analysing things that happened some time ago, having the client tell you that your application is broken while you didn&#8217;t even know what was happening is a real problem. One of the ways to deal with that is &hellip;<\/p>\n","protected":false},"author":76873,"featured_media":112,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[8],"tags":[],"class_list":["post-99664","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-enterprise-java"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.5 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Elastic Stack Introduction - Java Code Geeks<\/title>\n<meta name=\"description\" content=\"Interested to learn about Elastic Stack? Check our article provide the basics way to start collecting and analysing your logs with elasticstack\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.javacodegeeks.com\/2019\/11\/elastic-stack-introduction.html\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Elastic Stack Introduction - Java Code Geeks\" \/>\n<meta property=\"og:description\" content=\"Interested to learn about Elastic Stack? Check our article provide the basics way to start collecting and analysing your logs with elasticstack\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.javacodegeeks.com\/2019\/11\/elastic-stack-introduction.html\" \/>\n<meta property=\"og:site_name\" content=\"Java Code Geeks\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/javacodegeeks\" \/>\n<meta property=\"article:published_time\" content=\"2019-11-01T05:00:51+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.javacodegeeks.com\/wp-content\/uploads\/2012\/10\/enterprise-java-logo.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"150\" \/>\n\t<meta property=\"og:image:height\" content=\"150\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Andre Torres\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@javacodegeeks\" \/>\n<meta name=\"twitter:site\" content=\"@javacodegeeks\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Andre Torres\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"25 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/www.javacodegeeks.com\\\/2019\\\/11\\\/elastic-stack-introduction.html#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.javacodegeeks.com\\\/2019\\\/11\\\/elastic-stack-introduction.html\"},\"author\":{\"name\":\"Andre Torres\",\"@id\":\"https:\\\/\\\/www.javacodegeeks.com\\\/#\\\/schema\\\/person\\\/d15eabf874a51b64dc425a55a88703b5\"},\"headline\":\"Elastic Stack Introduction\",\"datePublished\":\"2019-11-01T05:00:51+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/www.javacodegeeks.com\\\/2019\\\/11\\\/elastic-stack-introduction.html\"},\"wordCount\":4119,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\\\/\\\/www.javacodegeeks.com\\\/#organization\"},\"image\":{\"@id\":\"https:\\\/\\\/www.javacodegeeks.com\\\/2019\\\/11\\\/elastic-stack-introduction.html#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.javacodegeeks.com\\\/wp-content\\\/uploads\\\/2012\\\/10\\\/enterprise-java-logo.jpg\",\"articleSection\":[\"Enterprise Java\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/www.javacodegeeks.com\\\/2019\\\/11\\\/elastic-stack-introduction.html#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/www.javacodegeeks.com\\\/2019\\\/11\\\/elastic-stack-introduction.html\",\"url\":\"https:\\\/\\\/www.javacodegeeks.com\\\/2019\\\/11\\\/elastic-stack-introduction.html\",\"name\":\"Elastic Stack Introduction - Java Code Geeks\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.javacodegeeks.com\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/www.javacodegeeks.com\\\/2019\\\/11\\\/elastic-stack-introduction.html#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/www.javacodegeeks.com\\\/2019\\\/11\\\/elastic-stack-introduction.html#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.javacodegeeks.com\\\/wp-content\\\/uploads\\\/2012\\\/10\\\/enterprise-java-logo.jpg\",\"datePublished\":\"2019-11-01T05:00:51+00:00\",\"description\":\"Interested to learn about Elastic Stack? Check our article provide the basics way to start collecting and analysing your logs with elasticstack\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/www.javacodegeeks.com\\\/2019\\\/11\\\/elastic-stack-introduction.html#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/www.javacodegeeks.com\\\/2019\\\/11\\\/elastic-stack-introduction.html\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.javacodegeeks.com\\\/2019\\\/11\\\/elastic-stack-introduction.html#primaryimage\",\"url\":\"https:\\\/\\\/www.javacodegeeks.com\\\/wp-content\\\/uploads\\\/2012\\\/10\\\/enterprise-java-logo.jpg\",\"contentUrl\":\"https:\\\/\\\/www.javacodegeeks.com\\\/wp-content\\\/uploads\\\/2012\\\/10\\\/enterprise-java-logo.jpg\",\"width\":150,\"height\":150,\"caption\":\"java-interview-questions-answers\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/www.javacodegeeks.com\\\/2019\\\/11\\\/elastic-stack-introduction.html#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/www.javacodegeeks.com\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Java\",\"item\":\"https:\\\/\\\/www.javacodegeeks.com\\\/category\\\/java\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"Enterprise Java\",\"item\":\"https:\\\/\\\/www.javacodegeeks.com\\\/category\\\/java\\\/enterprise-java\"},{\"@type\":\"ListItem\",\"position\":4,\"name\":\"Elastic Stack Introduction\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/www.javacodegeeks.com\\\/#website\",\"url\":\"https:\\\/\\\/www.javacodegeeks.com\\\/\",\"name\":\"Java Code Geeks\",\"description\":\"Java Developers Resource Center\",\"publisher\":{\"@id\":\"https:\\\/\\\/www.javacodegeeks.com\\\/#organization\"},\"alternateName\":\"JCG\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/www.javacodegeeks.com\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/www.javacodegeeks.com\\\/#organization\",\"name\":\"Exelixis Media P.C.\",\"url\":\"https:\\\/\\\/www.javacodegeeks.com\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.javacodegeeks.com\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/www.javacodegeeks.com\\\/wp-content\\\/uploads\\\/2022\\\/06\\\/exelixis-logo.png\",\"contentUrl\":\"https:\\\/\\\/www.javacodegeeks.com\\\/wp-content\\\/uploads\\\/2022\\\/06\\\/exelixis-logo.png\",\"width\":864,\"height\":246,\"caption\":\"Exelixis Media P.C.\"},\"image\":{\"@id\":\"https:\\\/\\\/www.javacodegeeks.com\\\/#\\\/schema\\\/logo\\\/image\\\/\"},\"sameAs\":[\"https:\\\/\\\/www.facebook.com\\\/javacodegeeks\",\"https:\\\/\\\/x.com\\\/javacodegeeks\"]},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/www.javacodegeeks.com\\\/#\\\/schema\\\/person\\\/d15eabf874a51b64dc425a55a88703b5\",\"name\":\"Andre Torres\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/d31284760a81875079ef459ff625db1c9149c99687d412a4e5ed57aa91fb4d7c?s=96&d=mm&r=g\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/d31284760a81875079ef459ff625db1c9149c99687d412a4e5ed57aa91fb4d7c?s=96&d=mm&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/d31284760a81875079ef459ff625db1c9149c99687d412a4e5ed57aa91fb4d7c?s=96&d=mm&r=g\",\"caption\":\"Andre Torres\"},\"description\":\"Andre is a software developer from Brazil who discovered his passion for programming while working as tech support for an ERP in the fashion industry. Later, he became a developer in the same company working with Delphi. In 2017 he decided to move to Europe and landed a job in the gaming industry in London, where he stayed working in the payments team for almost a year, until he joined Codurance.\",\"sameAs\":[\"https:\\\/\\\/codurance.com\"],\"url\":\"https:\\\/\\\/www.javacodegeeks.com\\\/author\\\/andre-torres\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Elastic Stack Introduction - Java Code Geeks","description":"Interested to learn about Elastic Stack? Check our article provide the basics way to start collecting and analysing your logs with elasticstack","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.javacodegeeks.com\/2019\/11\/elastic-stack-introduction.html","og_locale":"en_US","og_type":"article","og_title":"Elastic Stack Introduction - Java Code Geeks","og_description":"Interested to learn about Elastic Stack? Check our article provide the basics way to start collecting and analysing your logs with elasticstack","og_url":"https:\/\/www.javacodegeeks.com\/2019\/11\/elastic-stack-introduction.html","og_site_name":"Java Code Geeks","article_publisher":"https:\/\/www.facebook.com\/javacodegeeks","article_published_time":"2019-11-01T05:00:51+00:00","og_image":[{"width":150,"height":150,"url":"https:\/\/www.javacodegeeks.com\/wp-content\/uploads\/2012\/10\/enterprise-java-logo.jpg","type":"image\/jpeg"}],"author":"Andre Torres","twitter_card":"summary_large_image","twitter_creator":"@javacodegeeks","twitter_site":"@javacodegeeks","twitter_misc":{"Written by":"Andre Torres","Est. reading time":"25 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.javacodegeeks.com\/2019\/11\/elastic-stack-introduction.html#article","isPartOf":{"@id":"https:\/\/www.javacodegeeks.com\/2019\/11\/elastic-stack-introduction.html"},"author":{"name":"Andre Torres","@id":"https:\/\/www.javacodegeeks.com\/#\/schema\/person\/d15eabf874a51b64dc425a55a88703b5"},"headline":"Elastic Stack Introduction","datePublished":"2019-11-01T05:00:51+00:00","mainEntityOfPage":{"@id":"https:\/\/www.javacodegeeks.com\/2019\/11\/elastic-stack-introduction.html"},"wordCount":4119,"commentCount":0,"publisher":{"@id":"https:\/\/www.javacodegeeks.com\/#organization"},"image":{"@id":"https:\/\/www.javacodegeeks.com\/2019\/11\/elastic-stack-introduction.html#primaryimage"},"thumbnailUrl":"https:\/\/www.javacodegeeks.com\/wp-content\/uploads\/2012\/10\/enterprise-java-logo.jpg","articleSection":["Enterprise Java"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.javacodegeeks.com\/2019\/11\/elastic-stack-introduction.html#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.javacodegeeks.com\/2019\/11\/elastic-stack-introduction.html","url":"https:\/\/www.javacodegeeks.com\/2019\/11\/elastic-stack-introduction.html","name":"Elastic Stack Introduction - Java Code Geeks","isPartOf":{"@id":"https:\/\/www.javacodegeeks.com\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.javacodegeeks.com\/2019\/11\/elastic-stack-introduction.html#primaryimage"},"image":{"@id":"https:\/\/www.javacodegeeks.com\/2019\/11\/elastic-stack-introduction.html#primaryimage"},"thumbnailUrl":"https:\/\/www.javacodegeeks.com\/wp-content\/uploads\/2012\/10\/enterprise-java-logo.jpg","datePublished":"2019-11-01T05:00:51+00:00","description":"Interested to learn about Elastic Stack? Check our article provide the basics way to start collecting and analysing your logs with elasticstack","breadcrumb":{"@id":"https:\/\/www.javacodegeeks.com\/2019\/11\/elastic-stack-introduction.html#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.javacodegeeks.com\/2019\/11\/elastic-stack-introduction.html"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.javacodegeeks.com\/2019\/11\/elastic-stack-introduction.html#primaryimage","url":"https:\/\/www.javacodegeeks.com\/wp-content\/uploads\/2012\/10\/enterprise-java-logo.jpg","contentUrl":"https:\/\/www.javacodegeeks.com\/wp-content\/uploads\/2012\/10\/enterprise-java-logo.jpg","width":150,"height":150,"caption":"java-interview-questions-answers"},{"@type":"BreadcrumbList","@id":"https:\/\/www.javacodegeeks.com\/2019\/11\/elastic-stack-introduction.html#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.javacodegeeks.com\/"},{"@type":"ListItem","position":2,"name":"Java","item":"https:\/\/www.javacodegeeks.com\/category\/java"},{"@type":"ListItem","position":3,"name":"Enterprise Java","item":"https:\/\/www.javacodegeeks.com\/category\/java\/enterprise-java"},{"@type":"ListItem","position":4,"name":"Elastic Stack Introduction"}]},{"@type":"WebSite","@id":"https:\/\/www.javacodegeeks.com\/#website","url":"https:\/\/www.javacodegeeks.com\/","name":"Java Code Geeks","description":"Java Developers Resource Center","publisher":{"@id":"https:\/\/www.javacodegeeks.com\/#organization"},"alternateName":"JCG","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.javacodegeeks.com\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.javacodegeeks.com\/#organization","name":"Exelixis Media P.C.","url":"https:\/\/www.javacodegeeks.com\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.javacodegeeks.com\/#\/schema\/logo\/image\/","url":"https:\/\/www.javacodegeeks.com\/wp-content\/uploads\/2022\/06\/exelixis-logo.png","contentUrl":"https:\/\/www.javacodegeeks.com\/wp-content\/uploads\/2022\/06\/exelixis-logo.png","width":864,"height":246,"caption":"Exelixis Media P.C."},"image":{"@id":"https:\/\/www.javacodegeeks.com\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/javacodegeeks","https:\/\/x.com\/javacodegeeks"]},{"@type":"Person","@id":"https:\/\/www.javacodegeeks.com\/#\/schema\/person\/d15eabf874a51b64dc425a55a88703b5","name":"Andre Torres","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/d31284760a81875079ef459ff625db1c9149c99687d412a4e5ed57aa91fb4d7c?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/d31284760a81875079ef459ff625db1c9149c99687d412a4e5ed57aa91fb4d7c?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/d31284760a81875079ef459ff625db1c9149c99687d412a4e5ed57aa91fb4d7c?s=96&d=mm&r=g","caption":"Andre Torres"},"description":"Andre is a software developer from Brazil who discovered his passion for programming while working as tech support for an ERP in the fashion industry. Later, he became a developer in the same company working with Delphi. In 2017 he decided to move to Europe and landed a job in the gaming industry in London, where he stayed working in the payments team for almost a year, until he joined Codurance.","sameAs":["https:\/\/codurance.com"],"url":"https:\/\/www.javacodegeeks.com\/author\/andre-torres"}]}},"_links":{"self":[{"href":"https:\/\/www.javacodegeeks.com\/wp-json\/wp\/v2\/posts\/99664","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.javacodegeeks.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.javacodegeeks.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.javacodegeeks.com\/wp-json\/wp\/v2\/users\/76873"}],"replies":[{"embeddable":true,"href":"https:\/\/www.javacodegeeks.com\/wp-json\/wp\/v2\/comments?post=99664"}],"version-history":[{"count":0,"href":"https:\/\/www.javacodegeeks.com\/wp-json\/wp\/v2\/posts\/99664\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.javacodegeeks.com\/wp-json\/wp\/v2\/media\/112"}],"wp:attachment":[{"href":"https:\/\/www.javacodegeeks.com\/wp-json\/wp\/v2\/media?parent=99664"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.javacodegeeks.com\/wp-json\/wp\/v2\/categories?post=99664"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.javacodegeeks.com\/wp-json\/wp\/v2\/tags?post=99664"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}