2. ELK<\/h2>\n
It is a collection of three open source products:<\/p>\n
- \n
- Elastic search<\/strong> is a JSON Based NoSQL database<\/li>\n
- Logstash<\/strong> a log pipeline tool that takes inputs from various sources, executes different conversions and exports the data to various targets (here to elastic search)<\/li>\n
- Kibana<\/strong> is a visualization layer that works on top of the elastic search<\/li>\n<\/ul>\n
Refer to the architecture given below:<\/p>\n
\n![\"\"](\"http:\/\/www.javacodegeeks.com\/wp-content\/uploads\/2018\/12\/Pic_1.jpg\")
ELK Stack<\/figcaption><\/figure>\n<\/div>\n Log stash fetches the logs from the microservices. Fetched logs
\nare transformed into JSON and fed to the elastic search. Logs present in elastic search are viewed by the developer using the Kibana.<\/p>\n3. Installing ELK<\/h2>\n
ELK is java based. Before installing ELK it has to be made sure that
JAVA_HOME<\/code> andPATH<\/code> are set up and
\ninstallation is done using JDK 1.8.<\/p>\n3.1 Elasticsearch<\/h3>\n
- \n
- The latest version of Elasticsearch can be downloaded from the download page<\/a> and it can be
\nextracted in any folder<\/li>\n- It can be executed from the command prompt by using
bin\\elasticsearch.bat<\/code> <\/li>\n- By default, it would start at http:\/\/localhost:9200<\/a><\/li>\n<\/ul>\n
3.2 Kibana<\/h3>\n
- \n
- The latest version of Kibana can be downloaded from the download
\npage <\/a>and
\nit can be extracted into any folder<\/li>\n- It can be executed from the command prompt using
bin\\kibana.bat<\/code><\/li>\n- Once started successfully, Kibana will start on default port
5601<\/code> and
\nKibana UI will be available at http:\/\/localhost:5601<\/a><\/li>\n<\/ul>\n3.3 Logstash<\/h3>\n
- \n
- The latest version of Logstash can be downloaded
\nfrom the download page<\/a> and it
\ncan be extracted into any folder<\/li>\n- Create one file cst_logstash.conf<\/strong> as
\nper configuration
\ninstructions<\/a><\/li>\n- It can be executed from the command prompt using
bin\/logstash
\n-f cst_logstash.conf<\/code> to
\nstart logstash<\/li>\n<\/ul>\n4. Creating a sample microservice component <\/h2>\n
It is required to create a microservice so that logstash can
\npoint to the API log. Below listing shows the code for the sample microservice.<\/p>\npom.xml<\/em><\/span><\/p>\n
<project xmlns=\"http:\/\/maven.apache.org\/POM\/4.0.0\" xmlns:xsi=\"http:\/\/www.w3.org\/2001\/XMLSchema-instance\"\n\txsi:schemaLocation=\"http:\/\/maven.apache.org\/POM\/4.0.0 http:\/\/maven.apache.org\/xsd\/maven-4.0.0.xsd\">\n\t<modelVersion>4.0.0<\/modelVersion>\n\t<groupId>com.xyz.app<\/groupId>\n\t<artifactId>ArtDemo1001_Rest_Controller_Full_Deployment_Logging<\/artifactId>\n\t<version>0.0.1-SNAPSHOT<\/version>\n\t<!-- Add Spring repositories -->\n\t<!-- (you don't need this if you are using a .RELEASE version) -->\n\t<repositories>\n\t\t<repository>\n\t\t\t<id>spring-snapshots<\/id>\n\t\t\t<url>http:\/\/repo.spring.io\/snapshot<\/url>\n\t\t\t<snapshots>\n\t\t\t\t<enabled>true<\/enabled>\n\t\t\t<\/snapshots>\n\t\t<\/repository>\n\t\t<repository>\n\t\t\t<id>spring-milestones<\/id>\n\t\t\t<url>http:\/\/repo.spring.io\/milestone<\/url>\n\t\t<\/repository>\n\t<\/repositories>\n\t<pluginRepositories>\n\t\t<pluginRepository>\n\t\t\t<id>spring-snapshots<\/id>\n\t\t\t<url>http:\/\/repo.spring.io\/snapshot<\/url>\n\t\t<\/pluginRepository>\n\t\t<pluginRepository>\n\t\t\t<id>spring-milestones<\/id>\n\t\t\t<url>http:\/\/repo.spring.io\/milestone<\/url>\n\t\t<\/pluginRepository>\n\t<\/pluginRepositories>\n\t<parent>\n\t\t<groupId>org.springframework.boot<\/groupId>\n\t\t<artifactId>spring-boot-starter-parent<\/artifactId>\n\t\t<version>1.5.2.RELEASE<\/version>\n\t<\/parent>\n\t<properties>\n\t\t<project.build.sourceEncoding>UTF-8<\/project.build.sourceEncoding>\n\t\t<project.reporting.outputEncoding>UTF-8<\/project.reporting.outputEncoding>\n\t\t<java.version>1.8<\/java.version>\n\t\t<spring-cloud.version>Dalston.SR3<\/spring-cloud.version>\n\t<\/properties>\n\t<!-- Add typical dependencies for a web application -->\n\t<dependencies>\n\t\t<dependency>\n\t\t\t<groupId>org.springframework.boot<\/groupId>\n\t\t\t<artifactId>spring-boot-starter-web<\/artifactId>\n\t\t<\/dependency>\n\t<\/dependencies>\n\n\t<!-- Package as an executable jar -->\n\t<build>\n\t\t<plugins>\n\t\t\t<plugin>\n\t\t\t\t<groupId>org.springframework.boot<\/groupId>\n\t\t\t\t<artifactId>spring-boot-maven-plugin<\/artifactId>\n\t\t\t<\/plugin>\n\t\t<\/plugins>\n\t<\/build>\n\n\t<dependencyManagement>\n\t\t<dependencies>\n\t\t\t<dependency>\n\t\t\t\t<groupId>org.springframework.cloud<\/groupId>\n\t\t\t\t<artifactId>spring-cloud-dependencies<\/artifactId>\n\t\t\t\t<version>${spring-cloud.version}<\/version>\n\t\t\t\t<type>pom<\/type>\n\t\t\t\t<scope>import<\/scope>\n\t\t\t<\/dependency>\n\t\t<\/dependencies>\n\t<\/dependencyManagement>\n\n<\/project>\n<\/pre>\nAbove code of the pom.xml has configured dependencies that are required by a spring boot based project.
<\/div> <\/div><\/p>\nEmployeeDAO.java<\/em><\/span>
\n<\/p>\npackage com.xyz.app.dao;\n\nimport java.util.Collection;\nimport java.util.LinkedHashMap;\nimport java.util.Map;\n\nimport org.springframework.stereotype.Repository;\n\nimport com.xyz.app.model.Employee;\n\n@Repository\npublic class EmployeeDAO {\n\t\/**\n\t * Map is used to Replace the Database \n\t * *\/\n\tstatic public Map<Integer,Employee> mapOfEmloyees = \n new LinkedHashMap<Integer,Employee>();\n\tstatic int count=10004;\n\tstatic\n\t{\n\t\tmapOfEmloyees.put(10001, new Employee(\"Jack\",10001,12345.6,1001));\n\t\tmapOfEmloyees.put(10002, new Employee(\"Justin\",10002,12355.6,1002));\n\t\tmapOfEmloyees.put(10003, new Employee(\"Eric\",10003,12445.6,1003));\n\t}\n\t\n\t\/**\n\t * Returns all the Existing Employees\n\t * *\/\n\tpublic Collection getAllEmployee(){\n\t\treturn mapOfEmloyees.values();\t\t\t\n\t}\n\t\n\n\t\/**Get Employee details using EmployeeId .\n\t * Returns an Employee object response with Data if Employee is Found\n\t * Else returns a null\n\t * *\/\n\tpublic Employee getEmployeeDetailsById(int id){\n\t\treturn mapOfEmloyees.get(id);\n\t}\n\t\/**Create Employee details.\n\t * Returns auto-generated Id\n\t * *\/\n\tpublic Integer addEmployee(Employee employee){\n\t\tcount++;\n\t\temployee.setEmployeeId(count);\n\t\tmapOfEmloyees.put(count, employee);\n\t\treturn count;\n\t}\n\t\n\t\/**Update the Employee details,\n\t * Receives the Employee Object and returns the updated Details \n\t * *\/\n\tpublic Employee updateEmployee (Employee employee){\n\t\tmapOfEmloyees.put(employee.getEmployeeId(), employee);\n\t\treturn employee;\n\t}\n\t\/**Delete the Employee details,\n\t * Receives the EmployeeID and returns the deleted employee's Details \n\t * *\/\n\tpublic Employee removeEmployee (int id){\n\t\tEmployee emp= mapOfEmloyees.remove(id);\n\t\treturn emp;\n\t}\n\t\n}\n<\/pre>\nAbove code represents the DAO layer of the application. CRUD operations are executed on Map collection containing Employee objects, in order to avoid database dependency and keep the application light. <\/p>\n
EmployeeController.java<\/em><\/span>
\n<\/p>\npackage com.xyz.app.controller;\n\nimport java.util.Collection;\n\nimport org.apache.log4j.Logger;\nimport org.springframework.beans.factory.annotation.Autowired;\nimport org.springframework.http.HttpStatus;\nimport org.springframework.http.MediaType;\nimport org.springframework.http.ResponseEntity;\nimport org.springframework.web.bind.annotation.PathVariable;\nimport org.springframework.web.bind.annotation.RequestBody;\nimport org.springframework.web.bind.annotation.RequestMapping;\nimport org.springframework.web.bind.annotation.RequestMethod;\nimport org.springframework.web.bind.annotation.RestController;\n\nimport com.xyz.app.dao.EmployeeDAO;\nimport com.xyz.app.model.Employee;\n\n@RestController\npublic class EmployeeController {\n\t\n\t@Autowired \n\tprivate EmployeeDAO employeeDAO;\n\t\n\t\n\tpublic static Logger logger = Logger.getLogger(EmployeeController.class);\n\t\n\t\/** Method is used to get all the employee details and return the same \n\t *\/ \n\t@RequestMapping(value=\"emp\/controller\/getDetails\",method=RequestMethod.GET,produces=MediaType.APPLICATION_JSON_VALUE)\n\tpublic ResponseEntity<Collection> getEmployeeDetails(){\n\t\tlogger.info(\"From Producer method[getEmployeeDetails] start\");\n\t\t\tlogger.debug(\"From Producer method[getEmployeeDetails] start\");\n\t\t\tCollection listEmployee =employeeDAO.getAllEmployee();\n\t\t\tlogger.debug(\"From Producer method[getEmployeeDetails] start\");\n\t\tlogger.info(\"From Producer method[getEmployeeDetails] end\");\n\t\treturn new ResponseEntity<Collection>(listEmployee, HttpStatus.OK);\n\t}\n\t\/** Method finds an employee using employeeId and returns the found Employee \n\t\tIf no employee is not existing corresponding to the employeeId, \n\t\tthen null is returned with HttpStatus.INTERNAL_SERVER_ERROR as status\n\t *\/ \n\t@RequestMapping(value=\"emp\/controller\/getDetailsById\/{id}\",method=RequestMethod.GET,produces=MediaType.APPLICATION_JSON_VALUE)\n\tpublic ResponseEntity getEmployeeDetailByEmployeeId(@PathVariable(\"id\") int myId){\n\t\tlogger.info(\"From Producer method[getEmployeeDetailByEmployeeId] start\");\n\t\tEmployee employee = employeeDAO.getEmployeeDetailsById(myId);\n\t\tif(employee!=null)\n\t\t{\n\t\t\tlogger.info(\"From Producer method[getEmployeeDetailByEmployeeId] end\");\n\t\t\treturn new ResponseEntity(employee,HttpStatus.OK);\n\t\t}\n\t\telse\n\t\t{\n\t\t\tlogger.info(\"From Producer method[getEmployeeDetailByEmployeeId] end\");\n\t\t\treturn new ResponseEntity(HttpStatus.NOT_FOUND);\n\t\t}\n\t\t\n\t}\n\t\n\t\/** Method creates an employee and returns the auto-generated employeeId *\/ \n\t@RequestMapping(value=\"\/emp\/controller\/addEmp\",\n\t\t\tmethod=RequestMethod.POST,\n\t\t\tconsumes=MediaType.APPLICATION_JSON_VALUE,\n\t\t\tproduces=MediaType.TEXT_HTML_VALUE)\n\tpublic ResponseEntity addEmployee(@RequestBody Employee employee){\n\t\tlogger.info(\"From Producer method[addEmployee] start\");\n\t\t\tlogger.debug(\"From Producer method[addEmployee] start\");\n\t\t\tint empId= employeeDAO.addEmployee(employee);\n\t\t\tlogger.debug(\"From Producer method[addEmployee] start\");\n\t\tlogger.info(\"From Producer method[addEmployee] end\");\n\t\treturn new ResponseEntity(\"Employee added successfully with id:\"+empId,HttpStatus.CREATED);\n\t}\n\n\t\/** Method updates an employee and returns the updated Employee \n \t\tIf Employee to be updated is not existing, then null is returned with \n \t\tHttpStatus.INTERNAL_SERVER_ERROR as status\n\t *\/ \n\t@RequestMapping(value=\"\/emp\/controller\/updateEmp\",\n\t\t\tmethod=RequestMethod.PUT,\n\t\t\tconsumes=MediaType.APPLICATION_JSON_VALUE,\n\t\t\tproduces=MediaType.APPLICATION_JSON_VALUE)\n\tpublic ResponseEntity updateEmployee(@RequestBody Employee employee){\n\t\tlogger.info(\"From Producer method[updateEmployee] start\");\n\t\tif(employeeDAO.getEmployeeDetailsById(employee.getEmployeeId())==null){\n\t\t\tEmployee employee2=null;\n\t\t\treturn new ResponseEntity(employee2,HttpStatus.INTERNAL_SERVER_ERROR);\n\t\t}\n\t\tSystem.out.println(employee);\n\t\temployeeDAO.updateEmployee(employee);\n\t\tlogger.info(\"From Producer method[updateEmployee] end\");\n\t\treturn new ResponseEntity(employee,HttpStatus.OK);\n\t}\n\t\n\t\/** Method deletes an employee using employeeId and returns the deleted Employee \n\t \tIf Employee to be deleted is not existing, then null is returned with \n\t \tHttpStatus.INTERNAL_SERVER_ERROR as status\n\t *\/ \n\t@RequestMapping(value=\"\/emp\/controller\/deleteEmp\/{id}\",\n\t\t\tmethod=RequestMethod.DELETE,\n\t\t\tproduces=MediaType.APPLICATION_JSON_VALUE)\n\tpublic ResponseEntity deleteEmployee(@PathVariable(\"id\") int myId){\n\t\tlogger.info(\"From Producer method[deleteEmployee] start\");\n\t\tif(employeeDAO.getEmployeeDetailsById(myId)==null){\n\t\t\tEmployee employee2=null;\n\t\t\treturn new ResponseEntity(employee2,HttpStatus.INTERNAL_SERVER_ERROR);\n\t\t}\n\t\tEmployee employee = employeeDAO.removeEmployee(myId);\n\t\tSystem.out.println(\"Removed: \"+employee);\n\t\tlogger.info(\"From Producer method[deleteEmployee] end\");\n\t\treturn new ResponseEntity(employee,HttpStatus.OK);\n\t}\n}\n<\/pre>\nAbove code represents the controller layer of the application with the request handlers. Request handlers invoke the DAO layer functions and perform the CRUD operations.
\napplication.properties<\/em><\/span>
\n<\/p>\nserver.port = 8090\n logging.level.com.xyz.app.controller.EmployeeController=DEBUG\n #name of the log file to be created\n #same file will be given as input to logstash\n logging.file=app.log\n spring.application.name = producer\n<\/pre>\n
Above code represents the properties configured for a spring boot based application.<\/p>\n
5. Logstash configuration<\/h2>\n
As discussed in section 3.3 it is required to create a configuration file for logstash. This configuration file will be used by logstash to take input from the microservice logs. Logs are transformed into JSON and fed to elasticsearch.<\/p>\n
\ncst_logstash.conf
\n<\/em><\/span>
\n<\/p>\ninput {\n file {\n # If more than one log files from different microservices have to be tracked then a comma-separated list of log files can \n # be provided\n path => [\"PATH-TO-UPDATE\/app.log\"]\n codec => multiline {\n pattern => \"^%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{TIME}.*\"\n negate => \"true\"\n what => \"previous\"\n }\n }\n}\noutput {\n stdout {\n codec => rubydebug\n }\n # Sending properly parsed log events to elasticsearch\n elasticsearch {\n hosts => [\"localhost:9200\"]\n }\n}\n<\/pre>\n\nThe above logstash configuration file listens to
\nthe log file and pushes log messages to elastic search.
Note<\/strong>: Change the log path as per to your setup.<\/p>\n6. Execution and Output<\/h2>\n
6.1 Executing microservice for logs<\/h3>\n
Spring Boot application can be deployed using
clean install spring-boot:run<\/code> and following
\nURL can be hit from a browser or postman client: http:\/\/localhost:8090\/emp\/controller\/getDetails<\/a>. This will hit the microservice and produce logs on the microservice’s side. These logs will be read by logstash and pushed to elastic search and further, these logs can be viewed using Kibana using next steps.<\/p>\n6.2 Steps to view output on Kibana<\/h3>\n
- \n
- Configure the index in
\nthe management console. Use index value aslogstash-*<\/strong><\/code> as a default
\nconfiguration. Open the link: http:\/\/localhost:5601\/app\/kibana#\/management\/kibana\/index?_g=()<\/a> and it will display
\nthe screen as shown below:<\/li>\n<\/ul>\n\n![\"\"](\"http:\/\/www.javacodegeeks.com\/wp-content\/uploads\/2018\/12\/Kib_1_1.jpg\")
Kibana Index Creation- 1<\/figcaption><\/figure>\n<\/div>\n <\/p>\n
- \n
- Click on the Next Step and
\nthe following screen is displayed<\/li>\n<\/ul>\n\n![\"\"](\"http:\/\/www.javacodegeeks.com\/wp-content\/uploads\/2018\/12\/Kib_2_2.jpg\")
Kibana Index Creation- 2<\/figcaption><\/figure>\n<\/div>\n Select the option as highlighted above and click on “Create index pattern”<\/p>\n
- \n
- On selecting Discover option from the left side menu, the page is displayed as shown below:<\/li>\n<\/ul>\n\n
![\"\"](\"http:\/\/www.javacodegeeks.com\/wp-content\/uploads\/2018\/12\/Kib_3_3.jpg\")
Viewing Logs on Kibana- 1<\/figcaption><\/figure>\n<\/div>\n <\/p>\n
- \n
- Logs can be visualized and filtered based upon the properties highlighted above. Upon hovering on any of the properties, “add” button for the property is displayed. Here on selecting the message property view is displayed as shown below:<\/li>\n<\/ul>\n\n
![\"\"](\"http:\/\/www.javacodegeeks.com\/wp-content\/uploads\/2018\/12\/Kib_4_4.jpg\")
Viewing Logs on Kibana- 2<\/figcaption><\/figure>\n<\/div>\n 7. References<\/h2>\n
- Logs can be visualized and filtered based upon the properties highlighted above. Upon hovering on any of the properties, “add” button for the property is displayed. Here on selecting the message property view is displayed as shown below:<\/li>\n<\/ul>\n
- On selecting Discover option from the left side menu, the page is displayed as shown below:<\/li>\n<\/ul>\n
- Click on the Next Step and
- Create one file cst_logstash.conf<\/strong> as
- It can be executed from the command prompt using
- It can be executed from the command prompt by using
- Logstash<\/strong> a log pipeline tool that takes inputs from various sources, executes different conversions and exports the data to various targets (here to elastic search)<\/li>\n