{"id":65025,"date":"2017-04-12T13:00:31","date_gmt":"2017-04-12T10:00:31","guid":{"rendered":"http:\/\/www.javacodegeeks.com\/?p=65025"},"modified":"2023-12-11T10:18:28","modified_gmt":"2023-12-11T08:18:28","slug":"elasticsearch-java-developers-elasticsearch-ecosystem","status":"publish","type":"post","link":"https:\/\/www.javacodegeeks.com\/2017\/04\/elasticsearch-java-developers-elasticsearch-ecosystem.html","title":{"rendered":"Elasticsearch for Java Developers: Elasticsearch Ecosystem"},"content":{"rendered":"

This article is part of our Academy Course titled Elasticsearch Tutorial for Java Developers<\/a>.<\/p>\n

In this course, we provide a series of tutorials so that you can develop your own Elasticsearch based applications. We cover a wide range of topics, from installation and operations, to Java API Integration and reporting. With our straightforward tutorials, you will be able to get your own projects up and running in minimum time. Check it out here<\/a>!<\/em><\/p>\n

<\/a>1. Introduction<\/h2>\n

In this last part of the tutorial we are going to look around and learn how perfectly Elasticsearch<\/a> fits into Java ecosystem and inspires many interesting projects. One of the best ways to illustrate that is to take a look at the marriage of Elasticsearch<\/a> and Hibernate<\/a> framework, an exceptionally beloved choice among Java developers for managing the persistence layer.<\/p>\n

\n

Table Of Contents<\/h3>\n
\n
1. Introduction<\/a><\/dt>\n
2. Elasticsearch for Hibernate Users<\/a><\/dt>\n
3. Elastic Stack: Get It All<\/a><\/dt>\n
4. Supercharge Elasticsearch with Plugins<\/a><\/dt>\n
5. Conclusions<\/a><\/dt>\n<\/dl>\n<\/div>\n

Additionally, at the very end we are going to glance through a tremendously popular suite of the applications, known as Elastic Stack<\/a>, and what you could do with it. Although it goes well beyond Java applications only, it is hard to overestimate the value it offers to modern, highly distributed software systems.<\/p>\n

<\/a>2. Elasticsearch for Hibernate Users<\/h2>\n

It is practically impossible to find any Java developer out there who has not heard about the Hibernate<\/a> framework. On the other hand, not so many developers know that there are quite a few projects hidden under the Hibernate<\/a> umbrella and one of them is a real gem called Hibernate Search<\/a>.<\/p>\n

\n

Hibernate Search transparently indexes your objects and offers fast regular, full-text and geolocation search. Ease of use and easy clustering are core. – <\/em>http:\/\/hibernate.org\/search\/<\/em><\/a><\/p>\n<\/blockquote>\n

Hibernate Search<\/a> has started as a simple glue layer between Hibernate<\/a> and Apache Lucene<\/a> and used to offer a very limited set of supported backends to manage search indices. But the situation is changing for the better with a just recently<\/a> released final version of the Hibernate Search<\/a> 5.7.0<\/code> along with full-fledged Elasticsearch<\/a> support (while still bearing an experimental<\/code> label). What it practically means is that if your persistence layer is managed by Hibernate<\/a>, then by plugging in Hibernate Search<\/a> you could enrich your data model with full-text search capabilities, all that backed by Elasticsearch<\/a>. Sounds exciting, right?<\/p>\n

To get a feeling of how things work under the hood, let us take a look at our catalog<\/code> data model expressed in terms of JPA<\/a> entities decorated with Hibernate Search<\/a> annotations, starting with the Book<\/code> class.<\/p>\n

@Entity\n@Table(name = \"BOOKS\")\n@Indexed(index = \"catalog\")\npublic class Book {\n    @Id\n    @Field(name = \"isbn\", analyze = Analyze.NO)\n    private String id;\n\n    @Field\n    @Column(name = \"TITLE\", nullable = false)\n    private String title;\n\n    @IndexedEmbedded(depth = 1)\n    @ElementCollection\n    private Set categories = new HashSet<>(); \n\n    @Field(analyze = Analyze.NO)\n    @Column(name = \"PUBLISHER\", nullable = false)\n    private String publisher;\n    \n    @Field\n    @Column(name = \"DESCRIPTION\", nullable = false, length = 4096)\n    private String description;\n    \n    @Field(name = \"published_date\", analyze = Analyze.NO)\n    @Column(name = \"PUBLISHED_DATE\", nullable = false)\n    @DateBridge(resolution = Resolution.DAY)\n    private LocalDate publishedDate;\n\n    @NumericField @Field(name = \"rating\")\n    @Column(name = \"RATING\", nullable = false)\n    private int rating;\n    \n    @IndexedEmbedded\n    @ManyToMany\n    private Set authors = new HashSet();    \n}\n<\/pre>\n
For seasoned Java developers it is a familiar piece of code to describe persistent entities, with only a couple of Hibernate Search<\/a> annotations (like @Field<\/code>, @DateBridge<\/code>, @IndexedEmbedded<\/code>) added on top. \u00a0We are not going to discuss them here unfortunately, this topic itself is worth of a complete tutorial<\/a> but please do not hesitate to refer to the official documentation<\/a> for more details. With that being said, we just move on to the Category<\/code> class.<\/p>\n
@Embeddable\npublic class Category {\n\t@Field(analyze = Analyze.NO)\n\t@Column(name = \"NAME\", nullable = false)\n\tprivate String name;\t\n}\n<\/pre>\n
Followed by the Author<\/code> class right after.<\/p>\n
@Entity\n@Indexed(index = \"catalog\")\n@Table(name = \"AUTHORS\")\npublic class Author {\n    @Id\n    private String id;\n\n    @Field(name = \"first_name\", analyze = Analyze.NO)\n    @Column(name = \"FIRST_NAME\", nullable = false)\n    private String firstName;\n\n    @Field(name = \"last_name\", analyze = Analyze.NO)\n    @Column(name = \"LAST_NAME\", nullable = false)\n    private String lastName;\n}\n<\/pre>\n
Thanks to Spring Framework<\/a>, and particularly to Spring Boot<\/a> magic, the configuration of the Hibernate<\/a> and Hibernate Search<\/a> pointing to Elasticsearch<\/a> as the search backend is as easy as adding a couple of lines to application.yml<\/code> file.<\/p>\n
spring:\n  jpa:\n    properties:\n      hibernate:\n        search:\n          default:\n            indexmanager: elasticsearch\n            elasticsearch:\n                host: http:\/\/localhost:9200\n<\/pre>\n
Frankly speaking, you may need to tailor this configuration quite a bit to fit the needs of your applications, to our luck the official documentation<\/a> covers this part pretty well. Every time the instances of the Book<\/code> or Author<\/code> classes are created, modified or deleted, the Hibernate Search<\/a> takes care of keeping Elasticsearch<\/a> index in sync, it is completely transparent.<\/p>\n
How the search looks like? Well, Hibernate Search<\/a> does provide its own Query DSL<\/a> abstraction layer, based on Apache Lucene<\/a> queries (at the same time leaving a route<\/a> to use some native Elasticsearch<\/a> features), for example:
 
<\/div> <\/div><\/p>\n
@Autowired private EntityManager entityManager;\n\t\nfinal FullTextEntityManager fullTextEntityManager = Search\n    .getFullTextEntityManager(entityManager);\n\t\t\nfinal QueryBuilder qb = fullTextEntityManager\n    .getSearchFactory()\n    .buildQueryBuilder()\n    .forEntity(Book.class)\n    .get();\n\t\t\nfinal FullTextQuery query = fullTextEntityManager\n    .createFullTextQuery(\n        qb.bool()\n            .must(\n                qb.keyword()\n                    .onField(\"categories.name\")\n                    .matching(\"analytics\")\n                    .createQuery()\n            )\n            .must(\n                qb.keyword()\n                    .onField(\"authors.last_name\")\n                    .matching(\"Tong\")\n                    .createQuery()\n            ).createQuery(), Book.class);\n\t\t\nfinal List books = query.getResultList();\n...\n<\/pre>\n
There are definitely a lot of similarities to Elasticsearch’s<\/a> own Query DSL<\/a> so this code snippet should look already familiar to you. But before you get too excited, there are several limitations related to Hibernate Search<\/a> and Elasticsearch<\/a> integration. First of all, the latest version of Elasticsearch<\/a> which is supported by Hibernate Search<\/a> at the moment is 2.4.4<\/code>. Not bad but quite far from the current 5.x<\/code> release branch, hopefully this is going to be addressed soon. Secondly, indeed, the subset of the Elasticsearch<\/a> features exposed over the Hibernate Search<\/a> APIs, in particular Query DSL<\/a>, is quite limited but frankly speaking, might be enough for many applications.<\/p>\n
Anyway, why we are mentioning Hibernate Search<\/a> in the first place? Simple, if your applications are built on top of Hibernate<\/a> persistence, using Hibernate Search<\/a> is probably the fastest and cheapest way to benefit from full-text search capabilities for your data models, leveraging Elasticsearch<\/a> behind the curtain.<\/p>\n
<\/a>3. Elastic Stack: Get It All<\/h2>\n
If you already run into mysterious ELK<\/a> abbreviation and were curious what it means than this section would help you to find the answers. ELK<\/a> is essentially a bundle of products, consisting of (E)lasticsearch<\/a>, (L)ogstash<\/a> and (K)ibana<\/a>, therefore just ELK<\/a> in short. Recently, with the addition of the Beats<\/a>, a new member of this awesome family, the ELK<\/a> is often referred as Elastic Stack<\/a> now.<\/p>\n
Undoubtedly, Elasticsearch<\/a> is the heart and soul of ELK<\/a> so let us talk about what those other products are and why they are useful.<\/p>\n
\n
Kibana<\/em><\/a> lets you visualize your <\/em>Elasticsearch<\/em><\/a> data and navigate the <\/em>Elastic Stack<\/em><\/a>, so you can do anything from learning why you’re getting paged at 2:00 a.m. to understanding the impact rain might have on your quarterly numbers. – <\/em>https:\/\/www.elastic.co\/products\/kibana<\/em><\/a><\/p>\n<\/blockquote>\n
Basically, Kibana<\/a> is just a web application which is capable of creating powerful charts and dashboards based on the data you are having indexed in Elasticsearch<\/a>.<\/p>\n
\n
Logstash<\/em><\/a> is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to your favorite \u201cstash\u201d, f.e. <\/em>Elasticsearch<\/em><\/a> – <\/em>https:\/\/www.elastic.co\/products\/logstash<\/em><\/a><\/p>\n<\/blockquote>\n
Consequently, Logstash<\/a> is a terrific tool, capable of extracting, massaging and feeding the data to Elasticsearch<\/a> (and a myriad of other sources) which could be visualized using Kibana<\/a> later on. The Beats<\/a> are quite close to Logstash<\/a> but are not yet so powerful.<\/p>\n
\"Elastic<\/a>
Elastic Stack Illustrated<\/figcaption><\/figure><\/p>\n
One of the areas where Elastic Stack<\/a> is exceptionally useful and leads the race is collecting and analyzing the massive amounts of application logs. It may sound not very convincing, why would you need such a complex system in order to tail\/grep over a log file? But at scale, when you deal with hundreds or even thousands of the applications (think microservices<\/a>), the benefits become very obvious: you suddenly have a centralized places where the logs from all the applications are streamed into and could be searched against, analyzed, correlated and visualized.<\/p>\n
Without any further talks, let us demonstrate how a typical Spring Boot<\/a> application could be configured to send its logs to Logstash<\/a>, which is going to forward them to Elasticsearch<\/a> as-is, without \u00a0transformations applied. First, we need Logstash<\/a> to be installed, either as Docker container<\/a> or just running on local machine<\/a>, the official documentation<\/a> presents the installation steps exceptionally well.<\/p>\n
The only thing we need to tell to Logstash<\/a> is where to get logs from (using input plugins<\/a>) and where to send the massaged logs to (using output plugins<\/a>), all that through logstash.conf<\/code> configuration file.<\/p>\n
input {\n    tcp {\n        port => 7760\n    }\n}\n\noutput {\n    elasticsearch {\n        hosts => [ \"localhost:9200\" ]\n    }\n}\n<\/pre>\n
The number of input and output plugins supported by Logstash<\/a> is astonishing. To keep the example very simple, we are going to deliver the logs over the TCP socket input plugin<\/a> and forward straight to Elasticsearch<\/a> using Elasticsearch output plugin<\/a>.<\/p>\n
It looks great, but how we could send the logs from our Java applications to Logstash<\/a>? There are many ways to do that and the easiest one is probably by leveraging the capabilities of your logging framework. Most of the Java applications these days rely on the awesome Logback<\/a> framework and the community has implemented dedicated Logback encoder<\/a> to be used along with Logstash<\/a>.<\/p>\n
You just have to include an additional dependency into your project, like we do here, for example, using Apache Maven<\/a>:<\/p>\n
<dependency>\n    <groupId>net.logstash.logback<\/groupId>\n    <artifactId>logstash-logback-encoder<\/artifactId>\n    <version>4.9<\/version>\n<\/dependency>\n<\/pre>\n
And then add the Logstash<\/a> appender into your logback.xml<\/code> configuration file. To be noted, there are several appenders available, the one we are interested in is LogstashTcpSocketAppender<\/code> which talks to Logstash<\/a> over TCP socket. Please notice that the port under destination<\/code> tag should match to your Logstash<\/a> input plugin configuration, in our case it is 7760<\/code>.<\/p>\n
<appender name=\"logstash\" class=\"net.logstash.logback.appender.LogstashTcpSocketAppender\">\n    <destination>localhost:7760<\/destination>\n    <encoder class=\"net.logstash.logback.encoder.LogstashEncoder\" \/>\n<\/appender>\n<\/pre>\n
By and large, this is all we have to do! The logs will be shipped from our application to Elasticsearch<\/a> and we could explore them using Kibana<\/a> dashboards. When you download<\/a> and run Kibana<\/a> on your local machine, the web UI by default is available at http:\/\/localhost:5601<\/a>:<\/p>\n
\"Quick<\/a>
Quick Kibana Dashboard<\/figcaption><\/figure><\/p>\n
Easy, simple, powerful \u2026 the Elasticsearch<\/a> way. The only thing to remember is that you should better use the same versions of Elasticsearch<\/a> , Logstash<\/a> and Kibana<\/a>. As we have been using Elasticsearch<\/a> 5.2.0<\/code> along this tutorial, Logstash<\/a> and Kibana<\/a> should also be of 5.2.0<\/code> release.<\/p>\n
If you already using Elasticsearch<\/a> or are planning to do so, Elastic Stack<\/a> just opens a whole universe of interesting opportunities for you to discover and benefit from. Moreover, it is being constantly improved and enhanced with new features added every single release.
\n[ulp id=’lFK4fMZhpULqeQUl’]
\n <\/p>\n
<\/a>4. Supercharge Elasticsearch with Plugins<\/h2>\n
Elasticsearch<\/a> is wonderful but often command line tools or even Java APIs are not the best way to communicate with your clusters. Luckily, Elasticsearch<\/a> has extensibility built-in from the early days in a form of plugins.<\/p>\n
There are many plugins<\/a> and accompanying tools available at the moment, but it is worth to talk about three of them:<\/p>\n