{"id":44619,"date":"2015-09-27T22:11:26","date_gmt":"2015-09-27T19:11:26","guid":{"rendered":"http:\/\/www.javacodegeeks.com\/?p=44619"},"modified":"2023-12-07T11:00:43","modified_gmt":"2023-12-07T09:00:43","slug":"lucene-analysis-process-guide","status":"publish","type":"post","link":"https:\/\/www.javacodegeeks.com\/2015\/09\/lucene-analysis-process-guide.html","title":{"rendered":"Lucene Analysis Process Guide"},"content":{"rendered":"

This article is part of our Academy Course titled Apache Lucene Fundamentals<\/a>.<\/em><\/p>\n

In this course, you will get an introduction to Lucene. You will see why a library like this is important and then learn how searching works in Lucene. Moreover, you will learn how to integrate Lucene Search into your own applications in order to provide robust searching capabilities. Check it out here<\/a>!<\/p>\n

\n

Table Of Contents<\/h4>\n
\n
1. Introduction<\/a><\/dt>\n
2. Using Analyzers<\/a><\/dt>\n
\n
\n
2.1. Working process of Lucene Analyzer<\/a><\/dt>\n<\/dl>\n<\/dd>\n
3. Types of Analyzers<\/a><\/dt>\n<\/dl>\n<\/div>\n

<\/a>1. Introduction<\/h2>\n

Analysis, in Lucene, is the process of converting field text into its most fundamental indexed representation, terms. In general, the tokens are referred to as words (we are discussing this topic in reference to the English language only) to the analyzers. However, for special analyzers the token can be with more than one words, which includes spaces also. These terms are used to determine what documents match a query during searching. For example, if you indexed this sentence in a field the terms might start with for and example, and so on, as separate terms in sequence. An analyzer is an encapsulation of the analysis process. An analyzer tokenizes text by performing any number of operations on it, which could include extracting words, discarding punctuation, removing accents from characters, lowercasing (also called normalizing), removing common words, reducing words to a root form (stemming), or changing words into the basic form (lemmatization). This process is also called tokenization, and the chunks of text pulled from a stream of text are called tokens. Tokens, combined with their associated field name, are terms.<\/p>\n

<\/a>2. Using Analyzers<\/h2>\n

Lucene’s primary goal is to facilitate information retrieval. The emphasis on retrieval is important. You want to throw gobs of text at Lucene and have them be richly searchable by the individual words within that text. In order for Lucene to know what \u201cwords\u201d are, it analyzes the text during indexing, extracting it to terms. These terms are the primitive building blocks for searching.<\/p>\n

Choosing the right analyzer is a crucial development decision with Lucene, and one size definitely doesn\u2019t fit all. Language is one factor, because each has its own unique features. Another factor to consider is the domain of the text being analyzed; different industries have different terminology, acronyms, and abbreviations that may deserve attention. No single analyzer will suffice for all situations. It\u2019s possible that none of the built-in analysis options are adequate for our needs, and we\u2019ll have to invest in creating a custom analysis solution; fortunately, Lucene\u2018s building blocks make this quite easy.<\/p>\n

In general Lucene Analyzers are designed using the following steps:<\/b><\/p>\n

Actual Text \u2013> Basic Token Preparation \u2013> lower case filtering \u2013> stop word filtering (negation of not so useful words, which comprise in the 40-50% of words in a content) \u2013> Filtering by Custom Logic \u2013> Final Token preparation for indexing in lucene, which will be referenced in the searching of lucene.<\/p>\n

Different analyzers use different tokenizers and on the basis of that, the output token streams \u2013 sequences of group of text will be different.<\/p>\n

Stemmers are used to get the root of a word in question. For example, for the words running, ran, run etc. the root word will be run. This feature is used in analyzers to make the search scope higher in the content by the search api. If the root word is referred in the index then may be for the exact word, we can have more than one option in the index for searching and the probability of phrase matching may be higher here. So this concept, referred to as stemmers, is often used in analyser design.<\/p>\n

Stop words are the frequent, less useful words in written language. For English these words are \u201ca\u201d, \u201cthe\u201d, \u201cI\u201d etc.<\/p>\n

In different analyzers, the token streams are cleaned from Stop-words to make the index more useful for search results.<\/p>\n

<\/a>2.1. Working process of Lucene Analyzer<\/h3>\n

The analysis process has three parts. To help illustrate the process, we are going to use the following raw text as an example. (Processing an HTML or PDF document to obtain the title, main text, and other fields to be analyzed is called parsing and is beyond the scope of analysis. Let\u2019s assume a parser has already extracted this text out of a larger document.)<\/p>\n

<h1>Building a <em>top-notch<\/em> search engine<\/h1><\/pre>\n
First, character filters pre-process the raw text to be analyzed. For example, the HTML Strip character filter removes HTML. We\u2019re now left with this text:<\/p>\n
Building a top-notch search engine<\/pre>\n
Next, a tokenizer breaks up the pre-processed text into tokens. Tokens are usually words, but different tokenizers handle corner cases, such as \u201ctop-notch\u201d, differently. Some tokenizers, such as the Standard tokenizer, consider dashes to be word boundaries, so \u201ctop-notch\u201d would be two tokens (\u201ctop\u201d and \u201cnotch\u201d). Other tokenizers such as the Whitespace tokenizer only considers whitespace to be word boundaries, so \u201ctop-notch\u201d would be a single token. There are also some unusual tokenizers like the NGram tokenizer that generates tokens that are partial words.<\/p>\n
Assuming a dash is considered a word boundary, we now have:<\/p>\n
[Building] [a] [top] [notch] [search] [engine]<\/pre>\n
Finally, token filters perform additional processing on tokens, such as removing suffixes (called stemming) and converting characters to lower case. The final sequence of tokens might end up looking like this:<\/p>\n
[build] [a] [top] [notch] [search] [engine]<\/pre>\n
The combination of a tokenizer and zero or more filters makes up an analyzer. The Standard analyzer, which consists of a Standard tokenizer and the Standard, Lowercase, and Stop token filters, is used by default.<\/p>\n
Analyzers can do more complex manipulations to achieve better results. For example, an analyzer might use a token filter to spell check words or introduce synonyms so that searching for \u201csaerch\u201d or \u201cfind\u201d will both return the document that contained \u201cSearching\u201d. There are also different implementations of similar features to choose from.<\/p>\n
In addition to choosing between the included analyzers, we can create our own custom analyzer by chaining together an existing tokenizer and zero or more filters. The Standard analyzer doesn\u2019t do stemming, so you might want to create a custom analyzer that includes a stemming token filter.
 
<\/div> <\/div><\/p>\n
With so many possibilities, we can test out different combinations and see what works best for our situation.<\/p>\n
<\/a>3. Types of Analyzers<\/h2>\n
In Lucene, some of different analyzers are:<\/p>\n
a. Whitespace analyzer<\/b><\/p>\n
The Whitespace analyzer processes text into tokens based on whitespaces. All characters between whitespaces are indexed. Here, stop words are not used for analyzing and the letter cases are not changed. Whitespace analyzer is built using a Whitespace Tokenizer (a tokenizer of type whitespace that divides text at whitespace).<\/p>\n
b. SimpleAnalyser<\/b><\/p>\n
SimpleAnalyser uses letter tokenizer and lower case filtering to extract tokens from the contents and put it in lucene indexing. Simple analyzer is built using a Lower Case Tokenizer. A lower case tokenizer performs the function of Letter Tokenizer and Lower Case Token Filter together. It divides text at non-letters and converts them to lower case. While it is functionally equivalent to the combination of Letter Tokenizer and Lower Case Token Filter, there is a performance advantage to doing the two tasks at once, hence this (redundant) implementation.<\/p>\n
c. StopAnalyzer<\/b><\/p>\n
StopAnalyser removes common English words that are not so useful for indexing. These are accomplished by providing the analyzer a list of STOP_WORDS lists.<\/p>\n
Stop analyzer is built using lower case tokenizer with Stop token filter (A token filter of type stop that removes stop words from token streams)<\/p>\n
The following are settings that can be set for a stop analyzer type:<\/p>\n