CHAPTER 1

INTRODUCTION

1.1. Identification of Client / Need / Relevant Issue

What is Language Detection?

If the users of your application are multilingual, you naturally have the need to detect which
language they are speaking or writing in. In Natural Language Processing (NLP), language
detection is a computational approach to this problem. We can develop the program to be
effective commercially and is used in machine translation tool, web development, businesses
and hotels... etc.

Let us take an example:

When you ask Siri or Bixby to perform an action, for example, you say “Quel temps fait-il ?",
Siri automatically detects the language which in this case is French and responds appropriately.
For users they give their convenience and time and there is no need for using
extra materials like: Dictionary, Google, etc.

1.2. Identification of Problem

If your work involves regular contact with speakers of foreign languages, being able to talk
to them in their own languages will help you to communicate with them. Here our Language
detection tool will be useful. It may also help you to make sales and to negotiate and secure
contracts.
Knowledge of foreign languages may also increase your chances of finding a new job, getting a
promotion or a transfer overseas, or of going on foreign business trips. Basically it would be
helpful in your personal and company’s overall development.

1.3. Identification of Tasks

Natural Language Processing (NLP) is a field that focuses on making natural human language
usable by computer programs. NLTK, or Natural Language Toolkit, is a Python package that
you can use for NLP.
A lot of the data that you could be analysing is unstructured data and contains human-readable
text. Before you can analyse that data programmatically, you first need to pre-process it. In this
tutorial, you’ll take your first look at the kinds of text pre-processing tasks you can do with
NLTK so that you’ll be ready to apply them in future projects.

Tokenization
One of the very basic things we want to do is dividing a body of text into words or sentences. This
is called tokenization.
from nltk import word_tokenize, sent_tokenizesent = "I will walk
500 miles and I would walk 500 more, just to be the man who walks a
thousand miles to fall down at your
door!"print(word_tokenize(sent))print(sent_tokenize(sent))output:
[‘I’, ‘will’, ‘walk’, ‘500’, ‘miles’, ‘.’, ‘And’, ‘I’, ‘would’,
‘walk’, ‘500’, ‘more’, ‘,’, ‘just’, ‘to’, ‘be’, ‘the’, ‘man’,
‘who’, ‘walks’, ‘a’, ‘thousand’, ‘miles’, ‘to’, ‘fall’, ‘down’,
‘at’, ‘your’, ‘door’, ‘.’][‘I will walk 500 miles.’, ‘And I would
walk 500 more, just to be the man who walks a thousand miles to
fall down at your door.’]

Stemming
This is when ‘fluff’ letters (not words) are removed from a word and grouped together with its
“stem form”. For instance, the words ‘play’, ‘playing’, or ‘plays’ convey the same meaning
(although, again, not exactly, but for analysis with a computer, that sort of detail is still not a
viable option). So instead of having them as different words, we can put them together under the
same umbrella term ‘play’.
from nltk.stem import PorterStemmerstemmer = PorterStemmer()
words = ['play', 'playing', 'plays', 'played',
'playfullness', 'playful']
stemmed = [stemmer.stem(word) for word in words]
print(stemmed)output:
['play', 'play', 'play', 'play', 'playful', 'play']

sent2 = "I played the play playfully as the players were playing in
the play with playfullness"
token = word_tokenize(sent2)
stemmed = ""
for word in token:
stemmed += stemmer.stem(word) + " "
print(stemmed)output:
I play the play play as the player were play in the play with
playful

Tagging Parts of Speech (pos)

The next essential thing we want to do is tagging each word in the corpus (a corpus is just a ‘bag’ of
words) we created after converting sentences by tokenizing.

from nltk import pos_tag token = word_tokenize(sent) +

word_tokenize(sent2)
tagged = pos_tag(cleaned_token)
print(tagged)output:
[('I', 'PRP'), ('will', 'MD'), ('walk', 'VB'), ('500', 'CD'),
('miles', 'NNS'), ('and', 'CC'), ('I', 'PRP'), ('would', 'MD'),
('walk', 'VB'), ('500', 'CD'), ('more', 'JJR'), (',', ','),
('just', 'RB'), ('to', 'TO'), ('be', 'VB'), ('the', 'DT'), ('man',
'NN'), ('who', 'WP'), ('walks', 'VBZ'), ('a', 'DT'), ('thousand',
'NN'), ('miles', 'NNS'), ('to', 'TO'), ('fall', 'VB'), ('down',
'RP'), ('at', 'IN'), ('your', 'PRP$'), ('door', 'NN'), ('.', '.'),
('I', 'PRP'), ('played', 'VBD'), ('the', 'DT'), ('play', 'NN'),
('playfully', 'RB'), ('as', 'IN'), ('the', 'DT'), ('players',
'NNS'), ('were', 'VBD'), ('playing', 'VBG'), ('in', 'IN'), ('the',
'DT'), ('play', 'NN'), ('with', 'IN'), ('playfullness', 'NN'),
('.', '.')]
The pos_tag() method takes in a list of tokenized words, and tags each of them with a
corresponding Parts of Speech identifier into tuples.

1.4 Timeline
1.5 Organization of Report

i) Introduction: In this chapter we mainly give the overview of the project, basically
how Language identification is beneficial and how using python we can identify
different languages.

ii) Literature Review: In this part we mainly define the literature part, the early proposed
solutions, analysis of key features and drawbacks. The goals and objectives will also
be defined in this section.

iii) Design Process: In this part we try to draw various flowchart, try to define various
algorithms. We mainly use these algorithms and flowchart when we basically
implement our project.

iv) Result Analysis and Validation: In this chapter we mainly do the

coding/implementation part of our project. We mainly implement our project by using
various algorithms which we have defined in the design part.

v) Conclusion and future work: In this part we will include the result /outcome of our
project. We will also check whether there is any deviation of expected results with the
predicted result.
 CHAPTER 2
LITERATURE REVIEW

2.1. Timeline
Language identification systems have a long history, dating back to the early days of computing.
Here is a brief timeline of some of the key developments in this field:

1950s: The earliest language identification systems were simple rule-based systems that looked
for patterns in the text or speech. These systems were limited in their accuracy and could only
identify a small number of languages.

1970s: With the development of more powerful computers, researchers began to use statistical
models to improve the accuracy of language identification. These models could identify a larger
number of languages and were more robust to variations in the input data.

1990s: The use of machine learning algorithms, such as artificial neural networks, became more
common in language identification systems. These algorithms could automatically learn patterns
in the data and could be trained to identify a wide range of languages.

2000s: With the growth of the internet and the availability of large amounts of text and speech
data, researchers began to develop more sophisticated language identification systems that used
data-driven approaches. These systems could identify languages more accurately and could also
identify dialects and regional variations.

2010s: Deep learning algorithms, such as convolutional neural networks (CNNs) and recurrent
neural networks (RNNs), became popular in language identification. These algorithms could
handle large amounts of data and could identify languages with high accuracy, even in noisy or
complex environments.

2020s: Language identification systems continue to be refined and improved, with the use of
advanced techniques such as transfer learning and transformer models. These systems are
becoming more accurate and more adaptable to new languages and dialects.

2.2. Existing solutions

One of the earliest proposed solutions for language identification is the frequency-based
approach. This approach involves analyzing the frequency distribution of characters, words, or
n-grams in a given text to determine the language. In a study by Kessler (1995), a frequency-
based language identification system was developed using character n-grams and a decision tree
algorithm. The study showed that the system achieved an accuracy of over 90% on the test set,
but the performance decreased when dealing with short or noisy texts.
 FIGURE 2.1

Another early proposed solution for language identification is the rule-based approach. This
approach involves developing a set of rules or heuristics to determine the language of a given
text. In a study by Nagata (1998), a rule-based language identification system was developed
using language-specific features such as character sets, orthographic rules, and phonetic rules.
The study showed that the system achieved an accuracy of over 95% on the test set, but the
performance decreased when dealing with mixed or dialectal texts.

A third early proposed solution for language identification is the hybrid approach. This
approach involves combining multiple techniques such as frequency-based, rule-based, and
machine learning to develop a language identification system. In a study by Cavnar and Trenkle
(1994), a hybrid language identification system was developed using a combination of character
n-grams, decision trees, and nearest neighbour algorithms. The study showed that the system
achieved an accuracy of over 95% on the test set, but the performance was sensitive to the choice
of algorithms and features.

In conclusion, early proposed solutions for language identification include frequency-based, rule-
based, and hybrid approaches. These approaches have several key features such as simplicity,
interpretability, and accuracy, but they also have some drawbacks such as sensitivity to noise,
language mixtures, and dialectal variations. As NLP techniques and datasets continue to evolve,
new approaches and improvements to existing approaches can be expected to further enhance the
accuracy and robustness of language identifier systems.

2.3. Bibliometric analysis

The main goal of a language identifier project using NLTK is to develop a system that can
accurately identify the language of a given text. The project aims to achieve this goal by using
the Natural Language Toolkit (NLTK), which is a popular Python library for NLP tasks such as
tokenization, part-of-speech tagging, and text classification.
The specific objectives of a language identifier project using NLTK can include the following:
1. Data collection and preprocessing: The project should collect a large and diverse
corpus of text data in different languages and preprocess the data by cleaning, tokenizing,
and normalizing the text.
 2. Feature selection and extraction: The project should select and extract relevant features
from the preprocessed text data that can help distinguish between different languages.
These features can include character n-grams, word n-grams, syntactic patterns, and
semantic features.
3. Model development and training: The project should develop a language identification
model using machine learning or deep learning techniques such as decision trees, support
vector machines, or neural networks. The model should be trained on the preprocessed
and feature-selected data using appropriate evaluation metrics such as accuracy,
precision, recall, and F1-score.
4. Model evaluation and optimization: The project should evaluate the performance of the
language identification model on a test set of unseen data and optimize the model
parameters and hyperparameters to improve its accuracy and robustness.
5. Deployment and integration: The project should deploy the language identification
model as a web service or an API that can be integrated into other NLP applications or
tools. The project should also provide documentation and examples for users to interact
with the deployed model.
Overall, the goals and objectives of a language identifier project using NLTK are to develop a
robust and accurate system that can identify the language of a given text, and to provide a useful
tool for researchers and practitioners in various NLP domains such as machine translation,
sentiment analysis, and information retrieval

2.4. Review Summary

 Automatic Language Identification-2017

Author(s):
Nejla Qafmolla

Automatic Language Identification (LID) is the process of automatically identifying the

language of spoken utterance or written material. LID has received much attention due to its
application to major areas of research and long-aspired dreams in computational sciences,
namely Machine Translation (MT), Speech Recognition (SR) and Data Mining (DM). A
considerable increase in the amount of and access to data provided not only by experts but also
by users all over the Internet has resulted into both the development of different approaches in
the area of LID – so as to generate more efficient systems – as well as major challenges that are
still in the eye of the storm of this field. Despite the fact that the current approaches have
accomplished considerable success, future research concerning some issues remains on the table.
The aim of this paper shall not be to describe the historic background of this field of studies, but
rather to provide an overview of the current state of LID systems, as well as to classify the
approaches developed to accomplish them. LID systems have advanced and are continuously
evolving. Some of the issues that need special attention and improvement are semantics, the
identification of various dialects and varieties of a language, identification of spelling errors, data
retrieval, multilingual documents, MT and speech-to-speech translation. Methods applied to date
have been good from a technical point of view, but not from a semantic one.

 Automatic Detection and Language Identification of Multilingual Documents-2014

Author(s):
Marco Lui
Jey Han Lau
Timothy Baldwin

Language identification is the task of automatically detecting the language(s) present in a

document based on the content of the document. In this work, we address the problem of
detecting documents that contain text from more than one language (multilingual documents).
They introduced a method that is able to detect that a document is multilingual, identify the
languages present, and estimate their relative proportions. They demonstrates the effectiveness of
the method over synthetic data, as well as real-world multilingual documents collected from the
web.

 Improving Transformer Based End-to-End Code-Switching Speech Recognition

Using Language Identification- 2021

Author(s):
Zheying Huang
Pei Wang
Jian Wang
Haoran Miao
Ji Xu
Pengyuan Zhang

A Recurrent Neural Networks (RNN) based attention model has been used in code-switching
speech recognition (CSSR). However, due to the sequential computation constraint of RNN,
there are stronger short-range dependencies and weaker long-range dependencies, which makes
it hard to immediately switch languages in CSSR. Firstly, to deal with this problem, they
introduced the CTC-Transformer, relying entirely on a self-attention mechanism to draw global
dependencies and adopting connectionist temporal classification (CTC) as an auxiliary task for
better convergence. Secondly, they proposed two multi-task learning recipes, where a language
identification (LID) auxiliary task is learned in addition to the CTC-Transformer automatic
speech recognition (ASR) task. Thirdly, study a decoding strategy to combine the LID into an
ASR task. Experiments on the SEAME corpus demonstrate the effects of the proposed methods,
achieving a mixed error rate (MER) of 30.95%. It obtains up to 19.35% relative MER reduction
compared to the baseline RNN-based CTC-Attention system, and 8.86% relative MER reduction
compared to the baseline CTC-Transformer system.

 Multilingual native language identification-2015

Author(s):
Shervin malmasi
Mark dras

Presents first comprehensive study of Native Language Identification (NLI) applied to text
written in languages other than English, using data from six languages. NLI is the task of
predicting an author’s first language using only their writings in a second language, with
applications in Second Language Acquisition and forensic linguistics. Most research to date has
focused on English but there is a need to apply NLI to other languages, not only to gauge its
applicability but also to aid in teaching research for other emerging languages. With this goal,
they identified six typologically very different sources of non-English second language data and
conduct six experiments using a set of commonly used features. First two experiments evaluate
the features and corpora, showing that the features perform well and at similar rates across
languages. The third experiment compares non-native and native control data, showing that they
can be discerned with 95 per cent accuracy. Their fourth experiment provides a cross-linguistic
assessment of how the degree of syntactic data encoded in part-of-speech tags affects their
efficiency as classification features, finding that most differences between first language groups
lie in the ordering of the most basic word categories. They also tackled two questions that have
not previously been addressed for NLI. Other work in NLI has shown that ensembles of
classifiers over feature types work well and in our final experiment they used such an oracle
classifier to derive an upper limit for classification accuracy with our feature set. We also present
an analysis examining feature diversity, aiming to estimate the degree of overlap and
complementarity between the chosen features employing an association measure for binary data.
Finally, concluded with a general discussion and outline directions for future work.

 Exploiting native language interference for native language identification-2020

Author(s):
Ilia Markov
Vivi Nastase
Carlo Strapparava

Abstract Native language identification (NLI)—the task of automatically identifying the native
language (L1) of persons based on their writings in the second language (L2)—is based on the
hypothesis that characteristics of L1 will surface and interfere in the production of texts in L2 to
the extent that L1 is identifiable. They presented an in-depth investigation of features that model
a variety of linguistic phenomena potentially involved in native language interference in the
context of the NLI task: the languages’ structuring of information through punctuation usage,
emotion expression in language, and similarities of form with the L1 vocabulary through the use
of anglicized words, cognates, and other misspellings. The results of experiments with different
combinations of features in a variety of settings allow us to quantify the native language
interference value of these linguistic phenomena and show how robust they are in cross-corpus
experiments and with respect to proficiency. These experiments provide a deeper insight into the
NLI task, showing how native language interference explains the gap between baseline, corpus-
independent features, and the state of the art that relies on features/representations that cover
(indiscriminately) a variety of linguistic phenomena.
2.5. Problem Definition
While language identification systems have made great progress in recent years, there are still
some challenges that need to be addressed. Here are some of the main problems that lie in
language identification systems:

 Ambiguity: Some languages can be very similar to each other and share many common
words or phrases, which makes it difficult for language identification systems to
accurately distinguish between them.

 Code-switching: Many people speak more than one language, and they may switch
between languages in the middle of a conversation. This can confuse language
identification systems and make it difficult for them to determine which language is
being spoken at any given moment.

 Dialects and regional variations: Different regions within a country may have different
dialects or accents that can be very different from the standard language. Language
identification systems need to be able to recognize these variations and accurately
identify the language being spoken.

 Limited training data: Language identification systems rely on large amounts of training
data to learn the patterns and features of each language. However, for many less common
or lesser-known languages, there may not be enough data available to train a reliable
language identification system.

 Multilingualism: In some parts of the world, people speak multiple languages fluently
and may switch between them without even realizing it. This can make it difficult for
language identification systems to accurately identify the languages being spoken.

Noise and background interference: In real-world environments, there may be noise or

interference that can make it difficult for language identification systems to accurately recognize
speech and identify the language being spoken.

2.6. Goals/Objectives
 Develop an accurate and robust language identification system: The primary goal of a
language identifier project is to develop a system that can accurately and reliably identify
the language of a given text. The system should be able to handle a range of text types
and domains and perform well even in noisy and low-resource settings.
 Evaluate and compare different approaches and techniques: A language identifier project
may aim to evaluate and compare different approaches and techniques for language
identification, such as machine learning algorithms, feature extraction methods, and
language models. This can help identify the most effective and efficient methods for
language identification in different contexts.
 Improve multilingual text processing and analysis: Language identification is a crucial
step in multilingual text processing and analysis. By improving language identification, a
language identifier project can contribute to the development of more advanced
multilingual natural language processing tools, such as machine translation and cross-
 lingual information retrieval.
 Address real-world language identification challenges: A language identifier project may
aim to address specific real-world language identification challenges, such as identifying
languages in social media texts or in low-resource settings. By addressing these
challenges, the project can help improve the applicability and usefulness of language
identification systems in real-world contexts.
 Contribute to the advancement of the field: Ultimately, a language identifier project
should aim to contribute to the advancement of the field of natural language processing
and language technology. By developing new methods and techniques for language
identification and conducting rigorous evaluations and comparisons, the project can help
improve the state-of-the-art and drive future research in the field.
 CHAPTER 3
DESIGN FLOW/PROCESS

3.1. Evaluation & Selection of Specifications/Features

When evaluating and selecting specifications/features for a language identification tool, it is
important to consider the following factors:
 The accuracy of the tool. This is the most important factor to consider, as a high-accuracy
tool will be more useful in applications such as machine translation and text
classification.
 The speed of the tool. This is important for applications where the tool needs to be able to
process large amounts of text quickly, such as spam filtering.
 The ease of use of the tool. This is important for users who are not familiar with LI tools.
 The cost of the tool. This is an important factor for businesses and organizations that need
to use LI tools on a large scale.

3.2. Design Constraints

There are several design constraints that need to be considered when creating a language
detection model using NLP. These include:

 The size and quality of the training data: The size and quality of the training data will
have a big impact on the accuracy of the model. It is important to collect a dataset that is
representative of the languages that you want to detect.
 The features that are extracted: The features that are extracted will depend on the
machine learning algorithm that you are using. Some common features include the
frequency of different words, the length of sentences, and the use of punctuation.
 The machine learning algorithm that is used: There are a variety of machine learning
algorithms that can be used to train a language detection model. Some common
algorithms include support vector machines, naive Bayes, and decision trees.
 The evaluation metrics that are used: It is important to evaluate the model on a held-out
dataset to see how well it performs. This will help you to identify any problems with the
model and make necessary adjustments.
 The deployment environment: The model will need to be deployed in a production
environment. This will require you to consider factors such as the speed and accuracy of
the model, as well as the cost of deployment.

3.3. Analysis of Features and finalization subject to constraints

Analysis of Features

The following are some of the most important features to consider when developing a language
identification tool using NLP:

 The number of languages that the tool supports. This is an important factor to consider, as
 the tool will need to be able to identify the language of a text in order to be useful.
 The accuracy of the tool. This is the most important factor to consider, as a high-accuracy
tool will be more useful in applications such as machine translation and text
classification.
 The speed of the tool. This is important for applications where the tool needs to be able to
process large amounts of text quickly, such as spam filtering.
 The ease of use of the tool. This is important for users who are not familiar with language
identification tools.
 The cost of the tool. This is an important factor for businesses and organizations that need
to use language identification tools on a large scale.

Finalization Subject to Constraints

Once the features have been considered, it is possible to finalize the design of the language
identification tool. The following are some of the most important factors to consider when
finalizing the design:

 The constraints on the tool. These constraints may include the number of languages that
the tool needs to support, the accuracy that is required, the speed that is required, the ease
of use that is required, and the cost that is available.
 The trade-offs between the features. It is important to consider the trade-offs between the
features, such as the accuracy vs. the speed of the tool.
 The best practices for developing language identification tools. There are a number of
best practices that can be followed when developing language identification tools, such as
using a large training dataset and using a variety of features.

3.4. Design Flow

3.5. Design Selection

There are many possible designs for a language identifier, but some factors to consider when
designing one include accuracy, speed, and ease of use.

One approach is to use machine learning algorithms to train a model that can identify languages
based on their unique features, such as sentence structure, character sets, and word frequency.
This could involve collecting and preprocessing large amounts of text data in various languages,
then using supervised learning techniques to train the model to recognize patterns and make
predictions.

Another approach is to use a rule-based system that looks for specific language markers, such as
certain words or grammatical constructions that are unique to certain languages. This could
involve developing a set of rules for each language that the identifier needs to recognize.

In terms of implementation, the language identifier could be a standalone application or a

component of a larger system. It could also be integrated into web browsers, chatbots, or other
natural language processing systems.
Overall, the best design for a language identifier will depend on the specific use case and the
resources available for development and training. A well-designed language identifier should be
accurate, fast, and easy to use, and should be able to recognize a wide range of languages.

3.6. Implementation Plan/Methodology

 CHAPTER 4
RESULT ANALYSIS AND VALIDATION
4.1 Implementation of the solution

 A basic language identifier implemented using Google colab:

langdetect supports 55 languages out of the box (ISO 639-1 codes):

af, ar, bg, bn, ca, cs, cy, da, de, el, en, es, et, fa, fi, fr, gu, he,
hi, hr, hu, id, it, ja, kn, ko, lt, lv, mk, ml, mr, ne, nl, no, pa, pl,
pt, ro, ru, sk, sl, so, sq, sv, sw, ta, te, th, tl, tr, uk, ur, vi, zh-cn, zh-tw

 Figure 4.1 Implementation

 First we install the langdetect library using !pip command.
  Then import the nltk and langdetect libraries.
 Then import the detect function from langdetect.
 Write your text which you want to detect the language for.
 Apply the detect function and print the output in next line.

 Here is a comparison of two different language identification models:

Model Accuracy Speed Memory Language

Footprint Coverage
FastText 97% 120k sentences/s Small 159
LangDetect 95% 1100x slower Large 189
than FastText
Table 4.1 Comparison of two models

As you can see, FastText is more accurate than LangDetect, but it is also faster. FastText also has
a smaller memory footprint than LangDetect. However, LangDetect covers more languages than
FastText.

Ultimately, the best language identification model for you will depend on your specific needs. If
you need a model that is accurate and fast, then FastText is a good choice. If you need a model
that covers a large number of languages, then LangDetect is a good choice.

 FastText:
FastText is a library for learning of word embeddings and text classification created by
Facebook's AI Research (FAIR) lab. The model allows one to create an unsupervised learning or
supervised learning algorithm for obtaining vector representations for words. Facebook makes
available pretrained models for 294 languages. Several papers describe the techniques used by
fastText.

FastText represents each word as a bag of character n-grams, where n-grams are contiguous
sequences of characters within a word. By capturing subword information, FastText is able to
handle out-of-vocabulary words and provide meaningful representations for rare or unseen
words. This is particularly useful in morphologically rich languages or when dealing with
misspelled or slang words.

It is based on the word2vec model, but it also includes a character-level n-gram model. This
makes it possible to learn representations for rare words and words that are not present in the
training data.

FastText has been shown to be effective for a variety of natural language processing(NLP) tasks,
including text classification, sentiment analysis, and named entity recognition. It is a popular
choice for research and production applications.

Some of the benefits of using FastText:

  Fast: FastText is very fast, making it possible to train models on large datasets in a
reasonable amount of time.
 Efficient: FastText is also very efficient, making it possible to use models on mobile
devices and other resource-constrained platforms.
 Effective: FastText has been shown to be effective for a variety of natural language
processing tasks.

What is a character n-gram?

A character n-gram is a set of co-occurring characters within a given window. It’s very similar to
word n-grams, only that the window size is at the character level. And a bag of character n-
grams in the FastText case means a word is represented by a sum of its character n-grams.
If n=2, and your word is this your resulting n-grams would be:
<t

th

hi

is

s>

this

The last item is a special sequence. Here’s a visual example of how the neighbouring
word this is represented in learning to predict the word visual based on the sentence “this is
a visual example” (remember: the meaning of a word, is inferred by the company it keeps).

Another use of character n-gram representation is to infer the meaning of unseen words. For
example, if you are looking for the similarity of courageous and your corpora does not carry this
word, you can still infer its meaning from its sub words such as courage.
 Figure 4.1 SkipgramSI

 LangDetect:

LangDetect is a language identification model that uses a statistical approach to identify the
language of a text. It was developed by Nakatani Shuyo and is available as a Python
library.LangDetect works by first identifying the most common n-grams (sequences of n words)
in each language. It then calculates the probability of each n-gram occurring in the text. The
language with the highest probability is then identified as the language of the text.

LangDetect is a relatively simple model, but it is effective at identifying a wide range of

languages. It is also relatively fast and easy to use.

Here are some of the benefits of using LangDetect:

 Simple: LangDetect is a relatively simple model to understand and use.

 Effective: LangDetect is effective at identifying a wide range of languages.

 Fast: LangDetect is relatively fast, making it a good choice for applications where speed
is important.

Here are some additional details about LangDetect:

 N-grams: N-grams are sequences of n words. For example, a 2-gram is a sequence of two
words, such as "the cat".
  Probability:** The probability of an n-gram occurring in a language is calculated by
dividing the number of times the n-gram occurs in the language by the total number of n-
grams in the language.

 Language Identification:** The language of a text is identified by finding the language

with the highest probability of containing the text.

It's important to note that the langdetect model is a statistical model and may not always provide
perfect accuracy, especially when dealing with very short or ambiguous texts. Additionally, it
may struggle with identifying the correct language for texts that contain multiple languages or
code-switching between languages.
 CHAPTER 5
CONCLUSION AND FUTURE WORK
5.1 Conclusion