{"@attributes":{"version":"2.0"},"channel":{"title":"indicLP","link":"https:\/\/aakash-ez.github.io\/indicLP-docs\/","description":"Recent content on indicLP","generator":"Hugo -- gohugo.io","language":"en-us","lastBuildDate":"Fri, 29 Oct 2021 00:01:00 +0100","item":[{"title":"Tokenizing Sentences","link":"https:\/\/aakash-ez.github.io\/indicLP-docs\/blog\/tokenizer\/","pubDate":"Fri, 29 Oct 2021 00:00:00 +0100","guid":"https:\/\/aakash-ez.github.io\/indicLP-docs\/blog\/tokenizer\/","description":"Getting Started with indicLP IndicLP (Indic Language Processing) Library has been developed to act as a complete toolkit for programmers and researchers who are working on NLP projects in Indic Languages. Therefore, one of the most necessary functionalities that it supports is tokenization. Let us first consider what tokenization is, stanford NLP defines tokenization as:\n Given a character sequence and a defined document unit, tokenization is the task of chopping it up into pieces, called tokens , perhaps at the same time throwing away certain characters, such as punctuation."},{"title":"Quick Start","link":"https:\/\/aakash-ez.github.io\/indicLP-docs\/docs\/quick-start\/","pubDate":"Mon, 01 Jan 0001 00:00:00 +0000","guid":"https:\/\/aakash-ez.github.io\/indicLP-docs\/docs\/quick-start\/","description":"\nInstallation\nindicLP is an easy to install library, making itself available through pip for all python users, while also providing github repository for direct cloning.\nInstallation through pip package\npip install indicLP\n\nPrerequisite Libraries\nindicLP requires the following libraries for comfortable execution:\n\ntorchtext\nsnowballstemmer\nindic_transliteration\ngensim\npickle\ntarfile\n\n"},{"title":"Preprocessing","link":"https:\/\/aakash-ez.github.io\/indicLP-docs\/docs\/preprocessing\/","pubDate":"Mon, 01 Jan 0001 00:00:00 +0000","guid":"https:\/\/aakash-ez.github.io\/indicLP-docs\/docs\/preprocessing\/","description":"Getting Started\nPreprocessing Library contains all the necessary tools to clean the input text before passing it to a model.\nExample\nfrom indicLP.preprocessing import TextNormalizer, Embedding\nlanguage = "ta"\nnormalizerTamil = TextNormalizer(language)\nembedder = Embedding(lang=language)\n\nClasses Contained\nFollowing are the classes contained in preprocessing module:\n\nTextNormalizer: Class to perform tasks such as tokenizing, stopword removal and stemming.\nEmbedding: Class for performing word embedding and finding closely associated words."},{"title":"TextNormalizer","link":"https:\/\/aakash-ez.github.io\/indicLP-docs\/docs\/textnormalizer\/","pubDate":"Mon, 01 Jan 0001 00:00:00 +0000","guid":"https:\/\/aakash-ez.github.io\/indicLP-docs\/docs\/textnormalizer\/","description":"Getting Started\nTextNormalizer class provides functions like tokenizer, stop word removal and stemmer, to normalize the corpora before training the machine learning models.\nExample\n# coding:utf-8\nfrom indicLP.preprocessing import TextNormalizer, Embedding\nimport re\nlanguage = "hi"\nnormalizerHindi = TextNormalizer(language)\ntext = " \u0928\u092e\u0938\u094d\u0924\u0947! \u0906\u092a \u0915\u0948\u0938\u0947 \u0939\u0948\u0902? \u092e\u0941\u091d\u0947 \u0906\u092a\u0915\u0940 \u092c\u0939\u0941\u0924 \u092f\u093e\u0926 \u0906\u092f\u0940\u0964"\ntext = re.split('[\u0964;?\\.!]',text)\ntext = list(filter(None, text))\nprint(text)\n# Output is [' \u0928\u092e\u0938\u094d\u0924\u0947', ' \u0906\u092a \u0915\u0948\u0938\u0947 \u0939\u0948\u0902', ' \u092e\u0941\u091d\u0947 \u0906\u092a\u0915\u0940 \u092c\u0939\u0941\u0924 \u092f\u093e\u0926 \u0906\u092f\u0940']\nout = normalizerHindi."},{"title":"Tokenizer - Method","link":"https:\/\/aakash-ez.github.io\/indicLP-docs\/docs\/tokenizer\/","pubDate":"Mon, 01 Jan 0001 00:00:00 +0000","guid":"https:\/\/aakash-ez.github.io\/indicLP-docs\/docs\/tokenizer\/","description":"Getting Started\nTokenizer function is one of the most necessary method when it comes to preprocessing in NLP.\nExample\n# coding:utf-8\nfrom indicLP.preprocessing import TextNormalizer\nimport re\nlanguage = "hi"\nnormalizerHindi = TextNormalizer(language)\ntext = " \u0928\u092e\u0938\u094d\u0924\u0947! \u0906\u092a \u0915\u0948\u0938\u0947 \u0939\u0948\u0902? \u092e\u0941\u091d\u0947 \u0906\u092a\u0915\u0940 \u092c\u0939\u0941\u0924 \u092f\u093e\u0926 \u0906\u092f\u0940\u0964"\ntext = re.split('[\u0964;?\\.!]',text)\ntext = list(filter(None, text))\nprint(text)\n# Output is [' \u0928\u092e\u0938\u094d\u0924\u0947', ' \u0906\u092a \u0915\u0948\u0938\u0947 \u0939\u0948\u0902', ' \u092e\u0941\u091d\u0947 \u0906\u092a\u0915\u0940 \u092c\u0939\u0941\u0924 \u092f\u093e\u0926 \u0906\u092f\u0940']\nout = normalizerHindi."},{"title":"Stem - Method","link":"https:\/\/aakash-ez.github.io\/indicLP-docs\/docs\/stem\/","pubDate":"Mon, 01 Jan 0001 00:00:00 +0000","guid":"https:\/\/aakash-ez.github.io\/indicLP-docs\/docs\/stem\/","description":"Getting Started\nTokenizer function is one of the most necessary method when it comes to preprocessing in NLP.\nExample\n# coding:utf-8\nfrom indicLP.preprocessing import TextNormalizer\nlanguage = "hi"\nnormalizerHindi = TextNormalizer(language)\ntext = "\u0916\u093f\u0932\u093e\u0921\u093c\u093f\u092f\u094b\u0902"\nprint(normalizerHindi.stem(text))\n# Output is "\u0916\u093f\u0932\u093e\u0921\u093c"\nlanguage = "ta"\nnormalizerTamil = TextNormalizer(language)\ntext = "\u0baa\u0bc6\u0ba3\u0bcd\u0b95\u0bb3\u0bcd"\nprint(normalizerTamil.stem(text))\n# Output is "\u0baa\u0bc6\u0ba3\u0bcd"\n\nInput Arguments\nFollowing are the input arguments to be provided while using stem method:\n\nword: A string word that has to be stemmed."},{"title":"Remove Stop Words - Method","link":"https:\/\/aakash-ez.github.io\/indicLP-docs\/docs\/remove-stop-words\/","pubDate":"Mon, 01 Jan 0001 00:00:00 +0000","guid":"https:\/\/aakash-ez.github.io\/indicLP-docs\/docs\/remove-stop-words\/","description":"Getting Started\nremove_stop_words method in Embedding class can be used to perform stopword removal in input list.\nExample\n# coding:utf-8\nfrom indicLP.preprocessing import TextNormalizer\nimport re\nlanguage = "hi"\nnormalizerHindi = TextNormalizer(language)\ntext = " \u0928\u092e\u0938\u094d\u0924\u0947! \u0906\u092a \u0915\u0948\u0938\u0947 \u0939\u0948\u0902? \u092e\u0941\u091d\u0947 \u0906\u092a\u0915\u0940 \u092c\u0939\u0941\u0924 \u092f\u093e\u0926 \u0906\u092f\u0940\u0964"\ntext = re.split('[\u0964;?\\.!]',text)\ntext = list(filter(None, text))\nprint(text)\n# Output is [' \u0928\u092e\u0938\u094d\u0924\u0947', ' \u0906\u092a \u0915\u0948\u0938\u0947 \u0939\u0948\u0902', ' \u092e\u0941\u091d\u0947 \u0906\u092a\u0915\u0940 \u092c\u0939\u0941\u0924 \u092f\u093e\u0926 \u0906\u092f\u0940']\nout = normalizerHindi."},{"title":"Embedding","link":"https:\/\/aakash-ez.github.io\/indicLP-docs\/docs\/embedding\/","pubDate":"Mon, 01 Jan 0001 00:00:00 +0000","guid":"https:\/\/aakash-ez.github.io\/indicLP-docs\/docs\/embedding\/","description":"Getting Started\nEmbedding is a necessary functionality in modern NLP as most models and algorithms require real number data and not string such as words and sentences. The Embedding library is built on top of Gensim Library.\nExample\n# coding:utf-8\nfrom indicLP.preprocessing import Embedding\nlanguage = "ta"\nword = "\u0baa\u0bc1\u0ba4\u0bcd\u0ba4\u0b95\u0bae\u0bcd"\nembedder = Embedding(lang=language)\nprint(embedder.get_vector(word).shape)\n#Output is (300,)\nprint(embedder.get_closest(word))\n# Output is [('\u0ba8\u0bbe\u0bb5\u0bb2\u0bcd', 0.8771955370903015), ('\u0b95\u0b9f\u0bcd\u0b9f\u0bc1\u0bb0\u0bcd', 0.8479164242744446), ('\u0b95\u0b9f\u0bcd\u0b9f\u0bc1\u0bb0\u0bc8', 0.8334720134735107), ('\u0baa\u0ba4\u0bbf\u0baa\u0bcd\u0baa\u0bbf', 0.8099671006202698), ('\u0baa\u0ba4\u0bcd\u0ba4\u0bbf\u0bb0\u0bbf\u0b95\u0bcd\u0b95\u0bc8', 0."},{"title":"Get Vector - Method","link":"https:\/\/aakash-ez.github.io\/indicLP-docs\/docs\/get-vector\/","pubDate":"Mon, 01 Jan 0001 00:00:00 +0000","guid":"https:\/\/aakash-ez.github.io\/indicLP-docs\/docs\/get-vector\/","description":"Getting Started\nGet Vector function provides the embedding vector for a given input argument.\nExample\n# coding:utf-8\nfrom indicLP.preproccessing import Embedding\nlanguage = "ta"\nword = "\u0baa\u0bc1\u0ba4\u0bcd\u0ba4\u0b95\u0bae\u0bcd"\nembedder = Embedding(lang=language)\nprint(embedder.get_vector(word))\n#Output is:\n# [-0.38245687 -0.43007085 0.75659156 -0.029661 0.42061085 -0.78045815\n# -0.28127694 -1.0599478 -0.1264891 -0.42795 0.39396504 -0.79804516\n# 0.03756442 0.24138477 -0.1513924 0.31122482 0.46426052 -1.2080644\n# -0.40227187 0.7438162 -0.4533786 0.40707597 -0.25478044 0.19656514\n# 0.5389387 0.2874967 0.43010154 0.13535404 -0.4051279 0.03866816 ...\n\nInput Arguments\nFollowing are the input arguments to be provided while using get_vector method:"},{"title":"Get Closest - Method","link":"https:\/\/aakash-ez.github.io\/indicLP-docs\/docs\/get-closest\/","pubDate":"Mon, 01 Jan 0001 00:00:00 +0000","guid":"https:\/\/aakash-ez.github.io\/indicLP-docs\/docs\/get-closest\/","description":"Getting Started\nGet Closest function provides the N closely associated words for the given input string.\nExample\n# coding:utf-8\nfrom indicLP.preproccessing import Embedding\nlanguage = "ta"\nword = "\u0baa\u0bb3\u0bcd\u0bb3\u0bbf"\nembedder = Embedding(lang=language)\nfor i in closest:\n(word, score) = i\nprint(word,round(score,2))\n# \u0baa\u0bbe\u0b9f\u0b9a\u0bbe\u0bb2\u0bc8 0.87\n# \u0b89\u0baf\u0bb0\u0bcd\u0ba8\u0bbf\u0bb2\u0bcd 0.86\n# \u0b95\u0bb2\u0bcd\u0bb2\u0bc2\u0bb0\u0bbf 0.85\n# \u0baa\u0bbe\u0b9f\u0b9a\u0bbe\u0bb2\u0bcd 0.84\n# \u0b89\u0baf\u0bb0\u0bcd\u0ba8\u0bbf\u0bb2\u0bc8\u0baa\u0bcd\u0baa\u0bb3\u0bcd\u0bb3\u0bbf 0.81\n# \u0bae\u0bc7\u0bb2\u0bcd\u0ba8\u0bbf\u0bb2\u0bcd 0.81\n# \u0baa\u0bb3\u0bcd\u0bb3\u0bbf\u0b95\u0bcd\u0b95\u0bc2\u0b9f\u0bae\u0bcd 0.79\n# \u0b95\u0bb2\u0bcd 0.78\n# \u0bae\u0bbe\u0ba3 0.77\n# \u0baa\u0b9f\u0bbf\u0baa\u0bcd\u0baa\u0bc1 0."},{"title":"Convert - Method","link":"https:\/\/aakash-ez.github.io\/indicLP-docs\/docs\/convert\/","pubDate":"Mon, 01 Jan 0001 00:00:00 +0000","guid":"https:\/\/aakash-ez.github.io\/indicLP-docs\/docs\/convert\/","description":"Getting Started\nConvert function performs the task of transliteration, from one script to another. This is built on top of indic-transliteration library.\nExample\n# coding:utf-8\nfrom indicLP.transliterate import Transliterate\nhi2ta = Transliterate("hi","ta")\nprint(hi2ta.convert("\u0906\u0926\u092e\u0940"))\n# Output is \u0b86\u0ba4\u0bae\u0bc0\nen2hi = Transliterate("en","hi")\nprint(en2hi.convert("aadamii"))\n# Output is \u0906\u0926\u092e\u0940\n\nInput Arguments\nFollowing are the input arguments to be provided while using convert method:\n\ndata: A string representing the word which has to be transliterated."},{"title":"Revert - Method","link":"https:\/\/aakash-ez.github.io\/indicLP-docs\/docs\/revert\/","pubDate":"Mon, 01 Jan 0001 00:00:00 +0000","guid":"https:\/\/aakash-ez.github.io\/indicLP-docs\/docs\/revert\/","description":"Getting Started\nRevert function performs the task of transliteration, back to the original script. This is built on top of indic-transliteration library.\nExample\n# coding:utf-8\nfrom indicLP.transliterate import Transliterate\nhi2ta = Transliterate("hi","ta")\nen2hi = Transliterate("en","hi")\ntext = hi2ta.convert("\u0906\u0926\u092e\u0940")\nprint(text,hi2ta.revert(text))\n# Output is \u0b86\u0ba4\u0bae\u0bc0 \u0906\u0927\u092e\u0940\ntext = en2hi.convert("AdamI")\nprint(text,en2hi.revert(text))\n# Output is \u0906\u0926\u092e\u0940 AdamI\n\nInput Arguments\nFollowing are the input arguments to be provided while using revert method:\n\ndata: A string representing the word which has to be reverted back."},{"title":"BahdanauAttention","link":"https:\/\/aakash-ez.github.io\/indicLP-docs\/docs\/bahdanau-attention\/","pubDate":"Mon, 01 Jan 0001 00:00:00 +0000","guid":"https:\/\/aakash-ez.github.io\/indicLP-docs\/docs\/bahdanau-attention\/","description":"Getting Started\nBahdanau or Additive Attention layer for pytorch has been implemented, as it often plays a crucial part in NLP.\nExample\nclass BahdanauAttention(torch.nn.Module):\ndef __init__(self, encoder_dim, decoder_dim):\nsuper().__init__()\nself.encoder_dim = encoder_dim\nself.decoder_dim = decoder_dim\nself.V = torch.nn.Parameter(torch.rand(self.decoder_dim))\nself.W1 = torch.nn.Linear(self.decoder_dim, self.decoder_dim)\nself.W2 = torch.nn.Linear(self.encoder_dim, self.decoder_dim)\ndef forward(self, query, values):\nweights = self._get_weights(query,values)\nweights = torch.nn.functional.softmax(weights, dim = 0)\nreturn weights @ values\ndef _get_weights(self, query, values):\nquery = query."},{"title":"LuongAttention","link":"https:\/\/aakash-ez.github.io\/indicLP-docs\/docs\/luong-attention\/","pubDate":"Mon, 01 Jan 0001 00:00:00 +0000","guid":"https:\/\/aakash-ez.github.io\/indicLP-docs\/docs\/luong-attention\/","description":"Getting Started\nLuong or Multiplicative Attention layer for pytorch has been implemented, as it often plays a crucial part in NLP.\nExample\nclass LuongAttention(torch.nn.Module):\ndef __init__(self, encoder_dim: int, decoder_dim: int):\nsuper().__init__(encoder_dim, decoder_dim)\nself.W = torch.nn.Parameter(torch.FloatTensor(self.decoder_dim, self.encoder_dim).uniform_(-0.1, 0.1))\ndef forward(self, query, values):\nweights = self._get_weights(query,values)\nweights = torch.nn.functional.softmax(weights, dim = 0)\nreturn weights @ values\ndef _get_weights(self, query, values):\nweights = query @ self.W @ values.T\nreturn weights\/np.sqrt(self.decoder_dim)\n\nReference Materials\nFollowing are some reference materials for Preprocessing module"},{"title":"Load Dataset - Method","link":"https:\/\/aakash-ez.github.io\/indicLP-docs\/docs\/load-dataset\/","pubDate":"Mon, 01 Jan 0001 00:00:00 +0000","guid":"https:\/\/aakash-ez.github.io\/indicLP-docs\/docs\/load-dataset\/","description":"Getting Started\nDownload and load corpora and datasets directly in your program using\nExample\nfrom indicLP.datasets import Dataset\nfrom indicLP.utils import SentenceIterator\ndt = Dataset()\ndata = dt.load_dataset("ponniyin-selvan",True)\nsentenceIterator = SentenceIterator(data[0],"\\\\n")\ncount = 0\nfor i in sentenceIterator:\nprint(i)\ncount += 1\nif count == 5:\nbreak\n\nInput Arguments\nFollowing are the input arguments to be provided while using get_vector method:\n\ndataset_name (string): Name of the dataset to be loaded."},{"title":"Word Embedding","link":"https:\/\/aakash-ez.github.io\/indicLP-docs\/blog\/word-embedding\/","pubDate":"Fri, 29 Oct 2021 00:00:00 +0100","guid":"https:\/\/aakash-ez.github.io\/indicLP-docs\/blog\/word-embedding\/","description":"Getting Started Word Embedding has become one of the most crucial aspect of NLP tasks, as it helps in representing the meaning of words in a way computer can understand. Let us first consider the definition of Word Embedding, provided by Stanford’s RCpedia:\n Word Embeddings are a method to translate a string of text into an N-dimensional vector of real numbers. Many computational methods are not capable of accepting text as input."},{"title":"Transliteration in indicLP","link":"https:\/\/aakash-ez.github.io\/indicLP-docs\/blog\/transliteration\/","pubDate":"Fri, 29 Oct 2021 00:01:00 +0100","guid":"https:\/\/aakash-ez.github.io\/indicLP-docs\/blog\/transliteration\/","description":"Getting Started Transliteration is a topic that’s not often brought up in NLP tasks as more often than not we are dealing with corpus that contain latin text, thus making it standard. However this is not the case when deal with Indic Languages as more often than not we can have text from various scripts in our corpus. These could english translation of some words, quotes from other languages etc."}]}}