Practicle 7
DSBDA
1. sent_tokenize function from the nltk.tokenize module
- This function helps in tokenizing text based on sentences rather than words,
which can be a key step in many NLP pipelines.
2. word_tokenize function from the nltk.tokenize
- This function is very useful in text processing when you need to handle words
separately from punctuation for further analysis or tasks.
3. FreqDist function from the nltk.probability module
- You can use it to identify the most frequent words, which can help in tasks like
text summarization, keyword extraction, or language modeling.
4. fdist.plot()
- The `fdist.plot()` function is used to visually represent the frequency
distribution of words in a text, helping to identify the most common words and
their occurrence.
5. stopwords data from the NLTK library
- The `stopwords` corpus is used to retrieve a list of common words (like "the",
"is", "in") that are typically removed from text data during preprocessing to focus
on more meaningful words.
6. from nltk.tokenize import word_tokenize
- This code tokenizes a sentence into words and removes common stopwords to produce
a filtered list of meaningful words for further text analysis.
7. imports the PorterStemmer and word_tokenize functions from NLTK
- This code imports the PorterStemmer and word_tokenize functions from NLTK to
tokenize text into words and apply stemming, which reduces words to their root
forms (e.g., "running" becomes "run"), helping in text analysis by standardizing
different word variations.
8. WordNet and OMW-1.4 corpora from NLTK, and then imports the WordNetLemmatizer
and the PorterStemmer
- This code downloads the WordNet and OMW-1.4 corpora from NLTK, and then imports
the WordNetLemmatizer for lemmatization (which reduces words to their base form
using dictionary definitions) and the PorterStemmer for stemming (which reduces
words to their root form by stripping suffixes), enabling both techniques for word
normalization in text processing.
9. averaged_perceptron_tagger model from NLTK and then uses nltk.pos_tag()
- This code downloads the averaged_perceptron_tagger model from NLTK and then uses
nltk.pos_tag() to tag each token in the `tokens` list with its corresponding part
of speech (POS), such as noun, verb, or adjective, helping to understand the
grammatical structure of the text.
10. TfidfVectorizer from sklearn.feature_extraction.text
- This code imports the TfidfVectorizer from sklearn.feature_extraction.text, which
is used to convert a collection of text documents into a matrix of TF-IDF (Term
Frequency-Inverse Document Frequency) features, helping to quantify the importance
of words in a document relative to a corpus for tasks like text classification or
clustering.
11. get_feature_names_out() method of the TfidfVectorizer
- This code uses the get_feature_names_out() method of the TfidfVectorizer to
retrieve a list of all the unique words (features) extracted from the input text
corpus, which are then used to represent the documents in the feature matrix. This
helps in understanding which words the model considers when analyzing the text.
