CT107-3-3-TXSA - Text Analytics and Sentiment Analysis Text Normalisation
Lab 4: Text Normalization: Stemming and Lemmatization
This session covers:
Different types of Stemmers & stemming Porter | Lancaster | SnowBall
Different types of Lemmatizer & lemmatizing WordNet lemmatizer | textblob lemmatizer
Lemmatization with & without POS tags
Learning Outcome:
Apply text mining and natural language processing methodologies to textual data.
QUICK REVIEW
Mapping from foxes to fox is called stemming. Morphological STEMMING parsing or stemming applies to many
affixes other than plurals; for example we might need to take any English verb form ending in -ing (going, talking,
congratulating) and parse it into its verbal stem plus the -ing morpheme.
The Porter algorithm is a simple and efficient way to do stemming, stripping off affixes. It is not as accurate as a
transducer model that includes a lexicon, but may be preferable for applications like information retrieval in which
exact morphological structure is not needed.
PRACTICES
4.1. Stemmers
Use the simple following code to implement porter stemmer.
from nltk.stem import PorterStemmer
from nltk.tokenize import word_tokenize
ps = PorterStemmer()
example_words1 = ["python", "pythoner", "pythoning", "pythoned", "pythonly"]
example_words2 = ["List", "listed", "lists", "listing", "listings"]
for w in example_words1:
print(ps.stem(w))
for w in example_words2:
print(ps.stem(w))
You can use this algorithm combining with a tokenization code in order to stem the words in a sentence, as below:
from nltk.stem import PorterStemmer
from nltk.tokenize import word_tokenize
ps = PorterStemmer()
new_text = """It is very important to be pythonly while you are pythoning
with python. All pythoners have pythoned poorly at least once."""
words = word_tokenize(new_text)
print([ps.stem(w) for w in words])
Level 3 Asia Pacific University (APU) Page 1 of 5
CT107-3-3-TXSA - Text Analytics and Sentiment Analysis Text Normalisation
NLTK includes several off-the-shelf stemmers, and if you ever need a stemmer you should use one of these in
preference to crafting your own using regular expressions, since these handle a wide range of irregular cases. The Porter
and Lancaster stemmers follow their own rules for stripping affixes. Observe that the Porter stemmer correctly handles
the word lying (mapping it to lie), while the Lancaster stemmer does not.
from nltk.stem import PorterStemmer, LancasterStemmer
from nltk.tokenize import word_tokenize
raw = """DENNIS: Listen, strange women lying in ponds distributing swords
is no basis for a system of government. Supreme executive power derives from
a mandate from the masses, not from some farcical aquatic ceremony."""
tokens = word_tokenize(raw)
porter = PorterStemmer()
lancaster = LancasterStemmer()
print([porter.stem(t) for t in tokens])
print("\n").
print([lancaster.stem(t) for t in tokens])
Compare and discuss the results of two stemmers (Porter and Lancaster), if you observe any difference.
4.2. Lemmatization using WordNet lemmatizer
The WordNet lemmatizer only removes affixes if the resulting word is in its dictionary. This additional checking
process makes the lemmatizer slower than the above stemmers.
from nltk.stem import WordNetLemmatizer, PorterStemmer
lemmatizer = WordNetLemmatizer()
print("rocks :", lemmatizer.lemmatize("rocks"))
print("\nproduced :", lemmatizer.lemmatize("produced", pos ="v"))
ps = PorterStemmer()
print("\nStem of the word produced :", ps.stem("produced"))
print("\nbetter :", lemmatizer.lemmatize("better", pos ="a"))
print("\nwomen :", lemmatizer.lemmatize("women", pos ="n"))
Notice that it doesn't handle lying, but it converts women to woman.
from nltk.stem import PorterStemmer
from nltk.tokenize import word_tokenize
raw = """DENNIS: Listen, strange women lying in ponds distributing swords
is no basis for a system of government. Supreme executive power derives from
a mandate from the masses, not from some farcical aquatic ceremony."""
tokens = word_tokenize(raw)
wnl = nltk.WordNetLemmatizer()
print([wnl.lemmatize(t) for t in tokens])
print()
Level 3 Asia Pacific University (APU) Page 2 of 5
CT107-3-3-TXSA - Text Analytics and Sentiment Analysis Text Normalisation
for t in tokens:
print ("{0:20}{1:20}".format(t, wnl.lemmatize(t, pos="v")))
print()
example_words = ["List", "listed", "lists", "listing", "listings"]
print([wnl.lemmatize(w) for w in example_words])
print()
for words in example_words:
print ("{0:20}{1:20}".format(words, wnl.lemmatize(words, pos="v")))
The WordNet lemmatizer is a good choice if you want to compile the vocabulary of some texts and want a list of valid
lemmas (or lexicon headwords). The results would result lemma not as 100% accurate according to the lemmas found in
the dictionary.
However, to have the exact lemma as per the dictionary, POS tagging better be included in the code.
for t in tokens:
print ("{0:20}{1:20}".format(t, wnl.lemmatize(t, pos="v")))
4.3. Lemmatization using TextBlob
Words can be lemmatized by calling the lemmatize method via the TextBlob objects
from textblob import TextBlob
sentence = TextBlob('DENNIS: Listen, strange women lying in ponds distributing
swords is no basis for a system of government. Supreme executive power derives
from a mandate from the masses, not from some farcical aquatic ceremony.')
tokens = sentence.words
print(tokens)
print
tokens.lemmatize()
for t in tokens:
print ("{0:20}{1:20}".format(t, wnl.lemmatize(t, pos="v")))
4.4. Stemming & Lemmatization
Stemming and Lemmatization both generate the root form of the inflected words. The difference is that stem might not
be an actual word whereas, lemma is an actual language word.
import nltk
from nltk.stem import PorterStemmer
from nltk.stem import LancasterStemmer
from nltk.tokenize import word_tokenize
from nltk.stem.snowball import SnowballStemmer
#file = open ("D:/APU/TXSA-CT107-3-3/TUTORIAL/sample01.txt")
#raw = file.read()
raw = """DENNIS: Listen, strange women lying in ponds distributing swords
is no basis for a system of government. Supreme executive power derives from
a mandate from the masses, not from some farcical aquatic ceremony."""
Level 3 Asia Pacific University (APU) Page 3 of 5
CT107-3-3-TXSA - Text Analytics and Sentiment Analysis Text Normalisation
words = raw.lower()
print(words)
print()
tokens = word_tokenize(words)
print("Tokens")
print(tokens)
print()
print("Lemmas")
wnl = nltk.WordNetLemmatizer()
print([wnl.lemmatize(t, pos = "v") for t in tokens])
print()
print("Porter Stemming")
ps = PorterStemmer()
print ([ps.stem(t) for t in tokens])
print()
print("Lancaster Stemming")
ls = LancasterStemmer()
print ([ls.stem(t) for t in tokens])
print()
print("Snowball Stemming")
sn = nltk.SnowballStemmer("english")
print([sn.stem(t) for t in tokens])
NOTE:
Stemming and Lemmatization both generate the root form of the inflected words. The difference is that stem might not
be an actual word whereas, lemma is an actual language word. Whereas, in lemmatization, you used WordNet corpus
and a corpus for stop words as well to produce lemma which makes it slower than stemming.
4.5. Stemmers --> Snowball Stemmer
import nltk
print(nltk.SnowballStemmer.languages)
print(len(nltk.SnowballStemmer.languages))
print()
text = "This is achieved in practice during stemming, a text preprocessing
operation."
tokens = nltk.tokenize.word_tokenize(text)
print()
stemmer = nltk.SnowballStemmer('english')
print([stemmer.stem(t) for t in tokens])
print()
text2 = "Ceci est réalisé en pratique lors du stemming, une opération de
prétraitement de texte."
tokens2 = nltk.tokenize.word_tokenize(text2)
print()
stemmer = nltk.SnowballStemmer('french')
print([stemmer.stem(t) for t in tokens2])
4.6. Snowball Stemmer --> for other space delimited languages
from textblob import TextBlob
import nltk
en_blob = TextBlob(u'This is achieved in practice during stemming, a text
preprocessing operation.')
Level 3 Asia Pacific University (APU) Page 4 of 5
CT107-3-3-TXSA - Text Analytics and Sentiment Analysis Text Normalisation
print(en_blob.detect_language())
fr_blob = en_blob.translate(from_lang="en", to='fr')
print(fr_blob)
tokens = fr_blob.words
print(tokens)
print()
stemmer = nltk.SnowballStemmer('french')
print([stemmer.stem(t) for t in tokens])
References:
1. Regular Expressions: The Complete Tutorial, by Jan Goyvaerts, 2007.
2. Speech and Language Processing, by Dan Jurafsky and James H. Martin. Prentice Hall Series in Artificial
Intelligence, 2008.
3. Natural Language Processing with Python, by Steven Bird, Ewan Klein and Edward Loper, 2014.
4. Lemmatization approaches with examples in Python (https://www.machinelearningplus.com/nlp/lemmatization-
examples-python/)
Revision Quiz:
https://quizlet.com/512734559/test?
answerTermSides=2&promptTermSides=6&questionCount=7&questionTypes=14&showImages=true
Take-home task:
Perform the lemmatization including the POS tags such as ADJECTIVE, NOUN, VERB, and ADVERB. Write the
suitable python code to get the proper output.
Level 3 Asia Pacific University (APU) Page 5 of 5
