Normalizing Textual Data with Python
In this article, we will learn How to Normalizing Textual Data with
Python. Let’s discuss some concepts :
Textual data ask systematically collected material consisting
of written, printed, or electronically published words, typically
either purposefully written or transcribed from speech.
Text normalization is that the method of transforming text
into one canonical form that it’d not have had before.
Normalizing text before storing or processing it allows for
separation of concerns since the input is sure to be consistent
before operations are performed thereon. Text normalization
requires being conscious of what sort of text is to be
normalized and the way it’s to be processed afterwards; there’s
no all-purpose normalization procedure.
Steps Required
Here, we will discuss some basic steps need for Text
normalization.
Input text String,
Convert all letters of the string to one case(either lower or
upper case),
If numbers are essential to convert to words else remove all
numbers,
Remove punctuations, other formalities of grammar,
Remove white spaces,
Remove stop words,
And any other computations.
We are doing Text normalization with above-mentioned steps,
every step can be done in some ways. So we will discuss each
and everything in this whole process.
Text String
# input string
string = " Python 3.0, released in 2008, was a major revision of the
language that is not completely backward compatible and much Python 2 code does
not run unmodified on Python 3. With Python 2's end-of-life, only Python
3.6.x[30] and later are supported, with older versions still supporting e.g.
Windows 7 (and old installers not restricted to 64-bit Windows)."
print(string)
Python 3.0, released in 2008, was a major revision of
the language that is not completely backward compatible and
much Python 2 code does not run unmodified on Python 3. With
Python 2's end-of-life, only Python 3.6.x[30] and later are
supported, with older versions still supporting e.g. Windows
7 (and old installers not restricted to 64-bit Windows).
Output:
” Python 3.0, released in 2008, was a major revision of the
language that is not completely backward compatible and much
Python 2 code does not run unmodified on Python 3. With Python
2’s end-of-life, only Python 3.6.x[30] and later are supported, with
older versions still supporting e.g. Windows 7 (and old installers
not restricted to 64-bit Windows).”
Case Conversion (Lower Case)
In Python, lower() is a built-in method used for string handling.
The lower() methods returns the lowercased string from the given
string. It converts all uppercase characters to lowercase. If no
uppercase characters exist, it returns the original string.
# input string
string = " Python 3.0, released in 2008, was a major revision of the
language that is not completely backward compatible and much Python 2 code does
not run unmodified on Python 3. With Python 2's end-of-life, only Python
3.6.x[30] and later are supported, with older versions still supporting e.g.
Windows 7 (and old installers not restricted to 64-bit Windows)."
# convert to lower case
lower_string = string.lower()
print(lower_string)
Output:
” python 3.0, released in 2008, was a major revision of the
language that is not completely backward compatible and much
python 2 code does not run unmodified on python 3. with python
2’s end-of-life, only python 3.6.x[30] and later are supported, with
older versions still supporting e.g. windows 7 (and old installers
not restricted to 64-bit windows).”
Removing Numbers
Remove numbers if they’re not relevant to your analyses. Usually,
regular expressions are used to remove numbers.
# import regex
import re
# input string
string = " Python 3.0, released in 2008, was a major revision of the
language that is not completely backward compatible and much Python 2 code does
not run unmodified on Python 3. With Python 2's end-of-life, only Python
3.6.x[30] and later are supported, with older versions still supporting e.g.
Windows 7 (and old installers not restricted to 64-bit Windows)."
# convert to lower case
lower_string = string.lower()
# remove numbers
no_number_string = re.sub(r'\d+','',lower_string)
print(no_number_string)
Output:
” python ., released in , was a major revision of the language
that is not completely backward compatible and much python
code does not run unmodified on python . with python ‘s end-of-
life, only python ..x[] and later are supported, with older versions
still supporting e.g. windows (and old installers not restricted to -
bit windows).”
Removing punctuation
The part of replacing with punctuation can also be performed
using regex. In this, we replace all punctuation by empty string
using certain regex.
# import regex
import re
# input string
string = " Python 3.0, released in 2008, was a major revision of the
language that is not completely backward compatible and much Python 2 code does
not run unmodified on Python 3. With Python 2's end-of-life, only Python
3.6.x[30] and later are supported, with older versions still supporting e.g.
Windows 7 (and old installers not restricted to 64-bit Windows)."
# convert to lower case
lower_string = string.lower()
# remove numbers
no_number_string = re.sub(r'\d+','',lower_string)
# remove all punctuation except words and space
no_punc_string = re.sub(r'[^\w\s]','', no_number_string)
print(no_punc_string)
Output:
‘ python released in was a major revision of the language
that is not completely backward compatible and much python
code does not run unmodified on python with python s endoflife
only python x and later are supported with older versions still
supporting eg windows and old installers not restricted to bit
windows’
Removing White space
The strip() function is an inbuilt function in Python programming
language that returns a copy of the string with both leading and
trailing characters removed (based on the string argument
passed).
# import regex
import re
# input string
string = " Python 3.0, released in 2008, was a major revision of the
language that is not completely backward compatible and much Python 2 code does
not run unmodified on Python 3. With Python 2's end-of-life, only Python
3.6.x[30] and later are supported, with older versions still supporting e.g.
Windows 7 (and old installers not restricted to 64-bit Windows)."
# convert to lower case
lower_string = string.lower()
# remove numbers
no_number_string = re.sub(r'\d+','',lower_string)
# remove all punctuation except words and space
no_punc_string = re.sub(r'[^\w\s]','', no_number_string)
# remove white spaces
no_wspace_string = no_punc_string.strip()
print(no_wspace_string)
Output:
‘python released in was a major revision of the language that is
not completely backward compatible and much python code
does not run unmodified on python with python s endoflife only
python x and later are supported with older versions still
supporting eg windows and old installers not restricted to bit
windows’
Removing Stop Words
Stop words” are the foremost common words during a language
like “the”, “a”, “on”, “is”, “all”. These words don’t carry important
meaning and are usually faraway from texts. It is possible to get
rid of stop words using tongue Toolkit (NLTK), a set of libraries
and programs for symbolic and statistical tongue processing.
Python3
# download stopwords
import nltk
nltk.download('stopwords')
# import nltk for stopwords
from nltk.corpus import stopwords
stop_words = set(stopwords.words('english'))
print(stop_words)
# assign string
no_wspace_string='python released in was a major revision of the language that
is not completely backward compatible and much python code does not run
unmodified on python with python s endoflife only python x and later are
supported with older versions still supporting eg windows and old installers not
restricted to bit windows'
# convert string to list of words
lst_string = [no_wspace_string][0].split()
print(lst_string)
# remove stopwords
no_stpwords_string=""
for i in lst_string:
if not i in stop_words:
no_stpwords_string += i+' '
# removing last space
no_stpwords_string = no_stpwords_string[:-1]
print(no_stpwords_string)
Output:
In this, we can normalize the textual data using Python. Below is
the complete python program:
Python3
# import regex
import re
# download stopwords
import nltk
nltk.download('stopwords')
# import nltk for stopwords
from nltk.corpus import stopwords
stop_words = set(stopwords.words('english'))
# input string
string = " Python 3.0, released in 2008, was a major revision of the
language that is not completely backward compatible and much Python 2 code does
not run unmodified on Python 3. With Python 2's end-of-life, only Python
3.6.x[30] and later are supported, with older versions still supporting e.g.
Windows 7 (and old installers not restricted to 64-bit Windows)."
# convert to lower case
lower_string = string.lower()
# remove numbers
no_number_string = re.sub(r'\d+','',lower_string)
# remove all punctuation except words and space
no_punc_string = re.sub(r'[^\w\s]','', no_number_string)
# remove white spaces
no_wspace_string = no_punc_string.strip()
no_wspace_string
# convert string to list of words
lst_string = [no_wspace_string][0].split()
print(lst_string)
# remove stopwords
no_stpwords_string=""
for i in lst_string:
if not i in stop_words:
no_stpwords_string += i+' '
# removing last space
no_stpwords_string = no_stpwords_string[:-1]
# output
print(no_stpwords_string)
Output:
