{"id":93720,"date":"2019-05-09T09:29:54","date_gmt":"2019-05-09T13:29:54","guid":{"rendered":"https:\/\/www.kdnuggets.com\/?p=93720"},"modified":"2019-05-09T09:41:04","modified_gmt":"2019-05-09T13:41:04","slug":"a-complete-exploratory-data-analysis-and-visualization-for-text-data-combine-visualization-and-nlp-to-generate-insights","status":"publish","type":"post","link":"https:\/\/www.kdnuggets.com\/2019\/05\/complete-exploratory-data-analysis-visualization-text-data.html","title":{"rendered":"A Complete Exploratory Data Analysis and Visualization for Text Data: Combine Visualization and NLP to Generate Insights"},"content":{"rendered":"<div align=\"right\"><img loading=\"lazy\" decoding=\"async\" src=\"\/images\/comment.gif\" width=\"16\" height=\"12\" alt=\"c\"> <a href=\"\/2019\/05\/complete-exploratory-data-analysis-visualization-text-data.html?page=2#comments\">comments<\/a><\/div>\n<p><b>By <a href=\"https:\/\/www.linkedin.com\/in\/susanli\/\" target=\"_blank\" rel=\"noopener noreferrer\">Susan Li<\/a>, Sr. Data Scientist<\/b><\/p><div class=\"kdnug-after-first-paragraph kdnug-entity-placement\" id=\"kdnug-838142930\"><div id=\"kdnug-3196850881\"><a data-no-instant=\"1\" href=\"https:\/\/brightdata.com\/products\/web-scraper?pscd=get.brightdata.com&#038;ps_partner_key=MTgwODIzNDM4ZTJl&#038;ps_xid=ddqsY59sX6XPwj&#038;gsxid=ddqsY59sX6XPwj&#038;gspk=MTgwODIzNDM4ZTJl&#038;utm_source=affiliates&#038;utm_campaign=MTgwODIzNDM4ZTJl\" rel=\"noopener nofollow\" class=\"a2t-link\" target=\"_blank\"><p>\t\t\t\t<img decoding=\"async\" style=\"max-width: 100%; height: auto;\" src=\"https:\/\/www.kdnuggets.com\/wp-content\/uploads\/s1-brightdata-2512.jpg\" alt=\"BrightData WebScarping\" \/><br \/>\nThe most reliable Web Scraping API<\/p>\n<\/a><\/div><\/div>\n<div style=\"text-align:center\"><img decoding=\"async\" src=\"https:\/\/i.ibb.co\/Xjxfd2S\/text.jpg\" alt=\"figure-name\" width=\"99%\" \/><font size=\"-1\"><\/p>\n<div class=\"caption\">Photo credit: Pixabay<\/div>\n<p><\/font>\n<\/div>\n<p>Visually representing the content of a text document is one of the most important tasks in the field of <a href=\"https:\/\/en.wikipedia.org\/wiki\/Text_mining\" rel=\"noopener noreferrer\" target=\"_blank\">text mining<\/a>. As a data scientist or <a href=\"https:\/\/plot.ly\/python\/\" rel=\"noopener noreferrer\" target=\"_blank\">NLP<\/a> specialist, not only we explore the content of documents from different aspects and at different levels of details, but also we summarize a single document, show the words and topics, detect events, and create storylines.<\/p><div class=\"kdnug-in-content-1 kdnug-entity-placement\" style=\"text-align: center;padding-bottom: 180px;padding-top: 20px;\" id=\"kdnug-492257837\"><div id=\"kdnug-1852105998\"><a data-no-instant=\"1\" href=\"https:\/\/brightdata.com\/products\/web-scraper?pscd=get.brightdata.com&#038;ps_partner_key=MTgwODIzNDM4ZTJl&#038;ps_xid=ddqsY59sX6XPwj&#038;gsxid=ddqsY59sX6XPwj&#038;gspk=MTgwODIzNDM4ZTJl&#038;utm_source=affiliates&#038;utm_campaign=MTgwODIzNDM4ZTJl\" rel=\"noopener nofollow\" class=\"a2t-link\" target=\"_blank\"><p>\t\t\t\t<img decoding=\"async\" style=\"max-width: 100%; height: auto;\" src=\"https:\/\/www.kdnuggets.com\/wp-content\/uploads\/s1-brightdata-2512.jpg\" alt=\"BrightData WebScarping\" \/><br \/>\nThe most reliable Web Scraping API<\/p>\n<\/a><\/div><\/div>\n<p>However, there are some gaps between visualizing unstructured (text) data and structured data. For example, many text visualizations do not represent the text directly, they represent an output of a language model(word count, character length, word sequences, etc.).<\/p>\n<p>In this post, we will use <a href=\"https:\/\/www.kaggle.com\/nicapotato\/womens-ecommerce-clothing-reviews\" rel=\"noopener noreferrer\" target=\"_blank\">Womens Clothing E-Commerce Reviews data set<\/a>, and try to explore and visualize as much as we can, using <a href=\"https:\/\/plot.ly\/python\/\" rel=\"noopener noreferrer\" target=\"_blank\">Plotly\u2019s Python graphing library<\/a> and <a href=\"https:\/\/bokeh.pydata.org\/en\/latest\/\" rel=\"noopener noreferrer\" target=\"_blank\">Bokeh visualization library<\/a>. Not only we are going to explore text data, but also we will visualize numeric and categorical features. Let\u2019s get started!<\/p>\n<p>&nbsp;<\/p>\n<h3>The Data<\/h3>\n<p>&nbsp;<\/p>\n<div style=\"width:98%;border:1px solid #ccc;overflow:auto;padding-left:10px;padding-bottom:10px;padding-top:10px\">\n<pre>\r\ndf = pd.read_csv('Womens Clothing E-Commerce Reviews.csv')\r\n<\/pre>\n<\/div>\n<p><br class=\"blank\" \/><\/p>\n<div style=\"text-align:center\"><img decoding=\"async\" src=\"https:\/\/i.ibb.co\/FwzFh1d\/table1.png\" alt=\"figure-name\" width=\"50%\" \/><font size=\"-1\"><\/p>\n<div class=\"caption\">table 1<\/div>\n<p><\/font>\n<\/div>\n<p>After a brief inspection of the data, we found there are a series of data pre-processing we have to conduct.<\/p>\n<ul class=\"six_ul\">\n<li>Remove the \u201cTitle\u201d feature.\n<li>Remove the rows where \u201cReview Text\u201d were missing.\n<li>Clean \u201cReview Text\u201d column.\n<li>Using <a href=\"https:\/\/textblob.readthedocs.io\/en\/dev\/\" rel=\"noopener noreferrer\" target=\"_blank\"><strong>TextBlob<\/strong><\/a> to calculate sentiment polarity which lies in the range of [-1,1] where 1 means positive sentiment and -1 means a negative sentiment.\n<li>Create new feature for the length of the review.\n<li>Create new feature for the word count of the review.\n<\/ul>\n<div style=\"width:98%;border:1px solid #ccc;overflow:auto;padding-left:10px;padding-bottom:10px;padding-top:10px\">\n<pre>\r\ndf.drop('Unnamed: 0', axis=1, inplace=True)\r\ndf.drop('Title', axis=1, inplace=True)\r\ndf = df[~df['Review Text'].isnull()]\r\n\r\ndef preprocess(ReviewText):\r\n    ReviewText = ReviewText.str.replace(\"(<br \/>)\", \"\")\r\n    ReviewText = ReviewText.str.replace('(<a>).*(<\/a>)', '')\r\n    ReviewText = ReviewText.str.replace('(&amp;amp)', '')\r\n    ReviewText = ReviewText.str.replace('(&amp;gt)', '')\r\n    ReviewText = ReviewText.str.replace('(&amp;lt)', '')\r\n    ReviewText = ReviewText.str.replace('(\\xa0)', ' ')  \r\n    return ReviewText\r\ndf['Review Text'] = preprocess(df['Review Text'])\r\n\r\ndf['polarity'] = df['Review Text'].map(lambda text: TextBlob(text).sentiment.polarity)\r\ndf['review_len'] = df['Review Text'].astype(str).apply(len)\r\ndf['word_count'] = df['Review Text'].apply(lambda x: len(str(x).split()))\r\n<\/pre>\n<\/div>\n<div style=\"text-align:center\">text_preprocessing.py<\/div>\n<p><br class=\"blank\" \/><\/p>\n<p>To preview whether the sentiment polarity score works, we randomly select 5 reviews with the highest sentiment polarity score (1):<\/p>\n<div style=\"width:98%;border:1px solid #ccc;overflow:auto;padding-left:10px;padding-bottom:10px;padding-top:10px\">\n<pre>\r\nprint('5 random reviews with the highest positive sentiment polarity: \\n')\r\ncl = df.loc[df.polarity == 1, ['Review Text']].sample(5).values\r\nfor c in cl:\r\n    print(c[0])\r\n<\/pre>\n<\/div>\n<p><br class=\"blank\" \/><\/p>\n<div style=\"text-align:center\"><img decoding=\"async\" src=\"https:\/\/i.ibb.co\/F76pNRX\/figure1.png\" alt=\"figure-name\" width=\"99%\" \/><font size=\"-1\"><\/p>\n<div class=\"caption\">Figure 1<\/div>\n<p><\/font>\n<\/div>\n<p>Then randomly select 5 reviews with the most neutral sentiment polarity score (zero):<\/p>\n<div style=\"width:98%;border:1px solid #ccc;overflow:auto;padding-left:10px;padding-bottom:10px;padding-top:10px\">\n<pre>\r\nprint('5 random reviews with the most neutral sentiment(zero) polarity: \\n')\r\ncl = df.loc[df.polarity == 0, ['Review Text']].sample(5).values\r\nfor c in cl:\r\n    print(c[0])\r\n<\/pre>\n<\/div>\n<p><br class=\"blank\" \/><\/p>\n<div style=\"text-align:center\"><img decoding=\"async\" src=\"https:\/\/i.ibb.co\/njLNnnv\/figure2.png\" alt=\"figure-name\" width=\"99%\" \/><font size=\"-1\"><\/p>\n<div class=\"caption\">Figure 2<\/div>\n<p><\/font>\n<\/div>\n<p>There were only 2 reviews with the most negative sentiment polarity score:<\/p>\n<div style=\"width:98%;border:1px solid #ccc;overflow:auto;padding-left:10px;padding-bottom:10px;padding-top:10px\">\n<pre>\r\nprint('2 reviews with the most negative polarity: \\n')\r\ncl = df.loc[df.polarity == -0.97500000000000009, ['Review Text']].sample(2).values\r\nfor c in cl:\r\n    print(c[0])\r\n<\/pre>\n<\/div>\n<p><br class=\"blank\" \/><\/p>\n<div style=\"text-align:center\"><img decoding=\"async\" src=\"https:\/\/i.ibb.co\/DD2ZQ1X\/figure3.png\" alt=\"figure-name\" width=\"99%\" \/><font size=\"-1\"><\/p>\n<div>Figure 3<\/div>\n<p><\/font>\n<\/div>\n<p>It worked!<\/p>\n<p>&nbsp;<\/p>\n<h3>Univariate visualization with Plotly<\/h3>\n<p>&nbsp;<br \/>\nSingle-variable or univariate visualization is the simplest type of visualization which consists of observations on only a single characteristic or attribute. Univariate visualization includes histogram, bar plots and line charts.<\/p>\n<p><b><strong>The distribution of review sentiment polarity score<\/strong><\/b><\/p>\n<div style=\"width:98%;border:1px solid #ccc;overflow:auto;padding-left:10px;padding-bottom:10px;padding-top:10px\">\n<pre>\r\ndf['polarity'].iplot(\r\n    kind='hist',\r\n    bins=50,\r\n    xTitle='polarity',\r\n    linecolor='black',\r\n    yTitle='count',\r\n    title='Sentiment Polarity Distribution')\r\n<\/pre>\n<\/div>\n<p><br class=\"blank\" \/><\/p>\n<div style=\"text-align:center\">\n<a href=\"https:\/\/plot.ly\/~susanli2005\/91\/\" target=\"_blank\" title=\"Plot 91\" style=\"text-align: center\"><img decoding=\"async\" src=\"https:\/\/plot.ly\/~susanli2005\/91.png\" alt=\"Plot 91\" style=\"max-width: 100%;width: 600px\" width=\"600\" \/><\/a><font size=\"-1\"><\/p>\n<div>Figure 4<\/div>\n<p><\/font>\n<\/div>\n<p>Vast majority of the sentiment polarity scores are greater than zero, means most of them are pretty positive.<\/p>\n<p><b>The distribution of review ratings<\/b><\/p>\n<div style=\"width:98%;border:1px solid #ccc;overflow:auto;padding-left:10px;padding-bottom:10px;padding-top:10px\">\n<pre>\r\ndf['Rating'].iplot(\r\n    kind='hist',\r\n    xTitle='rating',\r\n    linecolor='black',\r\n    yTitle='count',\r\n    title='Review Rating Distribution')\r\n<\/pre>\n<\/div>\n<p><br class=\"blank\" \/><\/p>\n<div style=\"text-align:center\">\n    <a href=\"https:\/\/plot.ly\/~susanli2005\/93\/\" target=\"_blank\" title=\"Plot 93\" style=\"text-align: center\"><img decoding=\"async\" src=\"https:\/\/plot.ly\/~susanli2005\/93.png\" alt=\"Plot 93\" style=\"max-width: 100%;width: 600px\" width=\"600\" \/><\/a><br \/>\n    <font size=\"-1\"><\/p>\n<div>Figure 5<\/div>\n<p><\/font>\n<\/div>\n<p>The ratings are in align with the polarity score, that is, most of the ratings are pretty high at 4 or 5 ranges.<\/p>\n<p><b>The distribution of reviewers age<\/b><\/p>\n<div style=\"width:98%;border:1px solid #ccc;overflow:auto;padding-left:10px;padding-bottom:10px;padding-top:10px\">\n<pre>\r\ndf['Age'].iplot(\r\n    kind='hist',\r\n    bins=50,\r\n    xTitle='age',\r\n    linecolor='black',\r\n    yTitle='count',\r\n    title='Reviewers Age Distribution')\r\n<\/pre>\n<\/div>\n<p><br class=\"blank\" \/><\/p>\n<div style=\"text-align:center\">\n    <a href=\"https:\/\/plot.ly\/~susanli2005\/95\/\" target=\"_blank\" title=\"Plot 95\" style=\"text-align: center\"><img decoding=\"async\" src=\"https:\/\/plot.ly\/~susanli2005\/95.png\" alt=\"Plot 95\" style=\"max-width: 100%;width: 600px\" width=\"600\" \/><\/a><br \/>\n    <font size=\"-1\"><\/p>\n<div>Figure 6<\/div>\n<p><\/font>\n<\/div>\n<p>Most reviewers are in their 30s to 40s.<\/p>\n<p><b>The distribution review text lengths<\/b><\/p>\n<div style=\"width:98%;border:1px solid #ccc;overflow:auto;padding-left:10px;padding-bottom:10px;padding-top:10px\">\n<pre>\r\ndf['review_len'].iplot(\r\n    kind='hist',\r\n    bins=100,\r\n    xTitle='review length',\r\n    linecolor='black',\r\n    yTitle='count',\r\n    title='Review Text Length Distribution')\r\n<\/pre>\n<\/div>\n<p><br class=\"blank\" \/><\/p>\n<div style=\"text-align:center\">\n    <a href=\"https:\/\/plot.ly\/~susanli2005\/97\/\" target=\"_blank\" title=\"Plot 97\" style=\"text-align: center\"><img decoding=\"async\" src=\"https:\/\/plot.ly\/~susanli2005\/97.png\" alt=\"Plot 97\" style=\"max-width: 100%;width: 600px\" width=\"600\" \/><\/a><br \/>\n    <font size=\"-1\"><\/p>\n<div>Figure 7<\/div>\n<p><\/font>\n<\/div>\n<p><b>The distribution of review word count<\/b><\/p>\n<div style=\"width:98%;border:1px solid #ccc;overflow:auto;padding-left:10px;padding-bottom:10px;padding-top:10px\">\n<pre>\r\ndf['word_count'].iplot(\r\n    kind='hist',\r\n    bins=100,\r\n    xTitle='word count',\r\n    linecolor='black',\r\n    yTitle='count',\r\n    title='Review Text Word Count Distribution')\r\n<\/pre>\n<\/div>\n<p><br class=\"blank\" \/><\/p>\n<div style=\"text-align:center\">\n    <a href=\"https:\/\/plot.ly\/~susanli2005\/99\/\" target=\"_blank\" title=\"Plot 99\" style=\"text-align: center\"><img decoding=\"async\" src=\"https:\/\/plot.ly\/~susanli2005\/99.png\" alt=\"Plot 99\" style=\"max-width: 100%;width: 600px\" width=\"600\" \/><\/a><br \/>\n    <font size=\"-1\"><\/p>\n<div>Figure 8<\/div>\n<p><\/font>\n<\/div>\n<p>There were quite number of people like to leave long reviews.<br \/>\nFor categorical features, we simply use bar chart to present the frequency.<\/p>\n<p><b>The distribution of division<\/b><\/p>\n<div style=\"width:98%;border:1px solid #ccc;overflow:auto;padding-left:10px;padding-bottom:10px;padding-top:10px\">\n<pre>\r\ndf.groupby('Division Name').count()['Clothing ID'].iplot(kind='bar', yTitle='Count', linecolor='black', opacity=0.8, title='Bar chart of Division Name', xTitle='Division Name')\r\n<\/pre>\n<\/div>\n<p><br class=\"blank\" \/><\/p>\n<div style=\"text-align:center\">\n    <a href=\"https:\/\/plot.ly\/~susanli2005\/101\/\" target=\"_blank\" title=\"Plot 101\" style=\"text-align: center\"><img decoding=\"async\" src=\"https:\/\/plot.ly\/~susanli2005\/101.png\" alt=\"Plot 101\" style=\"max-width: 100%;width: 600px\" width=\"600\" \/><\/a><br \/>\n    <font size=\"-1\"><\/p>\n<div>Figure 9<\/div>\n<p><\/font>\n<\/div>\n<p>General division has the most number of reviews, and Initmates division has the least number of reviews.<\/p>\n<p><b>The distribution of department<\/b><\/p>\n<div style=\"width:98%;border:1px solid #ccc;overflow:auto;padding-left:10px;padding-bottom:10px;padding-top:10px\">\n<pre>\r\ndf.groupby('Department Name').count()['Clothing ID'].sort_values(ascending=False).iplot(kind='bar', yTitle='Count', linecolor='black', opacity=0.8, title='Bar chart of Department Name', xTitle='Department Name')\r\n<\/pre>\n<\/div>\n<p><br class=\"blank\" \/><\/p>\n<div style=\"text-align:center\">\n    <a href=\"https:\/\/plot.ly\/~susanli2005\/103\/\" target=\"_blank\" title=\"Plot 103\" style=\"text-align: center\"><img decoding=\"async\" src=\"https:\/\/plot.ly\/~susanli2005\/103.png\" alt=\"Plot 103\" style=\"max-width: 100%;width: 600px\" width=\"600\" \/><\/a><br \/>\n    <font size=\"-1\"><\/p>\n<div>Figure 10<\/div>\n<p><\/font>\n<\/div>\n<p>When comes to department, Tops department has the most reviews and Trend department has the least number of reviews.<\/p>\n<p><b>The distribution of class<\/b><\/p>\n<div style=\"width:98%;border:1px solid #ccc;overflow:auto;padding-left:10px;padding-bottom:10px;padding-top:10px\">\n<pre>\r\ndf.groupby('Class Name').count()['Clothing ID'].sort_values(ascending=False).iplot(kind='bar', yTitle='Count', linecolor='black', opacity=0.8, title='Bar chart of Class Name', xTitle='Class Name')<\/pre>\n<\/div>\n<p><br class=\"blank\" \/><\/p>\n<div style=\"text-align:center\">\n    <a href=\"https:\/\/plot.ly\/~susanli2005\/105\/\" target=\"_blank\" title=\"Plot 105\" style=\"text-align: center\"><img decoding=\"async\" src=\"https:\/\/plot.ly\/~susanli2005\/105.png\" alt=\"Plot 105\" style=\"max-width: 100%;width: 600px\" width=\"600\" \/><\/a><br \/>\n    <font size=\"-1\"><\/p>\n<div>Figure 11<\/div>\n<p><\/font>\n<\/div>\n<p>Now we come to \u201cReview Text\u201d feature, before explore this feature, we need to extract <a href=\"https:\/\/en.wikipedia.org\/wiki\/N-gram\" rel=\"noopener noreferrer\" target=\"_blank\">N-Gram<\/a> features. <a href=\"https:\/\/en.wikipedia.org\/wiki\/N-gram\" rel=\"noopener noreferrer\" target=\"_blank\">N-grams<\/a> are used to describe the number of words used as observation points, e.g., unigram means singly-worded, bigram means 2-worded phrase, and trigram means 3-worded phrase. In order to do this, we use <a href=\"https:\/\/scikit-learn.org\/stable\/modules\/generated\/sklearn.feature_extraction.text.CountVectorizer.html\" rel=\"noopener noreferrer\" target=\"_blank\">scikit-learn\u2019s <\/a><code><a href=\"https:\/\/scikit-learn.org\/stable\/modules\/generated\/sklearn.feature_extraction.text.CountVectorizer.html\" rel=\"noopener noreferrer\" target=\"_blank\">CountVectorizer<\/a><\/code><a href=\"https:\/\/scikit-learn.org\/stable\/modules\/generated\/sklearn.feature_extraction.text.CountVectorizer.html\" rel=\"noopener noreferrer\" target=\"_blank\"> function<\/a>.<\/p>\n<p>First, it would be interesting to compare unigrams before and after removing stop words.<\/p>\n<p><b><strong>The distribution of top unigrams before removing stop words<\/strong><\/b><\/p>\n<div style=\"width:98%;border:1px solid #ccc;overflow:auto;padding-left:10px;padding-bottom:10px;padding-top:10px\">\n<pre>\r\ndef get_top_n_words(corpus, n=None):\r\n    vec = CountVectorizer().fit(corpus)\r\n    bag_of_words = vec.transform(corpus)\r\n    sum_words = bag_of_words.sum(axis=0) \r\n    words_freq = [(word, sum_words[0, idx]) for word, idx in vec.vocabulary_.items()]\r\n    words_freq =sorted(words_freq, key = lambda x: x[1], reverse=True)\r\n    return words_freq[:n]\r\ncommon_words = get_top_n_words(df['Review Text'], 20)\r\nfor word, freq in common_words:\r\n    print(word, freq)\r\ndf1 = pd.DataFrame(common_words, columns = ['ReviewText' , 'count'])\r\ndf1.groupby('ReviewText').sum()['count'].sort_values(ascending=False).iplot(\r\nkind='bar', yTitle='Count', linecolor='black', title='Top 20 words in review before removing stop words')\r\n<\/pre>\n<\/div>\n<p><br class=\"blank\" \/><\/p>\n<div style=\"text-align:center\">\n<font size=\"-0.5\"><\/p>\n<div>top_unigram.py<\/div>\n<p><\/font>\n<\/div>\n<div style=\"text-align:center\">\n    <a href=\"https:\/\/plot.ly\/~susanli2005\/107\/\" target=\"_blank\" title=\"Plot 107\" style=\"text-align: center\"><img decoding=\"async\" src=\"https:\/\/plot.ly\/~susanli2005\/107.png\" alt=\"Plot 107\" style=\"max-width: 100%;width: 600px\" width=\"600\" \/><\/a><br \/>\n    <font size=\"-0.5\"><\/p>\n<div>Figure 12<\/div>\n<p><\/font>\n<\/div>\n<p><b>The distribution of top unigrams after removing stop words<\/b><\/p>\n<div style=\"width:98%;border:1px solid #ccc;overflow:auto;padding-left:10px;padding-bottom:10px;padding-top:10px\">\n<pre>\r\ndef get_top_n_words(corpus, n=None):\r\n    vec = CountVectorizer(stop_words = 'english').fit(corpus)\r\n    bag_of_words = vec.transform(corpus)\r\n    sum_words = bag_of_words.sum(axis=0) \r\n    words_freq = [(word, sum_words[0, idx]) for word, idx in vec.vocabulary_.items()]\r\n    words_freq =sorted(words_freq, key = lambda x: x[1], reverse=True)\r\n    return words_freq[:n]\r\ncommon_words = get_top_n_words(df['Review Text'], 20)\r\nfor word, freq in common_words:\r\n    print(word, freq)\r\ndf2 = pd.DataFrame(common_words, columns = ['ReviewText' , 'count'])\r\ndf2.groupby('ReviewText').sum()['count'].sort_values(ascending=False).iplot(\r\nkind='bar', yTitle='Count', linecolor='black', title='Top 20 words in review after removing stop words')\r\n<\/pre>\n<\/div>\n<p><br class=\"blank\" \/><\/p>\n<div style=\"text-align:center\">\n<font size=\"-0.5\"><\/p>\n<div>top_unigram_no_stopwords.py<\/div>\n<p><\/font>\n<\/div>\n<div style=\"text-align:center\">\n    <a href=\"https:\/\/plot.ly\/~susanli2005\/109\/\" target=\"_blank\" title=\"Plot 109\" style=\"text-align: center\"><img decoding=\"async\" src=\"https:\/\/plot.ly\/~susanli2005\/109.png\" alt=\"Plot 109\" style=\"max-width: 100%;width: 600px\" width=\"600\" \/><\/a><br \/>\n    <font size=\"-0.5\"><\/p>\n<div>Figure 13<\/div>\n<p><\/font>\n<\/div>\n<p>Second, we want to compare bigrams before and after removing stop words.<\/p>\n<p><b><strong>The distribution of top bigrams before removing stop words<\/strong><\/b><\/p>\n<div style=\"width:98%;border:1px solid #cc;overflow:auto;padding-left:10px;padding-bottom:10px;padding-top:10px\">\n<pre>\r\ndef get_top_n_bigram(corpus, n=None):\r\n    vec = CountVectorizer(ngram_range=(2, 2)).fit(corpus)\r\n    bag_of_words = vec.transform(corpus)\r\n    sum_words = bag_of_words.sum(axis=0) \r\n    words_freq = [(word, sum_words[0, idx]) for word, idx in vec.vocabulary_.items()]\r\n    words_freq =sorted(words_freq, key = lambda x: x[1], reverse=True)\r\n    return words_freq[:n]\r\ncommon_words = get_top_n_bigram(df['Review Text'], 20)\r\nfor word, freq in common_words:\r\n    print(word, freq)\r\ndf3 = pd.DataFrame(common_words, columns = ['ReviewText' , 'count'])\r\ndf3.groupby('ReviewText').sum()['count'].sort_values(ascending=False).iplot(\r\nkind='bar', yTitle='Count', linecolor='black', title='Top 20 bigrams in review before removing stop words')\r\n<\/pre>\n<\/div>\n<p><br class=\"blank\" \/><\/p>\n<div style=\"text-align:center\">\n<font size=\"-0.5\"><\/p>\n<div>top_bigram.py<\/div>\n<p><\/font>\n<\/div>\n<div style=\"text-align:center\"><img decoding=\"async\" src=\"https:\/\/www.kdnuggets.com\/wp-content\/uploads\/figure14.jpg\" alt=\"figure-name\" width=\"99%\" \/><font size=\"-1\"><\/p>\n<div>Figure 14<\/div>\n<p><\/font>\n<\/div>\n<p><b>The distribution of top bigrams after removing stop words<\/b><\/p>\n<div style=\"width:98%;border:1px solid #ccc;overflow:auto;padding-left:10px;padding-bottom:10px;padding-top:10px\">\n<pre>\r\ndef get_top_n_bigram(corpus, n=None):\r\n    vec = CountVectorizer(ngram_range=(2, 2), stop_words='english').fit(corpus)\r\n    bag_of_words = vec.transform(corpus)\r\n    sum_words = bag_of_words.sum(axis=0) \r\n    words_freq = [(word, sum_words[0, idx]) for word, idx in vec.vocabulary_.items()]\r\n    words_freq =sorted(words_freq, key = lambda x: x[1], reverse=True)\r\n    return words_freq[:n]\r\ncommon_words = get_top_n_bigram(df['Review Text'], 20)\r\nfor word, freq in common_words:\r\n    print(word, freq)\r\ndf4 = pd.DataFrame(common_words, columns = ['ReviewText' , 'count'])\r\ndf4.groupby('ReviewText').sum()['count'].sort_values(ascending=False).iplot(\r\nkind='bar', yTitle='Count', linecolor='black', title='Top 20 bigrams in review after removing stop words')\r\n<\/pre>\n<\/div>\n<p><br class=\"blank\" \/><\/p>\n<div style=\"text-align:center\">\n<font size=\"-0.5\"><\/p>\n<div>top_bigram_no_stopwords.py<\/div>\n<p><\/font>\n<\/div>\n<div style=\"text-align:center\">\n    <a href=\"https:\/\/plot.ly\/~susanli2005\/113\/\" target=\"_blank\" title=\"Plot 113\" style=\"text-align: center\"><img decoding=\"async\" src=\"https:\/\/plot.ly\/~susanli2005\/113.png\" alt=\"Plot 113\" style=\"max-width: 100%;width: 600px\" width=\"600\" \/><\/a><br \/>\n    <font size=\"-0.5\"><\/p>\n<div>Figure 15<\/div>\n<p><\/font>\n<\/div>\n<p>Last, we compare trigrams before and after removing stop words.<\/p>\n<p><b><strong>The distribution of Top trigrams before removing stop words<\/strong><\/b><\/p>\n<div style=\"width:98%;border:1px solid #ccc;overflow:auto;padding-left:10px;padding-bottom:10px;padding-top:10px\">\n<pre>\r\ndef get_top_n_trigram(corpus, n=None):\r\n    vec = CountVectorizer(ngram_range=(3, 3)).fit(corpus)\r\n    bag_of_words = vec.transform(corpus)\r\n    sum_words = bag_of_words.sum(axis=0) \r\n    words_freq = [(word, sum_words[0, idx]) for word, idx in vec.vocabulary_.items()]\r\n    words_freq =sorted(words_freq, key = lambda x: x[1], reverse=True)\r\n    return words_freq[:n]\r\ncommon_words = get_top_n_trigram(df['Review Text'], 20)\r\nfor word, freq in common_words:\r\n    print(word, freq)\r\ndf5 = pd.DataFrame(common_words, columns = ['ReviewText' , 'count'])\r\ndf5.groupby('ReviewText').sum()['count'].sort_values(ascending=False).iplot(\r\nkind='bar', yTitle='Count', linecolor='black', title='Top 20 trigrams in review before removing stop words')\r\n<\/pre>\n<\/div>\n<p><br class=\"blank\" \/><\/p>\n<div style=\"text-align:center\">\n<font size=\"-0.5\"><\/p>\n<div>top_trigram.py<\/div>\n<p><\/font>\n<\/div>\n<div style=\"text-align:center\"><img decoding=\"async\" src=\"https:\/\/www.kdnuggets.com\/wp-content\/uploads\/figure16.jpg\" alt=\"figure-name\" width=\"70%\" \/><font size=\"-1\"><\/p>\n<div class=\"caption\">Figure 16<\/div>\n<p><\/font>\n<\/div>\n<p><b>The distribution of Top trigrams after removing stop words<\/b><\/p>\n<div style=\"width:98%;border:1px solid #cc;overflow:auto;padding-left:10px;padding-bottom:10px;padding-top:10px\">\n<pre>\r\ndef get_top_n_trigram(corpus, n=None):\r\n    vec = CountVectorizer(ngram_range=(3, 3), stop_words='english').fit(corpus)\r\n    bag_of_words = vec.transform(corpus)\r\n    sum_words = bag_of_words.sum(axis=0) \r\n    words_freq = [(word, sum_words[0, idx]) for word, idx in vec.vocabulary_.items()]\r\n    words_freq =sorted(words_freq, key = lambda x: x[1], reverse=True)\r\n    return words_freq[:n]\r\ncommon_words = get_top_n_trigram(df['Review Text'], 20)\r\nfor word, freq in common_words:\r\n    print(word, freq)\r\ndf6 = pd.DataFrame(common_words, columns = ['ReviewText' , 'count'])\r\ndf6.groupby('ReviewText').sum()['count'].sort_values(ascending=False).iplot(\r\nkind='bar', yTitle='Count', linecolor='black', title='Top 20 trigrams in review after removing stop words')\r\n<\/pre>\n<\/div>\n<p><br class=\"blank\" \/><\/p>\n<div style=\"text-align:center\">\n<font size=\"-0.5\"><\/p>\n<div>top_trigram_no_stopwords.py<\/div>\n<p><\/font>\n<\/div>\n<div style=\"text-align:center\"><img decoding=\"async\" src=\"https:\/\/i.ibb.co\/F0dDxHC\/figure17.png\" alt=\"figure-name\" width=\"70%\" \/><font size=\"-1\"><\/p>\n<div class=\"caption\">Figure 17<\/div>\n<p><\/font>\n<\/div>\n<p><a href=\"https:\/\/en.wikipedia.org\/wiki\/Part-of-speech_tagging\" rel=\"noopener noreferrer\" target=\"_blank\"><strong><em>Part-Of-Speech Tagging (POS)<\/em><\/strong><\/a> is a process of assigning parts of speech to each word, such as noun, verb, adjective, etc<\/p>\n<p>We use a simple <a href=\"https:\/\/textblob.readthedocs.io\/en\/dev\/\" rel=\"noopener noreferrer\" target=\"_blank\"><strong><em>TextBlob<\/em><\/strong><\/a> API to dive into POS of our \u201cReview Text\u201d feature in our data set, and visualize these tags.<\/p>\n<p><b><strong>The distribution of top part-of-speech tags of review corpus<\/strong><\/b><\/p>\n<div style=\"width:98%;border:1px solid #ccc;overflow:auto;padding-left:10px;padding-bottom:10p;padding-top:10px\">\n<pre>\r\nblob = TextBlob(str(df['Review Text']))\r\npos_df = pd.DataFrame(blob.tags, columns = ['word' , 'pos'])\r\npos_df = pos_df.pos.value_counts()[:20]\r\npos_df.iplot(\r\n    kind='bar',\r\n    xTitle='POS',\r\n    yTitle='count', \r\ntitle='Top 20 Part-of-speech tagging for review corpus')\r\n<\/pre>\n<\/div>\n<p><br class=\"blank\" \/><\/p>\n<div style=\"text-align:center\">\n<font size=\"-0.5\"><\/p>\n<div>POS.py<\/div>\n<p><\/font>\n<\/div>\n<div style=\"text-align:center\">\n    <a href=\"https:\/\/plot.ly\/~susanli2005\/117\/\" target=\"_blank\" title=\"Plot 117\" style=\"text-align: center\"><img decoding=\"async\" src=\"https:\/\/plot.ly\/~susanli2005\/117.png\" alt=\"Plot 117\" style=\"max-width: 100%;width: 600px\" width=\"600\" \/><\/a><br \/>\n    <font size=\"-0.5\"><\/p>\n<div>Figure 18<\/div>\n<p><\/font>\n<\/div>\n<p>Box plot is used to compare the sentiment polarity score, rating, review text lengths of each department or division of the e-commerce store.<\/p>\n<p><b>What do the departments tell about Sentiment polarity<\/b><\/p>\n<div style=\"width:98%;border:1px solid #ccc;overflow:auto;padding-left:10px;padding-bottom:10px;padding-top:10px\">\n<pre>\r\ny0 = df.loc[df['Department Name'] == 'Tops']['polarity']\r\ny1 = df.loc[df['Department Name'] == 'Dresses']['polarity']\r\ny2 = df.loc[df['Department Name'] == 'Bottoms']['polarity']\r\ny3 = df.loc[df['Department Name'] == 'Intimate']['polarity']\r\ny4 = df.loc[df['Department Name'] == 'Jackets']['polarity']\r\ny5 = df.loc[df['Department Name'] == 'Trend']['polarity']\r\n\r\ntrace0 = go.Box(\r\n    y=y0,\r\n    name = 'Tops',\r\n    marker = dict(\r\n        color = 'rgb(214, 12, 140)',\r\n    )\r\n)\r\ntrace1 = go.Box(\r\n    y=y1,\r\n    name = 'Dresses',\r\n    marker = dict(\r\n        color = 'rgb(0, 128, 128)',\r\n    )\r\n)\r\ntrace2 = go.Box(\r\n    y=y2,\r\n    name = 'Bottoms',\r\n    marker = dict(\r\n        color = 'rgb(10, 140, 208)',\r\n    )\r\n)\r\ntrace3 = go.Box(\r\n    y=y3,\r\n    name = 'Intimate',\r\n    marker = dict(\r\n        color = 'rgb(12, 102, 14)',\r\n    )\r\n)\r\ntrace4 = go.Box(\r\n    y=y4,\r\n    name = 'Jackets',\r\n    marker = dict(\r\n        color = 'rgb(10, 0, 100)',\r\n    )\r\n)\r\ntrace5 = go.Box(\r\n    y=y5,\r\n    name = 'Trend',\r\n    marker = dict(\r\n        color = 'rgb(100, 0, 10)',\r\n    )\r\n)\r\ndata = [trace0, trace1, trace2, trace3, trace4, trace5]\r\nlayout = go.Layout(\r\n    title = \"Sentiment Polarity Boxplot of Department Name\"\r\n)\r\n\r\nfig = go.Figure(data=data,layout=layout)\r\niplot(fig, filename = \"Sentiment Polarity Boxplot of Department Name\")\r\n<\/pre>\n<\/div>\n<p><br class=\"blank\" \/><\/p>\n<div style=\"text-align:center\">\n<font size=\"-0.5\"><\/p>\n<div>department_polarity.py\n<\/div>\n<p><\/font>\n<\/div>\n<div style=\"text-align:center\">\n    <a href=\"https:\/\/plot.ly\/~susanli2005\/125\/\" target=\"_blank\" title=\"Plot 125\" style=\"text-align: center\"><img decoding=\"async\" src=\"https:\/\/plot.ly\/~susanli2005\/125.png\" alt=\"Plot 125\" style=\"max-width: 100%;width: 600px\" width=\"600\" \/><\/a><br \/>\n    <font size=\"-0.5\"><\/p>\n<div>Figure 19<\/div>\n<p><\/font>\n<\/div>\n<p>The highest sentiment polarity score was achieved by all of the six departments except Trend department, and the lowest sentiment polarity score was collected by Tops department. And the Trend department has the lowest median polarity score. If you remember, the Trend department has the least number of reviews. This explains why it does not have as wide variety of score distribution as the other departments.<\/p>\n<p><b>What do the departments tell about rating<\/b><\/p>\n<div style=\"width:98%;border:1px solid #ccc;overflow:auto;padding-left:10px;padding-bottom:10px;padding-top:10px\">\n<pre>\r\ny0 = df.loc[df['Department Name'] == 'Tops']['Rating']\r\ny1 = df.loc[df['Department Name'] == 'Dresses']['Rating']\r\ny2 = df.loc[df['Department Name'] == 'Bottoms']['Rating']\r\ny3 = df.loc[df['Department Name'] == 'Intimate']['Rating']\r\ny4 = df.loc[df['Department Name'] == 'Jackets']['Rating']\r\ny5 = df.loc[df['Department Name'] == 'Trend']['Rating']\r\n\r\ntrace0 = go.Box(\r\n    y=y0,\r\n    name = 'Tops',\r\n    marker = dict(\r\n        color = 'rgb(214, 12, 140)',\r\n    )\r\n)\r\ntrace1 = go.Box(\r\n    y=y1,\r\n    name = 'Dresses',\r\n    marker = dict(\r\n        color = 'rgb(0, 128, 128)',\r\n    )\r\n)\r\ntrace2 = go.Box(\r\n    y=y2,\r\n    name = 'Bottoms',\r\n    marker = dict(\r\n        color = 'rgb(10, 140, 208)',\r\n    )\r\n)\r\ntrace3 = go.Box(\r\n    y=y3,\r\n    name = 'Intimate',\r\n    marker = dict(\r\n        color = 'rgb(12, 102, 14)',\r\n    )\r\n)\r\ntrace4 = go.Box(\r\n    y=y4,\r\n    name = 'Jackets',\r\n    marker = dict(\r\n        color = 'rgb(10, 0, 100)',\r\n    )\r\n)\r\ntrace5 = go.Box(\r\n    y=y5,\r\n    name = 'Trend',\r\n    marker = dict(\r\n        color = 'rgb(100, 0, 10)',\r\n    )\r\n)\r\ndata = [trace0, trace1, trace2, trace3, trace4, trace5]\r\nlayout = go.Layout(\r\n    title = \"Rating Boxplot of Department Name\"\r\n)\r\n\r\nfig = go.Figure(data=data,layout=layout)\r\niplot(fig, filename = \"Rating Boxplot of Department Name\")\r\n<\/pre>\n<\/div>\n<p><br class=\"blank\" \/><\/p>\n<div style=\"text-align:center\">\n<font size=\"-0.5\"><\/p>\n<div>rating_division.py\n<\/div>\n<p><\/font>\n<\/div>\n<div style=\"text-align:center\">\n    <a href=\"https:\/\/plot.ly\/~susanli2005\/121\/\" target=\"_blank\" title=\"Plot 121\" style=\"text-align: center\"><img decoding=\"async\" src=\"https:\/\/plot.ly\/~susanli2005\/121.png\" alt=\"Plot 121\" style=\"max-width: 100%;width: 600px\" width=\"600\" \/><\/a><br \/>\n    <font size=\"-0.5\"><\/p>\n<div>Figure 20<\/div>\n<p><\/font>\n<\/div>\n<p>Except Trend department, all the other departments\u2019 median rating were 5. Overall, the ratings are high and sentiment are positive in this review data set.<\/p>\n<p><b>Review length by department<\/b><\/p>\n<div style=\"width:98%;border:1px solid #ccc;overflow:auto;padding-left:10px;padding-bottom:10px;padding-top:10px\">\n<pre>\r\ny0 = df.loc[df['Department Name'] == 'Tops']['review_len']\r\ny1 = df.loc[df['Department Name'] == 'Dresses']['review_len']\r\ny2 = df.loc[df['Department Name'] == 'Bottoms']['review_len']\r\ny3 = df.loc[df['Department Name'] == 'Intimate']['review_len']\r\ny4 = df.loc[df['Department Name'] == 'Jackets']['review_len']\r\ny5 = df.loc[df['Department Name'] == 'Trend']['review_len']\r\n\r\ntrace0 = go.Box(\r\n    y=y0,\r\n    name = 'Tops',\r\n    marker = dict(\r\n        color = 'rgb(214, 12, 140)',\r\n    )\r\n)\r\ntrace1 = go.Box(\r\n    y=y1,\r\n    name = 'Dresses',\r\n    marker = dict(\r\n        color = 'rgb(0, 128, 128)',\r\n    )\r\n)\r\ntrace2 = go.Box(\r\n    y=y2,\r\n    name = 'Bottoms',\r\n    marker = dict(\r\n        color = 'rgb(10, 140, 208)',\r\n    )\r\n)\r\ntrace3 = go.Box(\r\n    y=y3,\r\n    name = 'Intimate',\r\n    marker = dict(\r\n        color = 'rgb(12, 102, 14)',\r\n    )\r\n)\r\ntrace4 = go.Box(\r\n    y=y4,\r\n    name = 'Jackets',\r\n    marker = dict(\r\n        color = 'rgb(10, 0, 100)',\r\n    )\r\n)\r\ntrace5 = go.Box(\r\n    y=y5,\r\n    name = 'Trend',\r\n    marker = dict(\r\n        color = 'rgb(100, 0, 10)',\r\n    )\r\n)\r\ndata = [trace0, trace1, trace2, trace3, trace4, trace5]\r\nlayout = go.Layout(\r\n    title = \"Review length Boxplot of Department Name\"\r\n)\r\n\r\nfig = go.Figure(data=data,layout=layout)\r\niplot(fig, filename = \"Review Length Boxplot of Department Name\")\r\n<\/pre>\n<\/div>\n<p><br class=\"blank\" \/><\/p>\n<div style=\"text-align:center\">\n<font size=\"-0.5\"><\/p>\n<div>length_department.py\n<\/div>\n<p><\/font>\n<\/div>\n<div style=\"text-align:center\">\n    <a href=\"https:\/\/plot.ly\/~susanli2005\/123\/\" target=\"_blank\" title=\"Plot 123\" style=\"text-align: center\"><img decoding=\"async\" src=\"https:\/\/plot.ly\/~susanli2005\/123.png\" alt=\"Plot 123\" style=\"max-width: 100%;width: 600px\" width=\"600\" \/><\/a><br \/>\n    <font size=\"-0.5\"><\/p>\n<div>Figure 21<\/div>\n<p><\/font>\n<\/div>\n<p>The median review length of Tops &amp; Intimate departments are relative lower than those of the other departments.<\/p>\n<p><!--nextpage--><\/p>\n<div align=\"right\"><img loading=\"lazy\" decoding=\"async\" src=\"\/images\/comment.gif\" width=\"16\" height=\"12\" alt=\"c\"> <a href=\"\/2019\/05\/complete-exploratory-data-analysis-visualization-text-data.html?page=2#comments\">comments<\/a><\/div>\n<h3>Bivariate visualization with Plotly<\/h3>\n<p>&nbsp;<br \/>\nBivariate visualization is a type of visualization that consists two features at a time. It describes association or relationship between two features.<\/p>\n<p><b><strong>Distribution of sentiment polarity score by recommendations<\/strong><\/b><\/p>\n<div style=\"width:98%;border:1px solid #ccc;overflow:auto;padding-left:10px;padding-bottom:10px;padding-top:10px\">\n<pre>\r\nx1 = df.loc[df['Recommended IND'] == 1, 'polarity']\r\nx0 = df.loc[df['Recommended IND'] == 0, 'polarity']\r\n\r\ntrace1 = go.Histogram(\r\n    x=x0, name='Not recommended',\r\n    opacity=0.75\r\n)\r\ntrace2 = go.Histogram(\r\n    x=x1, name = 'Recommended',\r\n    opacity=0.75\r\n)\r\n\r\ndata = [trace1, trace2]\r\nlayout = go.Layout(barmode='overlay', title='Distribution of Sentiment polarity of reviews based on Recommendation')\r\nfig = go.Figure(data=data, layout=layout)\r\n\r\niplot(fig, filename='overlaid histogram')\r\n<\/pre>\n<\/div>\n<p><br class=\"blank\" \/><\/p>\n<div style=\"text-align:center\">\n<font size=\"-0.5\"><\/p>\n<div>\npolarity_recommendation.py\n<\/div>\n<p><\/font>\n<\/div>\n<div style=\"text-align:center\">\n    <a href=\"https:\/\/plot.ly\/~susanli2005\/129\/\" target=\"_blank\" title=\"Plot 129\" style=\"text-align: center\"><img decoding=\"async\" src=\"https:\/\/plot.ly\/~susanli2005\/129.png\" alt=\"Plot 129\" style=\"max-width: 100%;width: 600px\" width=\"600\" \/><\/a><br \/>\n    <font size=\"-0.5\"><\/p>\n<div>Figure 22<\/div>\n<p><\/font>\n<\/div>\n<p>It is obvious that reviews have higher polarity score are more likely to be recommended.<\/p>\n<p><b>Distribution of ratings by recommendations<\/b><\/p>\n<div style=\"width:98%;border:1px solid #ccc;overflow:auto;padding-left:10px;padding-top:10px\">\n<pre>\r\nx1 = df.loc[df['Recommended IND'] == 1, 'Rating']\r\nx0 = df.loc[df['Recommended IND'] == 0, 'Rating']\r\n\r\ntrace1 = go.Histogram(\r\n    x=x0, name='Not recommended',\r\n    opacity=0.75\r\n)\r\ntrace2 = go.Histogram(\r\n    x=x1, name = 'Recommended',\r\n    opacity=0.75\r\n)\r\n\r\ndata = [trace1, trace2]\r\nlayout = go.Layout(barmode='overlay', title='Distribution of Sentiment polarity of reviews based on Recommendation')\r\nfig = go.Figure(data=data, layout=layout)\r\n\r\niplot(fig, filename='overlaid histogram')\r\n<\/pre>\n<\/div>\n<p><br class=\"blank\" \/><\/p>\n<div style=\"text-align:center\">\n<font size=\"-0.5\"><\/p>\n<div>\nrating_recommendation.py\n<\/div>\n<p><\/font>\n<\/div>\n<div style=\"text-align:center\">\n    <a href=\"https:\/\/plot.ly\/~susanli2005\/132\/\" target=\"_blank\" title=\"Plot 132\" style=\"text-align: center\"><img decoding=\"async\" src=\"https:\/\/plot.ly\/~susanli2005\/132.png\" alt=\"Plot 132\" style=\"max-width: 100%;width: 600px\" width=\"600\" \/><\/a><br \/>\n    <font size=\"-0.5\"><\/p>\n<div>Figure 23<\/div>\n<p><\/font>\n<\/div>\n<p>Recommended reviews have higher ratings than those of not recommended ones.<\/p>\n<p><b><strong>Distribution of review lengths by recommendations<\/strong><\/b><\/p>\n<div style=\"width:98%;border:1px solid #ccc;overflow:auto;padding-left:10px;padding-top:10px\">\n<pre>\r\nx1 = df.loc[df['Recommended IND'] == 1, 'review_len']\r\nx0 = df.loc[df['Recommended IND'] == 0, 'review_len']\r\n\r\ntrace1 = go.Histogram(\r\n    x=x0, name='Not recommended',\r\n    opacity=0.75\r\n)\r\ntrace2 = go.Histogram(\r\n    x=x1, name = 'Recommended',\r\n    opacity=0.75\r\n)\r\n\r\ndata = [trace1, trace2]\r\nlayout = go.Layout(barmode = 'group', title='Distribution of Review Lengths Based on Recommendation')\r\nfig = go.Figure(data=data, layout=layout)\r\n\r\niplot(fig, filename='stacked histogram')\r\n<\/pre>\n<\/div>\n<p><\/p>\n<div style=\"text-align:center\">\n<font size=\"-0.5\"><\/p>\n<div>\nreview_length_recommend.py\n<\/div>\n<p><\/font>\n<\/div>\n<div style=\"text-align:center\"><img decoding=\"async\" src=\"https:\/\/www.kdnuggets.com\/wp-content\/uploads\/figure24.jpg\" alt=\"figure-name\" width=\"9%\" \/><font size=\"-1\"><\/p>\n<div class=\"caption\">Figure 24<\/div>\n<p><\/font>\n<\/div>\n<p>Recommended reviews tend to be lengthier than those of not recommended reviews.<\/p>\n<p><b><strong>2D Density jointplot of sentiment polarity vs. rating<\/strong><\/b><\/p>\n<div style=\"width:98%;border:1px solid #ccc;overflow:auto;padding-left:10px;padding-bottom:10px;padding-top:10px\">\n<pre>\r\ntrace1 = go.Scatter(\r\n    x=df['polarity'], y=df['Rating'], mode='markers', name='points',\r\n    marker=dict(color='rgb(102,0,0)', size=2, opacity=0.4)\r\n)\r\ntrace2 = go.Histogram2dContour(\r\n    x=df['polarity'], y=df['Rating'], name='density', ncontours=20,\r\n    colorscale='Hot', reversescale=True, showscale=False\r\n)\r\ntrace3 = go.Histogram(\r\n    x=df['polarity'], name='Sentiment polarity density',\r\n    marker=dict(color='rgb(102,0,0)'),\r\n    yaxis='y2'\r\n)\r\ntrace4 = go.Histogram(\r\n    y=df['Rating'], name='Rating density', marker=dict(color='rgb(102,0,0)'),\r\n    xaxis='x2'\r\n)\r\ndata = [trace1, trace2, trace3, trace4]\r\n\r\nlayout = go.Layout(\r\n    showlegend=False,\r\n    autosize=False,\r\n    width=600,\r\n    height=550,\r\n    xaxis=dict(\r\n        domain=[0, 0.85],\r\n        showgrid=False,\r\n        zeroline=False\r\n    ),\r\n    yaxis=dict(\r\n        domain=[0, 0.85],\r\n        showgrid=False,\r\n        zeroline=False\r\n    ),\r\n    margin=dict(\r\n        t=50\r\n    ),\r\n    hovermode='closest',\r\n    bargap=0,\r\n    xaxis2=dict(\r\n        domain=[0.85, 1],\r\n        showgrid=False,\r\n        zeroline=False\r\n    ),\r\n    yaxis2=dict(\r\n        domain=[0.85, 1],\r\n        showgrid=False,\r\n        zeroline=False\r\n    )\r\n)\r\n\r\nfig = go.Figure(data=data, layout=layout)\r\niplot(fig, filename='2dhistogram-2d-density-plot-subplots')\r\n<\/pre>\n<\/div>\n<p><br class=\"blank\" \/><\/p>\n<div style=\"text-align:center\">\n<font size=\"-0.5\"><\/p>\n<div>\nsentiment_polarity_rating.py\n<\/div>\n<p><\/font>\n<\/div>\n<div style=\"text-align:center\">\n    <a href=\"https:\/\/plot.ly\/~susanli2005\/134\/\" target=\"_blank\" title=\"Plot 134\" style=\"text-align: center\"><img decoding=\"async\" src=\"https:\/\/plot.ly\/~susanli2005\/134.png\" alt=\"Plot 134\" style=\"max-width: 100%;width: 600px\" width=\"600\" \/><\/a><br \/>\n    <font size=\"-0.5\"><\/p>\n<div>Figure 24<\/div>\n<p><\/font>\n<\/div>\n<p><b><strong>2D Density jointplot of age and sentiment polarity<\/strong><\/b><\/p>\n<div style=\"width:98%;border:1px solid #ccc;overflow:auto;padding-left:10px;padding-bottom:10px;padding-top:10px\">\n<pre>\r\ntrace1 = go.Scatter(\r\n    x=df['Age'], y=df['polarity'], mode='markers', name='points',\r\n    marker=dict(color='rgb(102,0,0)', size=2, opacity=0.4)\r\n)\r\ntrace2 = go.Histogram2dContour(\r\n    x=df['Age'], y=df['polarity'], name='density', ncontours=20,\r\n    colorscale='Hot', reversescale=True, showscale=False\r\n)\r\ntrace3 = go.Histogram(\r\n    x=df['Age'], name='Age density',\r\n    marker=dict(color='rgb(102,0,0)'),\r\n    yaxis='y2'\r\n)\r\ntrace4 = go.Histogram(\r\n    y=df['polarity'], name='Sentiment Polarity density', marker=dict(color='rgb(102,0,0)'),\r\n    xaxis='x2'\r\n)\r\ndata = [trace1, trace2, trace3, trace4]\r\n\r\nlayout = go.Layout(\r\n    showlegend=False,\r\n    autosize=False,\r\n    width=600,\r\n    height=550,\r\n    xaxis=dict(\r\n        domain=[0, 0.85],\r\n        showgrid=False,\r\n        zeroline=False\r\n    ),\r\n    yaxis=dict(\r\n        domain=[0, 0.85],\r\n        showgrid=False,\r\n        zeroline=False\r\n    ),\r\n    margin=dict(\r\n        t=50\r\n    ),\r\n    hovermode='closest',\r\n    bargap=0,\r\n    xaxis2=dict(\r\n        domain=[0.85, 1],\r\n        showgrid=False,\r\n        zeroline=False\r\n    ),\r\n    yaxis2=dict(\r\n        domain=[0.85, 1],\r\n        showgrid=False,\r\n        zeroline=False\r\n    )\r\n)\r\n\r\nfig = go.Figure(data=data, layout=layout)\r\niplot(fig, filename='2dhistogram-2d-density-plot-subplots')\r\n<\/pre>\n<\/div>\n<p><br class=\"blank\" \/><\/p>\n<div style=\"text-align:center\">\n<font size=\"-0.5\"><\/p>\n<div>\nage_polarity.py\n<\/div>\n<p><\/font>\n<\/div>\n<div style=\"text-align:center\">\n    <a href=\"https:\/\/plot.ly\/~susanli2005\/136\/\" target=\"_blank\" title=\"Plot 136\" style=\"text-align: center\"><img decoding=\"async\" src=\"https:\/\/plot.ly\/~susanli2005\/136.png\" alt=\"Plot 136\" style=\"max-width: 100%;width: 600px\" width=\"600\" \/><\/a><br \/>\n    <font size=\"-0.5\"><\/p>\n<div>Figure 25<\/div>\n<p><\/font>\n<\/div>\n<p>There were few people are very positive or very negative. People who give neutral to positive reviews are more likely to be in their 30s. Probably people at these age are likely to be more active.<\/p>\n<p>&nbsp;<\/p>\n<h3>Finding characteristic terms and their associations<\/h3>\n<p>&nbsp;<br \/>\nSometimes we want to analyzes words used by different categories and outputs some notable term associations. We will use <a href=\"https:\/\/github.com\/JasonKessler\/scattertext#using-scattertext-as-a-text-analysis-library-finding-characteristic-terms-and-their-associations\" rel=\"noopener noreferrer\" target=\"_blank\">scattertext<\/a> and <a href=\"https:\/\/github.com\/explosion\/spaCy\" rel=\"noopener noreferrer\" target=\"_blank\">spaCy<\/a> libraries to accomplish these.<\/p>\n<p>First, we need to turn the data frame into a Scattertext Corpus. To look for differences in department name, set the <code>category_col<\/code>parameter to <code>'Department Names'<\/code>, and use the review present in the <code>Review Text<\/code> column, to analyze by setting the <code>text<\/code> col parameter. Finally, pass a spaCy model in to the <code>nlp<\/code> argument and call <code>build()<\/code> to construct the corpus.<\/p>\n<p>Following are the terms that differentiate the review text from a general English corpus.<\/p>\n<div style=\"width:98%;border:1px solid #ccc;overflow:auto;padding-left:10px;padding-bottom:10px;padding-top:10px\">\n<pre>\r\ncorpus = st.CorpusFromPandas(df, category_col='Department Name', text_col='Review Text', nlp=nlp).build()\r\nprint(list(corpus.get_scaled_f_scores_vs_background().index[:10]))<\/pre>\n<\/div>\n<p><br class=\"blank\" \/><\/p>\n<div style=\"text-align:center\"><img decoding=\"async\" src=\"\/wp-content\/uploads\/figure26.png\" alt=\"figure-name\" width=\"80%\" \/><font size=\"-1\"><\/p>\n<div class=\"caption\">Figure 26<\/div>\n<p><\/font>\n<\/div>\n<p>Following are the terms in review text that are most associated with the Tops department:<\/p>\n<div style=\"width:98%;border:1px solid #ccc;overflow:auto;padding-left:10px;padding-bottom:10px;padding-top:10px\">\n<pre>\r\nterm_freq_df = corpus.get_term_freq_df()\r\nterm_freq_df['Tops Score'] = corpus.get_scaled_f_scores('Tops')\r\npprint(list(term_freq_df.sort_values(by='Tops Score', ascending=False).index[:10]))\r\n<\/pre>\n<\/div>\n<p><br class=\"blank\" \/><\/p>\n<div style=\"text-align:center\"><img decoding=\"async\" src=\"\/wp-content\/uploads\/figure27.png\" alt=\"figure-name\" width=\"60%\" \/><font size=\"-1\"><\/p>\n<div class=\"caption\">Figure 27<\/div>\n<p><\/font>\n<\/div>\n<p>Following are the terms that are most associated with the Dresses department:<\/p>\n<div style=\"width:98%;border:1px solid #ccc;overflow:auto;padding-left:10px;padding-bottom:10px;padding-top:10px\">\n<pre>term_freq_df['Dresses Score'] = corpus.get_scaled_f_scores('Dresses')\r\npprint(list(term_freq_df.sort_values(by='Dresses Score', ascending=False).index[:10]))<\/pre>\n<\/div>\n<p><br class=\"blank\" \/><\/p>\n<div style=\"text-align:center\"><img decoding=\"async\" src=\"\/wp-content\/uploads\/figure28.png\" alt=\"figure-name\" width=\"60%\" \/><font size=\"-1\"><\/p>\n<div class=\"caption\">Figure 28<\/div>\n<p><\/font>\n<\/div>\n<p>&nbsp;<\/p>\n<h3>Topic Modeling Review Text<\/h3>\n<p>&nbsp;<br \/>\nFinally, we want to explore topic modeling algorithm to this data set, to see whether it would provide any benefit, and fit with what we are doing for our review text feature.<\/p>\n<p>We will experiment with Latent Semantic Analysis (LSA) technique in topic modeling.<\/p>\n<ul class=\"six_ul\">\n<li>Generating our document-term matrix from review text to a matrix of <a href=\"https:\/\/scikit-learn.org\/stable\/modules\/generated\/sklearn.feature_extraction.text.TfidfVectorizer.html\" rel=\"noopener noreferrer\" target=\"_blank\">TF-IDF features<\/a>.\n<li>LSA model replaces raw counts in the document-term matrix with a TF-IDF score.\n<li>Perform dimensionality reduction on the document-term matrix using <a href=\"https:\/\/scikit-learn.org\/stable\/modules\/generated\/sklearn.decomposition.TruncatedSVD.html\" rel=\"noopener noreferrer\" target=\"_blank\">truncated SVD<\/a>.\n<li>Because the number of department is 6, we set <code>n_topics=6<\/code>.\n<li>Taking the <code>argmax<\/code> of each review text in this topic matrix will give the predicted topics of each review text in the data. We can then sort these into counts of each topic.\n<li>To better understand each topic, we will find the most frequent three words in each topic.\n<\/ul>\n<div style=\"width:98%;border:1px solid #ccc;overflow:auto;padding-left:10px;padding-bottom:10px;padding-top:10px\">\n<pre>\r\nreindexed_data = df['Review Text']\r\ntfidf_vectorizer = TfidfVectorizer(stop_words='english', use_idf=True, smooth_idf=True)\r\nreindexed_data = reindexed_data.values\r\ndocument_term_matrix = tfidf_vectorizer.fit_transform(reindexed_data)\r\nn_topics = 6\r\nlsa_model = TruncatedSVD(n_components=n_topics)\r\nlsa_topic_matrix = lsa_model.fit_transform(document_term_matrix)\r\n\r\ndef get_keys(topic_matrix):\r\n    '''\r\n    returns an integer list of predicted topic \r\n    categories for a given topic matrix\r\n    '''\r\n    keys = topic_matrix.argmax(axis=1).tolist()\r\n    return keys\r\n\r\ndef keys_to_counts(keys):\r\n    '''\r\n    returns a tuple of topic categories and their \r\n    accompanying magnitudes for a given list of keys\r\n    '''\r\n    count_pairs = Counter(keys).items()\r\n    categories = [pair[0] for pair in count_pairs]\r\n    counts = [pair[1] for pair in count_pairs]\r\n    return (categories, counts)\r\n    \r\nlsa_keys = get_keys(lsa_topic_matrix)\r\nlsa_categories, lsa_counts = keys_to_counts(lsa_keys)\r\n\r\ndef get_top_n_words(n, keys, document_term_matrix, tfidf_vectorizer):\r\n    '''\r\n    returns a list of n_topic strings, where each string contains the n most common \r\n    words in a predicted category, in order\r\n    '''\r\n    top_word_indices = []\r\n    for topic in range(n_topics):\r\n        temp_vector_sum = 0\r\n        for i in range(len(keys)):\r\n            if keys[i] == topic:\r\n                temp_vector_sum += document_term_matrix[i]\r\n        temp_vector_sum = temp_vector_sum.toarray()\r\n        top_n_word_indices = np.flip(np.argsort(temp_vector_sum)[0][-n:],0)\r\n        top_word_indices.append(top_n_word_indices)   \r\n    top_words = []\r\n    for topic in top_word_indices:\r\n        topic_words = []\r\n        for index in topic:\r\n            temp_word_vector = np.zeros((1,document_term_matrix.shape[1]))\r\n            temp_word_vector[:,index] = 1\r\n            the_word = tfidf_vectorizer.inverse_transform(temp_word_vector)[0][0]\r\n            topic_words.append(the_word.encode('ascii').decode('utf-8'))\r\n        top_words.append(\" \".join(topic_words))         \r\n    return top_words\r\n    \r\n    top_n_words_lsa = get_top_n_words(3, lsa_keys, document_term_matrix, tfidf_vectorizer)\r\n\r\nfor i in range(len(top_n_words_lsa)):\r\nprint(\"Topic {}: \".format(i+1), top_n_words_lsa[i])\r\n<\/pre>\n<\/div>\n<p><br class=\"blank\" \/><\/p>\n<div style=\"text-align:center\">\n<font size=\"-0.5\"><\/p>\n<div>\ntopic_model_LSA.py\n<\/div>\n<p><\/font>\n<\/div>\n<div style=\"text-align:center\"><img decoding=\"async\" src=\"\/wp-content\/uploads\/figure29.png\" alt=\"figure-name\" width=\"70%\" \/><font size=\"-1\"><\/p>\n<div class=\"caption\">Figure 29<\/div>\n<p><\/font>\n<\/div>\n<div style=\"width:98%;border:1px solid #ccc;overflow:auto;padding-left:10px;padding-bottom:10px;padding-top:10px\">\n<pre>\r\ntop_3_words = get_top_n_words(3, lsa_keys, document_term_matrix, tfidf_vectorizer)\r\nlabels = ['Topic {}: \\n'.format(i) + top_3_words[i] for i in lsa_categories]\r\n\r\nfig, ax = plt.subplots(figsize=(16,8))\r\nax.bar(lsa_categories, lsa_counts);\r\nax.set_xticks(lsa_categories);\r\nax.set_xticklabels(labels);\r\nax.set_ylabel('Number of review text');\r\nax.set_title('LSA topic counts');\r\nplt.show();\r\n<\/pre>\n<\/div>\n<p><br class=\"blank\" \/><\/p>\n<div style=\"text-align:center\"><img decoding=\"async\" src=\"https:\/\/www.kdnuggets.com\/wp-content\/uploads\/figure30.jpg\" alt=\"figure-name\" width=\"70%\" \/><font size=\"-1\"><\/p>\n<div class=\"caption\">Figure 30<\/div>\n<p><\/font>\n<\/div>\n<p>By looking at the most frequent words in each topic, we have a sense that we may not reach any degree of separation across the topic categories. In another word, we could not separate review text by departments using topic modeling techniques.<\/p>\n<p>Topic modeling techniques have a number of important limitations. To begin, the term \u201ctopic\u201d is somewhat ambigious, and by now it is perhaps clear that topic models will not produce highly nuanced classification of texts for our data.<\/p>\n<p>In addition, we can observe that the vast majority of the review text are categorized to the first topic (Topic 0). The t-SNE visualization of LSA topic modeling won\u2019t be pretty.<\/p>\n<p>All the code can be found on the <a href=\"https:\/\/github.com\/susanli2016\/NLP-with-Python\/blob\/master\/EDA%20and%20visualization%20for%20Text%20Data.ipynb\" rel=\"noopener noreferrer\" target=\"_blank\">Jupyter notebook<\/a>. And code plus the interactive visualizations can be viewed on <a href=\"https:\/\/nbviewer.jupyter.org\/github\/susanli2016\/NLP-with-Python\/blob\/master\/EDA%20and%20visualization%20for%20Text%20Data.ipynb\" rel=\"noopener noreferrer\" target=\"_blank\">nbviewer<\/a>.<\/p>\n<p>Happy Monday!<\/p>\n<p>&nbsp;<br \/>\n<b>Bio: <a href=\"https:\/\/www.linkedin.com\/in\/susanli\/\" target=\"_blank\" rel=\"noopener noreferrer\">Susan Li<\/a><\/b> is changing the world, one article at a time. She is a Sr. Data Scientist, located in Toronto, Canada. <\/p>\n<p><a href=\"https:\/\/towardsdatascience.com\/a-complete-exploratory-data-analysis-and-visualization-for-text-data-29fb1b96fb6a\" target=\"_blank\" rel=\"noopener noreferrer\">Original<\/a>. Reposted with permission.<\/p>\n<p><b>Related:<\/b><\/p>\n<ul class=\"three_ul\">\n<li><a href=\"\/2019\/01\/elmo-contextual-language-embedding.html\">ELMo: Contextual Language Embedding<\/a>\n<li><a href=\"\/2019\/04\/text-preprocessing-nlp-machine-learning.html\">All you need to know about text preprocessing for NLP and Machine Learning<\/a>\n<li><a href=\"\/2018\/09\/machine-learning-text-classification-using-spacy-python.html\">Machine Learning for Text Classification Using SpaCy in Python<\/a>\n<\/ul>\n<p><a name=\"comments\"><\/a><\/p>\n<div id=\"disqus_thread\"><\/div>\n<p> <script type=\"text\/javascript\">\n var disqus_shortname = 'kdnuggets';\n <!--(function() { var dsq = document.createElement('script'); dsq.type = 'text\/javascript'; dsq.async = true; dsq.src = 'https:\/\/kdnuggets.disqus.com\/embed.js';-->\n <!--(document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq); })();-->\n <\/script><\/p>\n","protected":false},"excerpt":{"rendered":"Visually representing the content of a text document is one of the most important tasks in the field of text mining as a Data Scientist or NLP specialist. However, there are some gaps between visualizing unstructured (text) data and structured data.\n","protected":false},"author":167,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_seopress_titles_title":"","_seopress_titles_desc":"","_seopress_robots_index":"","_seopress_robots_follow":"","_seopress_robots_imageindex":"","_seopress_robots_snippet":"","_seopress_robots_primary_cat":"","_seopress_robots_breadcrumbs":"","_seopress_robots_freeze_modified_date":"","_seopress_robots_custom_modified_date":"","_seopress_robots_canonical":"","_seopress_social_fb_title":"","_seopress_social_fb_desc":"","_seopress_social_fb_img":"","_seopress_social_fb_img_attachment_id":0,"_seopress_social_fb_img_width":0,"_seopress_social_fb_img_height":0,"_seopress_social_twitter_title":"","_seopress_social_twitter_desc":"","_seopress_social_twitter_img":"","_seopress_social_twitter_img_attachment_id":0,"_seopress_social_twitter_img_width":0,"_seopress_social_twitter_img_height":0,"_seopress_redirections_value":"","_seopress_redirections_enabled":"","_seopress_redirections_enabled_regex":"","_seopress_redirections_logged_status":"","_seopress_redirections_param":"","_seopress_redirections_type":0,"_seopress_analysis_target_kw":"","inline_featured_image":false,"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"mc4wp_mailchimp_campaign":[],"footnotes":"","_links_to":"","_links_to_target":""},"categories":[5286],"tags":[252,199,2832,203,440],"class_list":["post-93720","post","type-post","status-publish","format-standard","hentry","category-kdnuggets-originals","tag-data-visualization","tag-nlp","tag-plotly","tag-python","tag-text-analytics"],"acf":[],"_links":{"self":[{"href":"https:\/\/www.kdnuggets.com\/wp-json\/wp\/v2\/posts\/93720","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.kdnuggets.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.kdnuggets.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.kdnuggets.com\/wp-json\/wp\/v2\/users\/167"}],"replies":[{"embeddable":true,"href":"https:\/\/www.kdnuggets.com\/wp-json\/wp\/v2\/comments?post=93720"}],"version-history":[{"count":0,"href":"https:\/\/www.kdnuggets.com\/wp-json\/wp\/v2\/posts\/93720\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.kdnuggets.com\/wp-json\/wp\/v2\/media?parent=93720"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.kdnuggets.com\/wp-json\/wp\/v2\/categories?post=93720"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.kdnuggets.com\/wp-json\/wp\/v2\/tags?post=93720"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}