“You don’t need a silver fork to eat good food.”
------------------------------------------------

### 🎬 Introduction Wine Reviews

![](https://hackernoon.com/hn-images/1*n8wiHEofEfPvuDbfNdE4Ag.png)

In this article, I will try to explore the Wine Reviews Dataset. It contains 130k of reviews in Wine Reviews. And at the end of this article, I will try to make simple text summarizer that will summarize given reviews. The summarized reviews can be used as a reviews title also.I will use spaCy as natural language processing library for handling this project.

### 📋 Object Of This Project

The objective of this project is to build a model that can create relevant summaries for reviews written on Wine reviews. This dataset contains above 130k reviews, and is hosted on [Kaggle](https://www.kaggle.com/zynicide/wine-reviews).

### What Is Text Summarization?

![](https://hackernoon.com/hn-images/1*4d7oCMtJbCcL6NU0ufCODw.png)

> Text summarization is the process of distilling the most important information from a source (or sources) to produce an abridged version for a particular user (or users) and task (or tasks).

### Why we need text summarization?

![](https://hackernoon.com/hn-images/1*RgiCt7qBf4JZAmsqA1vRug.png)

[Imgur](https://imgur.com/HyPUqwF)

In Recent Years we are witnessing the amount of textual information is increasing day by day .The Textual Information grows rapidly. It becomes more difficult for the user to read the textual information and also it leads to loss of interest. That is the reason why Text Summarization came into picture which will solve this problem.

### Types of Text Summarization Methods

Text summarization methods can be classified into different types.

![](https://hackernoon.com/hn-images/1*pMwBqRkk7agz30d3IxZ8YA.png)

**i. Based on input type:**

1.  Single Document, where the input length is short. Many of the early summarization systems dealt with single document summarization.
2.  Multi Document, where the input can be arbitrarily long.

**ii. Based on the purpose:**

1.  Generic, where the model makes no assumptions about the domain or content of the text to be summarized and treats all inputs as homogeneous. The majority of the work that has been done revolves around generic summarization.
2.  Domain-specific, where the model uses domain-specific knowledge to form a more accurate summary. For example, summarizing research papers of a specific domain, biomedical documents, etc.
3.  Query-based, where the summary only contains information which answers natural language questions about the input text.

**iii. Based on output type:**

1.  Extractive, where important sentences are selected from the input text to form a summary. Most summarization approaches today are extractive in nature.
2.  Abstractive, where the model forms its own phrases and sentences to offer a more coherent summary, like what a human would generate. This approach is definitely a more appealing, but much more difficult than extractive summarization.

### Prerequisites

This article makes the following assumptions:

*   You are familar with Python
*   You have Python 3.6 or greater installed on your system
*   spaCy package.

### What is spaCy?

spaCy is a relatively new package for “Industrial strength NLP in Python” developed by Matt Honnibal at [explosion.ai](https://explosion.ai/). It is designed with the applied data scientist in mind, meaning it does not weigh the user down with decisions over what esoteric algorithms to use for common tasks and it’s fast. Incredibly fast (it’s implemented in Cython). If you are familiar with the Python data science stack, spaCy is your `numpy` for NLP – it’s reasonably low-level, but very intuitive and performant.However, since SpaCy is a relative new NLP library, and it’s not as widely adopted as NLTK.

### Installation of spaCy

`**spaCy**`, its data, and its models can be easily installed using python package index and setup tools. Use the following command to install spacy in your machine:

! pip install spacy

In case of Python3, replace “pip” with “pip3” in the above command.

OR download the source from [here](https://pypi.org/project/spacy/) and run the following command, after unzipping:

!python setup.py install

To download all the data and models, run the following command, after the installation:

!python -m spacy.en.download all

You are now all set to explore and use spacy.

### Loading spaCy Libraries

![](https://hackernoon.com/hn-images/1*AxrO0hE3IG2nl_m0P2mk5g.png)

import spacy

### Implementation Section

### 1\. Import Packages

import numpy as np _\# linear algebra_  
import spacy  
nlp = spacy.load('en\_core\_web\_sm')  
import pandas as pd _\# data processing, CSV file I/O (e.g. pd.read\_csv)_  
import seaborn as sns  
import matplotlib.pyplot as plt  
from wordcloud import WordCloud  
import string  
import re  
from collections import Counter  
from time import time  
_\# from sklearn.feature\_extraction.stop\_words import ENGLISH\_STOP\_WORDS as stopwords_  
from nltk.corpus import stopwords  
import nltk  
import plotly.offline as py  
import plotly.graph\_objs as go  
import plotly.tools as tls  
%matplotlib inline  
  
stopwords = stopwords.words('english')  
sns.set\_context('notebook')

### 2\. Import Dataset

In this section, I will load the desired dataset for this notebook. This dataset has huge number of reviews. It will be hard to work with full dataset. So I will randomly sample the dataset into smaller chunks for easy purpose.

reviews = pd.read\_csv("../input/winemag-data-130k-v2.csv", nrows=5000,usecols =\['points', 'title', 'description'\],encoding='latin1')  
reviews = reviews.dropna()  
reviews.head(15)

![](https://hackernoon.com/hn-images/1*LKsgBLyTmpGyi_ZV7V2zCA.png)

### 3\. Text preprocessing

In this step, I will be using Spacy for preprocessing text, in others words I will clearing not useful features from reviews title like punctuation, stopwords. For this task, there are two useful libraries available in Python. 1. NLTK 2. Spacy. In this notebook, I will be working with Spacy because it is very fast and has many useful features compared to NLTK. So without further do let’s get started!

!python -m spacy download en\_core\_web\_lg  
nlp = spacy.load('en\_core\_web\_lg')  
def normalize\_text(text):  
    tm1 = re.sub('<pre>.\*?</pre>', '', text, flags=re.DOTALL)  
    tm2 = re.sub('<code>.\*?</code>', '', tm1, flags=re.DOTALL)  
    tm3 = re.sub('<\[^>\]+>©', '', tm1, flags=re.DOTALL)  
    return tm3.replace("**\\n**", "")

Output screen:

![](https://hackernoon.com/hn-images/1*UwGzf7d_5XQQigBkSvnVOg.png)

_\# in this step we are going to remove code syntax from text_   
reviews\['description\_Cleaned\_1'\] = reviews\['description'\].apply(normalize\_text)

print('Before normalizing text-----**\\n**')  
print(reviews\['description'\]\[2\])  
print('**\\n**After normalizing text-----**\\n**')  
print(reviews\['description\_Cleaned\_1'\]\[2\])

Output screen:

![](https://hackernoon.com/hn-images/1*vtzrh7XYwJgVrS_ePubHIQ.png)

We can see a huge difference after normalizing our text. Now we can see our text is more manageable. This will help us to explore the reviews and later making summarizer.

We are also seeing that there are some punctuation and stopwords. We also don’t need them. In the first place, I don’t remove them because we are gonna need this in future when we will make summarizer. So let’s make another column that will store our normalized text without punctuation and stopwords.

### 3.1 Clean text before feeding it to spaCy

punctuations = '!"#$%&**\\'**()\*+,-/:;<=>?@\[**\\\\**\]^\_\`{|}~©'  
_\# Define function to cleanup text by removing personal pronouns, stopwords, and puncuation_  
def cleanup\_text(docs, logging=False):  
    texts = \[\]  
    doc = nlp(docs, disable=\['parser', 'ner'\])  
    tokens = \[tok.lemma\_.lower().strip() for tok **in** doc if tok.lemma\_ != '-PRON-'\]  
    tokens = \[tok for tok **in** tokens if tok **not** **in** stopwords **and** tok **not** **in** punctuations\]  
    tokens = ' '.join(tokens)  
    texts.append(tokens)  
    return pd.Series(texts)  
reviews\['Description\_Cleaned'\] = reviews\['description\_Cleaned\_1'\].apply(lambda x: cleanup\_text(x, False))

print('Reviews description with punctuatin and stopwords---**\\n**')  
print(reviews\['description\_Cleaned\_1'\]\[0\])  
print('**\\n**Reviews description after removing punctuation and stopwrods---**\\n**')  
print(reviews\['Description\_Cleaned'\]\[0\])

Output screen:

![](https://hackernoon.com/hn-images/1*eBU7ikZJIpFraDXUTm50pQ.png)

Wow! See! Now our text looks much readable and less messy!

### 4\. Distribution of Points

In this section, I will try understand the distribution of points. Here points mean number of upvote the description got in social media(such as facebook,twitter etc).

plt.subplot(1, 2, 1)  
(reviews\['points'\]).plot.hist(bins=30, figsize=(30,5), edgecolor='white',range=\[0,150\])  
plt.xlabel('Number of points', fontsize=17)  
plt.ylabel('frequency', fontsize=17)  
plt.tick\_params(labelsize=15)  
plt.title('Number of points description', fontsize=17)  
plt.show()

Output screen:

![](https://hackernoon.com/hn-images/1*pnQOIIVSUSLfb4jiqDB_9g.png)

The description of points lies between 80 to 100 mostly. Majority of the description got points between 80 to 100.

### 5\. Analyze reviews description

In this section, I will try to analyze wine description. In Wine Reviews, the wine description plays a vital role. A good description can make your wine stand out. It also helps get a reviews faster. Lastly, It will help you get some points. Let’s see what we can find in the wine description.

reviews\['Title\_len'\] = reviews\['Description\_Cleaned'\].str.split().str.len()  
rev = reviews.groupby('Title\_len')\['points'\].mean().reset\_index()  
trace1 = go.Scatter(  
    x = rev\['Title\_len'\],  
    y = rev\['points'\],  
    mode = 'lines+markers',  
    name = 'lines+markers'  
)  
layout = dict(title= 'Average points by wine description Length',  
              yaxis = dict(title='Average points'),  
              xaxis = dict(title='wine description Length'))  
fig=dict(data=\[trace1\], layout=layout)  
py.iplot(fig)

Output screen:

![](https://hackernoon.com/hn-images/1*kgs8GZjoeeRXh11veiS7NQ.png)

### 6\. Description Summarizer

![](https://hackernoon.com/hn-images/1*nQF_558CKU4TGcL3DLV-4g.jpeg)

[Siraj Raval](https://www.youtube.com/watch?v=ogrJaOIuBx4&t=164s)

> In this step, I will try to make a description summarizer. There is a huge amount of research going for text summarization. But I will try to do a simple technique for text summarization. The technique describes below.

### 6.1 Convert Paragraphs to Sentences

> _We first need to convert the whole paragraph into sentences. The most common way of converting paragraphs to sentences is to split the paragraph whenever a period is encountered._

### 6.2 Text Preprocessing

> _After converting paragraph to sentences, we need to remove all the special characters, stop words and numbers from all the sentences._

### 6.3 Tokenizing the Sentences

> _We need to tokenize all the sentences to get all the words that exist in the sentences_

### 6.4 Find Weighted Frequency of Occurrence

> _Next we need to find the weighted frequency of occurrences of all the words. We can find the weighted frequency of each word by dividing its frequency by the frequency of the most occurring word._

### 6.5 Replace Words by Weighted Frequency in Original Sentences

> _The final step is to plug the weighted frequency in place of the corresponding words in original sentences and finding their sum. It is important to mention that weighted frequency for the words removed during preprocessing (stop words, punctuation, digits etc.) will be zero and therefore is not required to be added_

### 6.6 Sort Sentences in Descending Order of Sum

> _The final step is to sort the sentences in inverse order of their sum. The sentences with highest frequencies summarize the text._

### Function for text summarization:

This function help for summarization from big text.So we need this function all time when we want to summarization from text.The function below here:

def generate\_summary(text\_without\_removing\_dot, cleaned\_text):  
    sample\_text = text\_without\_removing\_dot  
    doc = nlp(sample\_text)  
    sentence\_list=\[\]  
    for idx, sentence **in** enumerate(doc.sents): _\# we are using spacy for sentence tokenization_  
        sentence\_list.append(re.sub(r'\[^\\w\\s\]','',str(sentence)))  
  
    stopwords = nltk.corpus.stopwords.words('english')  
  
    word\_frequencies = {}    
    for word **in** nltk.word\_tokenize(cleaned\_text):    
        if word **not** **in** stopwords:  
            if word **not** **in** word\_frequencies.keys():  
                word\_frequencies\[word\] = 1  
            else:  
                word\_frequencies\[word\] += 1  
  
  
    maximum\_frequncy = max(word\_frequencies.values())  
  
    for word **in** word\_frequencies.keys():    
        word\_frequencies\[word\] = (word\_frequencies\[word\]/maximum\_frequncy)  
  
  
    sentence\_scores = {}    
    for sent **in** sentence\_list:    
        for word **in** nltk.word\_tokenize(sent.lower()):  
            if word **in** word\_frequencies.keys():  
                if len(sent.split(' ')) < 30:  
                    if sent **not** **in** sentence\_scores.keys():  
                        sentence\_scores\[sent\] = word\_frequencies\[word\]  
                    else:  
                        sentence\_scores\[sent\] += word\_frequencies\[word\]  
  
  
    summary\_sentences = heapq.nlargest(7, sentence\_scores, key=sentence\_scores.get)  
  
    summary = ' '.join(summary\_sentences)  
    print("Original Text:**\\n**")  
    print(text\_without\_removing\_dot)  
    print('**\\n\\n**Summarized text:**\\n**')  
    print(summary)

Now we have written the function let’s try to summarize some descriptions.

generate\_summary(reviews\['description\_Cleaned\_1'\]\[8\], reviews\['Description\_Cleaned'\]\[8\])

Output screen:

![](https://hackernoon.com/hn-images/1*tp9ZkWmqkZEIyQPQWGJFcQ.png)

generate\_summary(reviews\['description\_Cleaned\_1'\]\[100\], reviews\['Description\_Cleaned'\]\[100\])

Output screen:

![](https://hackernoon.com/hn-images/1*rJo5LUCVcYaFeE8YPzZ1jg.png)

generate\_summary(reviews\['description\_Cleaned\_1'\]\[500\], reviews\['Description\_Cleaned'\]\[500\])

Output screen:

![](https://hackernoon.com/hn-images/1*MmJIEseHLbQF0gTPa269tw.png)

That’s awesome! We successfully made a simple winemag description summarizer.

### 7\. Conclusion

Thanks for reading this article. If you have any suggestion feel free to reach me in the comment or sent [mail](http://harunspeedy1995@gmail.com) or connect on [LinkedIn](https://www.linkedin.com/in/harun-ur-rashid6647/). Stay in touch for more update. Thank you. 😎

For the full code visit [Kaggle](https://www.kaggle.com/harunshimanto/summarization-with-wine-reviews-using-spacy) .

> “Let us celebrate the occasion with wine and sweet words.”

If you like this article then give 👏 clap. Happy Analysis!

Facebook

Twitter

YouTube

Too Long; Didn't Read

Get hands-on learning from ML experts on Coursera 

📚 Summarization With Wine Reviews Using spaCy📋

Too Long; Didn't Read

Companies Mentioned

@harunshimanto

RELATED STORIES