Data Preprocessing: 6 Necessary Steps for Data Scientists by@thems18

October 27th 2020 113,787 reads

Hello everyone, I am back with another topic which is **Data Preprocessing**. This is a part of the data analytics and machine learning process that data scientists spend most of their time on. In this article, I'll dive into the topic, why we use it, and the necessary steps.

**Data preprocessing** is a **data** mining technique that involves transforming raw **data** into an understandable format. Real-world **data** is often incomplete, inconsistent, and/or lacking in certain behaviors or trends, and is likely to contain many errors. **Data preprocessing** is a proven method of resolving such issues.

In the real world **data** are generally incomplete: lacking attribute values, lacking certain attributes of interest, or containing only aggregate **data**. Noisy: containing errors or outliers. Inconsistent: containing discrepancies in codes or names.

Taken from Google Images

Machine Learning ProcessSteps in Data Preprocessing

**Step 1 :**Import the libraries**Step 2 :**Import the data-set**Step 3 :**Check out the missing values**Step 4 :**See the Categorical Values**Step 5 :**Splitting the data-set into Training and Test Set**Step 6 :**Feature Scaling

So, without wasting further time let’s get started!!!

“ New beginnings often starts with painful endings. Be thankful for those closed doors and detours, they are guiding you to a better place”

This is how we import libraries in Python using import keyword and this is the most popular libraries which any Data Scientist used. (I used- Jupyter Notebook)

**NumPy** is the fundamental package for scientific computing with Python. It contains among other things:

1. A powerful N-dimensional array object

2. Sophisticated (broadcasting) functions

3. Tools for integrating C/C++ and FORTRAN code

4. Useful linear algebra, Fourier transform, and random number capabilities

**Pandas*** is for *data manipulation and analysis*. Pandas* is an open source, BSD-licensed library providing high-performance, easy-to-use data structures and data analysis tools for the Python programming language.

*Pandas* is a NumFOCUS sponsored project. This will help ensure the success of development of *pandas* as a world-class open-source project, and makes it possible to donate to the project.**Matplotlib** is a Python 2D plotting library which produces publication quality figures in a variety of hard copy formats and interactive environments across platforms.

Matplotlib can be used in Python scripts, the Python and IPython shells, the Jupyter notebook, web application servers, and four graphical user interface toolkits.**Seaborn **is a Python data visualization library based on matplotlib.

It provides a high-level interface for drawing attractive and informative statistical graphics.

**Warning**** **messages are typically issued in situations where it is useful to alert the user of some condition in a program, where that condition (normally) doesn’t warrant raising an exception and terminating the program. For example, one might want to issue a warning when a program uses an obsolete module.

Import the Dataset

**Dataset Info**

By using Pandas we import our data-set and the file I used here is .csv file [Note: It’s not necessarily every-time you deal with **CSV** file, sometimes you deal with **Html or Xlsx(Excel file)** ].

However, to access and to use fastly we use CSV files because of their light weights. After importing the dataset, you can see we use head function ( This function returns the first n rows for the object based on position. It is useful for quickly testing if your object has the right type of data in it. By default it returns 5 rows. )

The concept of missing values is important to understand in order to successfully manage data. If the missing values are not handled properly by the researcher, then he/she may end up drawing an inaccurate inference about the data. Due to improper handling, the result obtained by the researcher will differ from ones where the missing values are present.

**Missing Values**

Yes, you have seen it write this number 3720 is your missing values in each column.

May be or May not you have read somewhere, see 7 techniques to deal with Missing Values or 5 ways to deal with Missing Values. But, I will discuss with you only those two ways which are the most prominent.

This data preprocessing method is commonly used to handle the null values. Here, we either delete a particular row if it has a null value for a particular feature and a particular column if it has more than 75% of missing values.

This method is advised only when there are enough samples in the data set. One has to make sure that after we have deleted the data, there is no addition of bias. Removing the data will lead to loss of information which will not give the expected results while predicting the output.

Drop the Missing Values

This strategy can be applied on a feature which has numeric data like the year column or Home team goal column. We can calculate the **mean, median or mode** of the feature and replace it with the missing values.

This is an approximation which can add variance to the data set. But the loss of the data can be negated by this method which yields better results compared to removal of rows and columns. Replacing with the above three approximations are a statistical approach of handling the missing values. This method is also called as **leaking the data** while training. Another way is to approximate it with the deviation of neighbouring values. This works better if the data is linear.

Replace the Missing Value

Above strategy is good for numeric data. But what happen when Categorical data has missing values?

Let’s see how to deal with Categorical Data now . . . !

See the Categorical Data in Country

Since, machine learning models are based on Mathematical equations and you can intuitively understand that it would cause some problem if we can keep the Categorical data in the equations because we would only want numbers in the equations.

So, we need to encode the Categorical Variable…..

Like in our data set Country column will cause problem, so will convert into numerical values. To convert Categorical variable into Numerical data we can use LabelEncoder() class from preprocessing library.

See the Categorical data

Use LabelEncoder class to convert Categorical data into numerical one

label_encoder is object which is I use and help us in transferring Categorical data into Numerical data. Next, I fitted this label_encoder object to the first column of our matrix X and all this return the first column country of the matrix X encoded.

But there is a problem in it, the problem is still the same, machine learning models are based on equations and that’s good that we replaced the text by numbers so that we can include the numbers in the equations.

However, since 1>0 and 2>1(See the above data-set) , the equations in the model will think that Spain has a higher value than Germany and France, and Germany has a higher value than France.

Actually, this is a not the case, these are actually three Categories and there is no relational order between the three. So , we have to prevent this, we’re going to use what are **Dummy Variables.**

**Dummy Variables** is one that takes the value 0 or 1 to indicate the absence or presence of some categorical effect that may be expected to shift the outcome.

Instead of having one column here above, we are going to have three column.

Number of Columns = Number of Categories

In our case we going to have 3 columns.

**// To create dummy variable we are going to use OneHotEncoder Class from sklearn.preprocessing or you can use pandas get dummies method.**

In above code, I already explain how to use **OneHotEnCoder **Class, now I will show you with pandas how to use **get_dummies( ) **for creating Dummy Variables.

Using Pandas to create Dummy Variables

Concatenation

See how our Dataset looks like

Sometimes, we use **KNN Imputation(for Categorical variables):** In this method of imputation, the missing values of an attribute are imputed using the given number of attributes that are most similar to the attribute whose values are missing. The similarity of two attributes is determined using a distance function.

In any Machine Learning model is that we’re going to split data-set into two separate sets

- Training Set
- Test Set

**Why we need splitting ?**

Well here it’s your algorithm model that is going to learn from your data to make predictions. Generally we split the data-set into 70:30 ratio or 80:20 what does it mean, 70 percent data take in train and 30 percent data take in test. However, this Splitting can be varies according to the data-set shape and size.

Splitting the Data-set into two set — Train and Test Set

X_train is the training part of the matrix of features.

X_test is the test part of the matrix of features.

y_train is the training part of the dependent variable that is associated to X_train here.

y_test is the test part of the dependent variable that is associated to X_train here.

Step 6 : Feature Scaling

**Feature scaling** is the method to limit the range of variables so that they can be compared on common grounds.

Suppose we have this data-set

See the Age and Salary column. You can easily noticed Salary and Age variable don’t have the same scale and this will cause some issue in your machine learning model.

Because most of the Machine Learning models are based on **Euclidean Distance**.

Euclidean Formula

Let’s say we take two values from Age and Salary column

**Age- 40 and 27**

**Salary- 72000 and 48000**

One can easily compute and see that Salary column will be dominated in Euclidean Distance. And we don’t want this thing.

So, there are several ways of scaling your data. **Reference**

Picture taken from Wikipedia

Feature Scaling

After Feature Scaling all values comes into same scale

**Note:** Sometimes machine models are not based on Euclidean distances, we will still need to do features scaling because the algorithm will converge much faster. That will be the case for Decision Tree which are not based on ED but, if we do not do feature scaling then they will run a very long time.

So that’s our end with **Data Preprocessing** phase and I hope you like this Article . . . !! Please note I have not included outliers topic in it and taken very simple data-set to make you understand everything. Please feel free to comment and give some applaud on it. If you have any problem or query in any topic related to Data Science then do let me know in the comment Section.

I’ll share more concepts soon on **LinkedIn.com**** Article column **as well as **Medium****.**

Mohit Sharma(themenyouwanttobe&Co.)[email protected]/Telegram

Join Hacker Noon

Create your free account to unlock your custom reading experience.