#### Principal Component Analysis is one of the techniques used for dimensionality reduction

In the last blog, I had talked about how you can use *Autoencoders to represent the given input to dense latent space*. Here, we will see one of the classic algorithms that is being practiced since very long and continues to deliver desirable results.

#### What are dimensions?

In machine learning, *dimensionality* simply refers to the number of features (i.e. input variables) in your dataset. Consider an example —

**Let’s say you want to predict the price of the house. Then what all parameters/features will you consider?**

*Area sq.ft**Locality**# Bedrooms**Internet speed**Distance from hospital**Distance from main market**etc .. etc*

Just now itself we are dealing with 6–7 dimensional data for just predicting the house price. So, *these things that matter while making any decisions are called dimensions*.

#### What is high dimensional data?

For the purpose of **visualization** high dimension can be any number of dimension **above 3 to 4**. Whereas, in general, I personally have found reduction working really well when visualizing the **word embeddings** which are usually in the order of few 100s.

#### Why reduce the dimensions?

- Large dimensions are difficult to train on, need
**more computational power and time**. **Visualization is not possible**with very large dimensional data.- Loading very high dimensional data can be an issue with
**limited storage space in-memory**. - It can be used to reduce the dimension of the features, potentially leading to
**better performance**for the learning algorithm by**removing redundant, obsolete**and**highly correlated features**.

#### Always Remember

As a thumb rule, you should always do **feature standardization** before applying PCA to any dataset. Standardizing the features so that they are centered around 0 with a standard deviation of 1 is not only important if we are comparing measurements that have different units, but it is also a general requirement for many machine learning algorithms. All this because, we would want all the scale of measurements to be treated on the same scale.

#### Under the hood

PCA is a **variance** maximizer. It projects the original data onto the directions where variance is maximum.

Variance is the measure of how spread out the data is.

X(i) where i in [1,2,3,4,5] are the original data points in a 2-D space. Then Z(i) where i in [1,2,3,4,5] are the projected points on a 1-D space (Line). We chose the line going from -xy to +xy (dotted one) because the data is most spread in this direction. Now, for all the points in 2-D space X(i) we map them to 1-D space/component Z(i).

#### Let’s do one

We will be using **scikit-learn** for this experiment. People who are wondering what scikit-learn it read this.

It is very easy to apply this statistical technique in python.**Thanks to the community**! ✌️ We will be dealing with a **curated dataset** just for the purpose of this snippet.

```
from sklearn.decomposition import PCA
import matplotlib.pyplot as plt
from sklearn.preprocessing import scale
```

`data = open('custom_embed.csv', 'rb') `

```
labels, dimensions = [], []
for line in data:
line = line.split(",")
lab = line[0].strip()
dim = [float(j.strip()) for j in line[1].split()]
labels.append(lab)
dimensions.append(dim)
```

```
# scaling the values
X = scale(dimensions)
```

```
pca = PCA(n_components=2)
pca.fit(X)
X1=pca.fit_transform(X)
X1 = X1.tolist()
```

```
# plotting
x = [i[0] for i in X1]
y = [i[1] for i in X1]
n = ['king', 'school', 'university', 'man', 'emperor']
```

```
fig, ax = plt.subplots()
ax.scatter(x, y)
for i, txt in enumerate(n):
ax.annotate(txt, (x[i],y[i]))
```

`plt.show()`

It seems that we are successful in preserving the semantic properties of words as and when they are used even after reducing the dimension space from 300 to 2.

I have tried to keep this blog simple and intuitive as much possible. For in-depth details of the algorithms **see this**

*Feel free to comment and share your thoughts. Do **share** and **clap** if you ❤ it.*