Dare Drop

daredrop

A Machine Learning Text Classification Case Study with a Product-driven Twist

একটি পণ্য-চালিত টুইস্ট সহ একটি মেশিন লার্নিং পাঠ্য শ্রেণিবিন্যাস কেস স্টাডি

Eine Fallstudie zur Textklassifizierung durch maschinelles Lernen mit einer produktgesteuerten Wendung

Un estudio de caso de clasificación de textos con aprendizaje automático con un toque impulsado por el producto

Une étude de cas de classification de textes d'apprentissage automatique avec une touche axée sur le produit

उत्पाद-संचालित ट्विस्ट के साथ एक मशीन लर्निंग टेक्स्ट वर्गीकरण केस स्टडी

製品主導型のひねりを加えた機械学習テキスト分類のケーススタディ

제품 중심 트위스트를 사용한 기계 학습 텍스트 분류 사례 연구

Um estudo de caso de classificação de texto de aprendizado de máquina com um toque orientado ao produto

Пример классификации текста с помощью машинного обучения с учетом особенностей продукта

Ürün Odaklı Bir Yönlü Makine Öğrenimi Metin Sınıflandırması Örnek Olay İncelemesi

Nghiên cứu điển hình về phân loại văn bản học máy với khuynh hướng hướng đến sản phẩm

产品驱动的机器学习文本分类案例研究

How LLMs Have Transformed Working with Computers

Vector Databases - Basics of Vector Search and Langchain Package in Python

ভেক্টর ডেটাবেস - পাইথনে ভেক্টর অনুসন্ধান এবং ল্যাংচেইন প্যাকেজের মূল বিষয়গুলি

Vektordatenbanken – Grundlagen der Vektorsuche und des Langchain-Pakets in Python

Bases de datos vectoriales: conceptos básicos de la búsqueda vectorial y el paquete Langchain en Python

Bases de données vectorielles - Bases de la recherche de vecteurs et du package Langchain en Python

वेक्टर डेटाबेस - पायथन में वेक्टर खोज और लैंगचैन पैकेज की मूल बातें

ベクトル データベース - Python のベクトル検索とラングチェーン パッケージの基礎

벡터 데이터베이스 - Python의 벡터 검색 및 Langchain 패키지 기본 사항

Bancos de dados vetoriais - Noções básicas de pesquisa vetorial e pacote Langchain em Python

Векторные базы данных — основы векторного поиска и пакет Langchain в Python

Vektör Veritabanları - Python'da Vektör Arama ve Langchain Paketinin Temelleri

Cơ sở dữ liệu Vector - Khái niệm cơ bản về Tìm kiếm Vector và Gói Langchain trong Python

矢量数据库 - Python 中矢量搜索和 Langchain 包的基础知识

In the realm of machine learning and data science, feature engineering stands as a cornerstone of model performance improvement. While algorithms and models often grab the spotlight, it's the carefully crafted features that lay the foundation for predictive power. Imagine trying to build a house on a weak foundation – without strong features, even the most advanced algorithms may crumble when faced with complex datasets. Understanding Feature Engineering Feature engineering is the creative and strategic process of selecting, transforming, and creating features (input variables) from raw data to maximize a model's performance. It's the art of converting data into actionable insights, giving machines the contextual understanding they need to make accurate predictions. Whether it's predicting house prices, classifying customer preferences, or diagnosing medical conditions, feature engineering holds the key to unlocking the potential hidden within your data. The Importance of Careful Craftsmanship Imagine you're tasked with predicting house prices based on a variety of factors. You might initially be drawn to the obvious features like square footage, number of bedrooms, and location. However, the magic of feature engineering lies in unearthing the subtler aspects that influence the target variable. Does the ratio of bathrooms to bedrooms impact the price? What about the presence of a fireplace or the age of the roof? Feature engineering isn't merely about creating more features; it's about discerning which facets of the data truly matter. This process often requires domain knowledge, creativity, and a deep understanding of the problem you're tackling. By refining existing features and crafting new ones, you're essentially teaching your model to understand the data like an expert. Why should you care about feature engineering? The answer lies in model performance. Well-engineered features can lead to faster convergence during training, reduced overfitting, and ultimately, more accurate predictions. A machine learning model is only as good as the data it's fed, and well-engineered features provide a richer, more nuanced representation of that data. In this article, we'll delve into the world of feature engineering using the advanced house price prediction dataset from Kaggle. By following along, you'll gain insights into various techniques that can transform raw data into valuable predictors and you’ll see how the results of your model improve by applying different feature engineering methods. So, without wasting anytime, let’s start learning about the different methods of feature engineering. Methods Now that we have set the stage, it's time to delve into the exciting world of advanced feature engineering techniques. In this section, I'll guide you through the step-by-step implementation of four powerful methods that can supercharge your predictive models. Each method serves a unique purpose, offering insights and improvements that can make a substantial difference in your model's performance. Method 1: Mutual Information - Extracting Information from Relationships Imagine being able to select the most influential features for your model with surgical precision. Mutual Information allows you to achieve just that. By quantifying the relationship between each feature and the target variable, you can identify the key factors that impact your predictions. We'll walk you through the code and provide a detailed explanation of every step, helping you master this insightful technique. Method 2: Clustering - Discovering Patterns through Grouping Clusters often hide valuable patterns within your data. With clustering, you can uncover these hidden gems and leverage them to enhance your model's understanding. I'll guide you through the process of applying KMeans clustering to group similar instances together. You'll learn how to create new features based on these clusters and observe their impact on model performance. Method 3: PCA Dimensionality Reduction - Condensing Complexity High-dimensional data can be overwhelming, but Principal Component Analysis (PCA) offers a solution. By identifying the most influential dimensions, you can reduce complexity without sacrificing predictive power. This tutorial will lead you through the transformation of your data using PCA, providing insights into how this technique can streamline your model while retaining its accuracy. In addition to these methods, I'll also introduce you to —an often overlooked technique that can wield powerful results. By applying mathematical operations to selected columns, you can shape your data to better align with your model's assumptions. You'll explore logarithmic, square root, and inverse transformations, illustrating how they can uncover hidden relationships and boost your model's accuracy. Mathematical Transformations Throughout this section, I'll offer comprehensive explanations to ensure that you not only grasp the technical aspects but also understand the reasoning behind each method's application. By the end, you'll have gained a valuable toolkit of advanced feature engineering techniques that can be confidently applied to enhance your predictive models in various scenarios. Are you ready to explore the intricacies of each method, uncover their potential, and equip yourself with the expertise to engineer features that translate to more accurate and powerful predictive models? Let's begin this enlightening journey! Code Importing Packages Let’s first import all the packages that you need. import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error
from sklearn.feature_selection import SelectKBest, mutual_info_regression
from sklearn.cluster import KMeans
from sklearn.decomposition import PCA
from sklearn.preprocessing import LabelEncoder, StandardScaler
import numpy as np Getting The Data Now, let’s get the dataset from kaggle and preprocess it. # Load the dataset
data = pd.read_csv('train.csv')

# Data preprocessing
def preprocess_data(df):
    # Handle missing values
    df = df.fillna(df.median())

    # Handle categorical variables
    df = pd.get_dummies(df, drop_first=True)

    return df

data = preprocess_data(data)

# Prepare the data
X = data.drop('SalePrice', axis=1)
y = data['SalePrice']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42) To start, let's run a baseline model without any feature engineering and assess its accuracy. After that, we'll apply each of the four feature engineering methods (mutual information, clustering, PCA dimensionality reduction, and label encoding) individually and compare their effects on model performance. Base Line Model Before applying any feature engineering, we'll start with a baseline model. Here, we'll use a simple linear regression model to predict house prices using the original dataset. baseline_model = LinearRegression()
baseline_model.fit(X_train, y_train)
baseline_predictions = baseline_model.predict(X_test)
baseline_rmse = mean_squared_error(y_test, baseline_predictions, squared=False)
print(f"Baseline RMSE: {baseline_rmse}")

# ------------------ OUTPUT ---------------- #
Baseline RMSE: 49204.92 In the baseline model, you're starting with a simple linear regression model that uses the original features as they are. You're training the model using the training data, making predictions on the test data, and calculating the root mean squared error (RMSE) to measure how well the model performs on unseen data. Mutual Information Model # Method 1: Mutual Information
mi_selector = SelectKBest(score_func=mutual_info_regression, k=10)
X_train_mi = mi_selector.fit_transform(X_train, y_train)
X_test_mi = mi_selector.transform(X_test)
mi_model = LinearRegression()
mi_model.fit(X_train_mi, y_train)
mi_predictions = mi_model.predict(X_test_mi)
mi_rmse = mean_squared_error(y_test, mi_predictions, squared=False)
print(f"Mutual Information RMSE: {mi_rmse}")

# ------------------ OUTPUT ----------------- #
Mutual Information RMSE: 39410.99 Here, you're exploring the information each feature provides about the target variable. You select the top 10 features that have the highest mutual information scores with the target. Then, you train a new linear regression model using only these selected features. This helps ensure that your model focuses on the most informative features, and you calculate the RMSE to see how this model's predictions compare to the baseline. Clustering Model # Method 2: Clustering
num_clusters = 5
kmeans = KMeans(n_clusters=num_clusters, random_state=42)
X_clustered = X.copy()
X_clustered['Cluster'] = kmeans.fit_predict(X)
X_clustered = pd.get_dummies(X_clustered, columns=['Cluster'], prefix='Cluster')
X_train_clustered, X_test_clustered, _, _ = train_test_split(X_clustered, y, test_size=0.2, random_state=42)
cluster_model = LinearRegression()
cluster_model.fit(X_train_clustered, y_train)
cluster_predictions = cluster_model.predict(X_test_clustered)
cluster_rmse = mean_squared_error(y_test, cluster_predictions, squared=False)
print(f"Clustering RMSE: {cluster_rmse}")

# ------------------- OUTPUT -------------- #
Clustering RMSE: 47715.30 You're looking at grouping similar instances in your data using clustering. Specifically, you use the KMeans algorithm to divide your data into clusters. Each instance is assigned to a cluster, and you add this cluster information as a new categorical feature. By doing this, you're giving the model a way to consider the relationships between instances in terms of their clusters. After training a linear regression model on this clustered data, you calculate the RMSE to evaluate its performance. PCA - Dimensionality Reduction Model # Method 3: PCA Dimensionality Reduction
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X_train)
pca = PCA(n_components=10)
X_train_pca = pca.fit_transform(X_scaled)
X_test_pca = pca.transform(scaler.transform(X_test))
pca_model = LinearRegression()
pca_model.fit(X_train_pca, y_train)
pca_predictions = pca_model.predict(X_test_pca)
pca_rmse = mean_squared_error(y_test, pca_predictions, squared=False)
print(f"PCA RMSE: {pca_rmse}")

# --------------- OUTPUT -------------- #
PCA RMSE: 40055.78 PCA helps you reduce the complexity of your data by summarizing it into fewer dimensions. You standardize your data to make sure all features are on the same scale. Then, you use PCA to identify the most important patterns in your data and reduce the number of features to 10 principal components. Training a linear regression model on these components, you're able to capture the most significant information while simplifying the model. The RMSE helps you gauge if this approach is effective. Mathematical Transformations Model # Method 5: Mathematical Transformations
X_train_transformed = X_train.copy()
X_test_transformed = X_test.copy()

# Apply logarithmic transformation
log_columns = ['GrLivArea', 'LotArea']  # Specify columns to apply log transformation
X_train_transformed[log_columns] = np.log1p(X_train_transformed[log_columns])
X_test_transformed[log_columns] = np.log1p(X_test_transformed[log_columns])

# Apply square root transformation
sqrt_columns = ['GarageArea', '1stFlrSF']  # Specify columns to apply square root transformation
X_train_transformed[sqrt_columns] = np.sqrt(X_train_transformed[sqrt_columns])
X_test_transformed[sqrt_columns] = np.sqrt(X_test_transformed[sqrt_columns])

# Apply inverse transformation
inv_columns = ['YearBuilt', 'OverallQual']  # Specify columns to apply inverse transformation
X_train_transformed[inv_columns] = 1 / X_train_transformed[inv_columns]
X_test_transformed[inv_columns] = 1 / X_test_transformed[inv_columns]

math_transform_model = LinearRegression()
math_transform_model.fit(X_train_transformed, y_train)
math_transform_predictions = math_transform_model.predict(X_test_transformed)
math_transform_rmse = mean_squared_error(y_test, math_transform_predictions, squared=False)
print(f"Mathematical Transformations RMSE: {math_transform_rmse}")

# ------------------ OUTPUT --------------- #
Mathematical Transformations RMSE: 47143.21 Mathematical transformations involve altering feature values using mathematical operations to bring out underlying patterns. You apply logarithmic, square root, and inverse transformations to specific columns. For example, logarithmic transformation helps normalize skewed data, square root transformation can help with outliers, and inverse transformation can emphasize relationships with small values. You train a linear regression model using these transformed features and calculate the RMSE to assess whether the transformations have improved the model's predictive power. In all these methods, you're experimenting with different techniques to make your data more suitable for modeling. The goal is to find the method that leads to the lowest RMSE, indicating that your model's predictions are closer to the actual target values and hence more accurate. Comparing Results This is the root mean squared error (RMSE) of the baseline model, where no feature engineering or transformation has been applied. The model uses the original features as they are. An RMSE of 49204.92 indicates the average prediction error of the baseline model on the test data. Baseline RMSE: 49204.92 This RMSE represents the performance of the model after applying the mutual information feature selection method. It's significantly lower than the baseline RMSE, indicating that selecting the top k features based on their mutual information scores has led to improved model performance. Mutual Information RMSE: 39410.99 The RMSE here corresponds to the model's performance after introducing a new categorical feature based on clustering. The RMSE is close to the baseline RMSE, suggesting that the introduction of clustering did not lead to a significant improvement in this case. Clustering RMSE: 47715.30 This RMSE reflects the performance of the model after applying PCA for dimensionality reduction. It's a bit higher than the mutual information RMSE but lower than the baseline RMSE. The model using PCA-transformed features seems to perform moderately well. PCA RMSE: 40055.78 The RMSE here shows the performance of the model when categorical variables are label encoded. The RMSE matches the baseline RMSE, indicating that using label encoded features didn't lead to a noticeable improvement in this case. Label Encoding RMSE: 49204.92 This RMSE represents the performance of the model after applying various mathematical transformations to selected columns. The RMSE is lower than the baseline RMSE, suggesting that these transformations have led to improved model performance. Mathematical Transformations RMSE: 47143.21 In summary: seems to be the most effective feature selection method among the methods tried, as it significantly reduced the RMSE. Mutual Information and both resulted in improved model performance compared to the baseline. PCA Mathematical Transformations did not show a significant improvement in this particular scenario. Clustering Keep in mind that the actual RMSE values and their interpretation depend on various factors such as the dataset, the complexity of the model, and the nature of the target variable. The goal is to experiment with different feature engineering methods and select the one that leads to the best performance on unseen data.

The Beginner's Guide to Feature Engineering

La guía para principiantes de ingeniería de características

Le guide du débutant pour l'ingénierie des fonctionnalités

फ़ीचर इंजीनियरिंग के लिए शुरुआती मार्गदर्शिका

特徴量エンジニアリングの初心者ガイド

O guia para iniciantes em engenharia de recursos

Hướng dẫn dành cho người mới bắt đầu về Kỹ thuật tính năng

特征工程初学者指南

Mobile Price Classification: An Open Source Data Science Project with Dagshub

We cannot solve our problems with the same thinking we used to create them -Albert Einstein If you have ever built a Machine Learning model, you've probably felt, Model deployment is a part of Machine Learning which isn't paid as much attention as it should. well, this was cool, but how will other people be able to see how cool it is? Well, in this article, I will introduce you to a new tool that will help you generate a web app for your Machine Learning model, which you can, of course, share with other individuals to try out. I will be building a simple neural network model using scikit-learn (not TF, sorry😒) and creating a GUI for the model using Gradio (this is the new cool tool I spoke about). Let us Begin What is Gradio? According to the , Gradio allows you to quickly create customizable UI components around your TensorFlow or PyTorch models or even arbitrary Python functions. Not terribly informative ei😅. Gradio website If you have ever used a python GUI library like Tkinter, Gradio is like that. Gradio is a GUI library that allows you to create customizable GUI components for your Machine Learning model. Pre-requisite For you to successfully work through this tutorial You must have python installed Let's Get Building You can check out the GitHub repo for the project here Let us begin Install required packages Packages Let us install the required packages pip install sklearn pip install pandas pip install numpy pip install gradio Getting our data Our data is going to be in the. CSV format. You can get the data by clicking here Importing Packages We are going to import the required packages import numpy as np

import pandas as pd

import gradio as gr Next, we are going to filter the warnings so we don't see them. import warnings

warnings.filterwarnings('ignore') Import the data Next, we are going to import our data data = pd.read_csv('diabetes.csv') Now let's see a little preview of our dataset. data.head() Let us see the feature columns in our dataset. print (data.columns) Getting our Variables Next, we get our X and Y variables x = data.drop(['Outcome'], axis=1)

y = data['Outcome'] Splitting the data Now we are going to split our data using the sci-kit-learn's inbuilt function. train_test_split from sklearn.model_selection import train_test_split

x_train, x_test, y_train, y_test = train_test_split(x,y) Scaling our data Next, we are going to scale our data using sci-kit-learn's inbuilt object. StandardScaler from sklearn.preprocessing import StandardScaler

#instantiate StandardScaler object
scaler = StandardScaler()

#scale data
x_train_scaled = scaler.fit_transform(x_train)

x_test_scaled = scaler.fit_transform(x_test) In the code above, we scaled our data using the StandardScaler object made available to us through sci-kit-learn. To know more about Scaling and why it is done, click here Instantiating and training the model In this section, we are going to create and train our model. The model we are going to make us of will be a Multi-Layer Perceptron Classifier, a neural network built into scikit-learn. #import model object
from sklearn.neural_network import MLPClassifier
model =  MLPClassifier(max_iter=1000,  alpha=1)

#train model on training data
model.fit(x_train_scaled, y_train)

#getting model performance on test data
print("accuracy:", model.score(x_test_scaled, y_test)) Creating the function for Gradio Now is the fun part. Here we are going to create a function that will take in the features of the data set which our model was trained on and pass it as an array to our model to predict. Then we are going to build our gradio web app based on that function. To understand why we have to write a function, you must first understand that gradio builds GUI components for our Machine Learning model based on the function. The function provides a way for gradio to get input from users and pass it on to the ML model, which will then process it and then pass it back to gradio, which then passes the result out. Let's Write... First, we will get the feature columns which we will then pass onto our function. #geting our columns

print(data.columns) Now we will create our function. def diabetes(Pregnancies, Glucose, Blood_Pressure, SkinThickness, Insulin, BMI, Diabetes_Pedigree, Age):
#turning the arguments into a numpy array  

 x = np.array([Pregnancies,Glucose,Blood_Pressure,SkinThickness,Insulin,BMI,Diabetes_Pedigree,Age])

  prediction = model.predict(x.reshape(1, -1))

  return prediction In the code above, we passed the feature columns from our data as arguments into a function we named . Then we turned the arguments into a NumPy array which we then passed onto our model for prediction, and finally, we returned the predicted result of our model. diabetes Creating our Gradio Interface Now we are going to create our Web App interface using Gradio. outputs = gr.outputs.Textbox()

app = gr.Interface(fn=diabetes, inputs=['number','number','number','number','number','number','number','number'], outputs=outputs,description="This is a diabetes model") The first thing we did above was to create a variable named outputs, which hold the GUI component for our model result. The result of our model's prediction will be outputted in a text box. Then we instantiated the Gradio interface object and passed in our earlier function. Then we generated our Inputs GUI component, and then we told the radio to expect 8 inputs in form of numbers. diabetes The inputs represent the feature columns that are present in our dataset, the same 8 feature column names we passed into our function. diabetes Then we passed our earlier output variable into the outputs parameter present in the object. Finally, we passed in the description of our web app into the description parameter. Launching the Gradio Web App Now we're going to Launch our Gradio web app. app.launch() If you are launching the Gradio app as a script from that command line, you will be given a local host link that you will copy and paste in our browser to see your web app. NOTE: If you are launching the app from a jupyter notebook, you will see a live preview of the app as you run the cell, and you will also be provided with a link. Hosting and sharing your Web App If you want to share your web app, all you have to do is put in as a parameter in your launch object. share=True #To provide a shareable link
app.launch(share=True) A link will then be provided for you with a .gradio extension. But this shareable link lasts for only 24 hours and will last if only your system is running. Because Gradio just hosts the web app on your system. In simple words, for your link to work, your system has to be on because gradio uses your system to host the web app and once your system is off. The server connection is severed, and you get a 500😅. Luckily for us, Gradio also provides a way for you to permanently host your model. But the service is subscription-based, so you have to pay $7 monthly to access it. Permanent hosting is way out of the scope o this article (partly because the author is broke😅). But if you are interested in it, click . here Important Links Gradio Website Gradio Documentation Gradio on GitHub Summary The Gradio Library is really cool, and it helps solve a huge problem plaguing the Machine Learning Community, Model Deployment. About 90% of Machine Learning models built are not deployed, and Gradio is working to fix that. It also serves as a way for beginners and experts to show off their models and also test the models in real life. You can't go wrong with the Gradio Library. Also Published Here

Build a GUI for Your Machine Learning Models

Plagiarism is rampant on the internet and in the classroom. With so much content out there, it’s sometimes hard to know when something has been plagiarized. Authors writing blog posts may want to check if someone has stolen their work and posted it elsewhere. Teachers may want to check students’ papers against other scholarly articles for copied work. News outlets may want to check if a content farm has stolen their news articles and claimed the content as its own. So, how do we guard against plagiarism? Wouldn’t it be nice if we could have software do the heavy lifting for us? Using machine learning, we can build our own plagiarism checker that searches a vast database for stolen content. In this article, we’ll do exactly that. We’ll build a Python Flask app that uses — a similarity search service — to find possibly plagiarized content. Pinecone Demo App Overview Let’s take a look at the demo app we’ll be building today. Below, you can see a brief animation of the app in action. The UI features a simple text area input in which the user can paste the text from an article. When the user clicks the button, this input is used to query a database of articles. Results and their match scores are then displayed to the user. To help reduce the amount of noise, the app also includes a slider input in which the user can specify a similarity threshold to only show extremely strong matches. Submit As you can see, when original content is used as the search input, the match scores for possibly plagiarized articles are relatively low. However, if we were to copy and paste the text from one of the articles in our database, the results for the plagiarized article come back with a 99.99% match! So, how did we do it? In building the app, we start with a from Kaggle. This dataset contains 143,000 news articles from 15 major publications, but we’re just using the first 20,000. (The full dataset that this one is derived from contains over two million articles!) dataset of news articles Next, we clean up the dataset by renaming a couple columns and dropping a few unnecessary ones. Then, we run the articles through an embedding model to create — that’s metadata for machine learning algorithms to determine similarities between various inputs. We use the . Finally, we insert these vector embeddings into a managed by Pinecone. vector embeddings Average Word Embeddings Model vector database With the vector embeddings added to the database and indexed, we’re ready to start finding similar content. When users submit their article text as input, a request is made to an API endpoint that uses Pinecone’s SDK to query the index of vector embeddings. The endpoint returns 10 similar articles that were possibly plagiarized and displays them in the app’s UI. That’s it! Simple enough, right? If you’d like to try it out for yourself, you can . The contains instructions for how to run the app locally on your own machine. find the code for this app on GitHub README Demo App Code Walkthrough We’ve gone through the inner workings of the app, but how did we actually build it? As noted earlier, this is a Python Flask app that utilizes the Pinecone SDK. The HTML uses a template file, and the rest of the frontend is built using static CSS and JS assets. To keep things simple, all of the backend code is found in the file, which we’ve reproduced in full below: app.py from dotenv import load_dotenv
from flask import Flask
from flask import render_template
from flask import request
from flask import url_for
import json
import os
import pandas as pd
import pinecone
import re
import requests
from sentence_transformers import SentenceTransformer
from statistics import mean
import swifter

app = Flask(__name__)

PINECONE_INDEX_NAME = "plagiarism-checker"
DATA_FILE = "articles.csv"
NROWS = 20000

def initialize_pinecone():
    load_dotenv()
    PINECONE_API_KEY = os.environ["PINECONE_API_KEY"]
    pinecone.init(api_key=PINECONE_API_KEY)

def delete_existing_pinecone_index():
    if PINECONE_INDEX_NAME in pinecone.list_indexes():
        pinecone.delete_index(PINECONE_INDEX_NAME)

def create_pinecone_index():
    pinecone.create_index(name=PINECONE_INDEX_NAME, metric="cosine", shards=1)
    pinecone_index = pinecone.Index(name=PINECONE_INDEX_NAME)

    return pinecone_index

def create_model():
    model = SentenceTransformer('average_word_embeddings_komninos')

    return model

def prepare_data(data):
    # rename id column and remove unnecessary columns
    data.rename(columns={"Unnamed: 0": "article_id"}, inplace = True)
    data.drop(columns=['date'], inplace = True)

    # combine the article title and content into a single field
    data['content'] = data['content'].fillna('')
    data['content'] = data.content.swifter.apply(lambda x: ' '.join(re.split(r'(?<=[.:;])\s', x)))
    data['title_and_content'] = data['title'] + ' ' + data['content']

    # create a vector embedding based on title and article content
    encoded_articles = model.encode(data['title_and_content'], show_progress_bar=True)
    data['article_vector'] = pd.Series(encoded_articles.tolist())

    return data

def upload_items(data):
    items_to_upload = [(row.id, row.article_vector) for i, row in data.iterrows()]
    pinecone_index.upsert(items=items_to_upload)

def process_file(filename):
    data = pd.read_csv(filename, nrows=NROWS)
    data = prepare_data(data)
    upload_items(data)
    pinecone_index.info()

    return data

def map_titles(data):
    return dict(zip(uploaded_data.id, uploaded_data.title))

def map_publications(data):
    return dict(zip(uploaded_data.id, uploaded_data.publication))

def query_pinecone(originalContent):
    query_content = str(originalContent)
    query_vectors = [model.encode(query_content)]

    query_results = pinecone_index.query(queries=query_vectors, top_k=10)
    res = query_results[0]

    results_list = []

    for idx, _id in enumerate(res.ids):
        results_list.append({
            "id": _id,
            "title": titles_mapped[int(_id)],
            "publication": publications_mapped[int(_id)],
            "score": res.scores[idx],
        })

    return json.dumps(results_list)

initialize_pinecone()
delete_existing_pinecone_index()
pinecone_index = create_pinecone_index()
model = create_model()
uploaded_data = process_file(filename=DATA_FILE)
titles_mapped = map_titles(uploaded_data)
publications_mapped = map_publications(uploaded_data)

@app.route("/")
def index():
    return render_template("index.html")

@app.route("/api/search", methods=["POST", "GET"])
def search():
    if request.method == "POST":
        return query_pinecone(request.form.get("originalContent", ""))
    if request.method == "GET":
        return query_pinecone(request.args.get("originalContent", ""))
    return "Only GET and POST methods are allowed for this endpoint" Let’s go over the important parts of the file so that we understand it. app.py On lines 1–14, we import our app’s dependencies. Our app relies on the following: for reading environment variables from the file dotenv .env for the web application setup flask for working with JSON json also for getting environment variables os for working with the dataset pandas for working with the Pinecone SDK pinecone for working with regular expressions (RegEx) re for making API requests to download our dataset requests for some handy stats methods statistics for our embedding model sentence_transformers for working with the pandas dataframe swifter On line 16, we provide some boilerplate code to tell Flask the name of our app. On lines 18–20, we define some constants that will be used in the app. These include the name of our Pinecone index, the file name of the dataset, and the number of rows to read from the CSV file. On lines 22–25, our method gets our API key from the file and uses it to initialize Pinecone. initialize_pinecone .env On lines 27–29, our method searches our Pinecone instance for indexes with the same name as the one we’re using (“plagiarism-checker”). If an existing index is found, we delete it. delete_existing_pinecone_index On lines 31–35, our method creates a new index using the name we chose (“plagiarism-checker”), the “cosine” proximity metric, and only one shard. create_pinecone_index On lines 37–40, our method uses the library to work with the Average Word Embeddings Model. We’ll encode our vector embeddings using this model later. create_model sentence_transformers On lines 62–68, our method reads the CSV file and then calls the and methods on it. Those two methods are described next. process_file prepare_data upload_items On lines 42–56, our method adjusts the dataset by renaming the first “id” column and dropping the “date” column. It then combines the article title with the article content into a single field. We’ll use this combined field when creating the vector embeddings. prepare_data On lines 58–60, our method creates a vector embedding for each article by encoding it using our model. Then, we insert the vector embeddings into the Pinecone index. upload_items On lines 70–74, our and methods create some dictionaries of the titles and publication names to make it easier to find articles by their IDs later. map_titles map_publications Each of the methods we’ve described so far is called on lines 95–101 when the backend app is started. This work prepares us for the final step of actually querying the Pinecone index based on user input. On lines 103–113, we define two routes for our app: one for the home page and one for the API endpoint. The home page serves up the template file along with the JS and CSS assets, and the API endpoint provides the search functionality for querying the Pinecone index. index.html Finally, on lines 76–93, our method takes the user’s article content input, converts it into a vector embedding, and then queries the Pinecone index to find similar articles. This method is called when the endpoint is hit, which occurs any time the user submits a new search query. query_pinecone /api/search For the visual learners out there, here’s a diagram outlining how the app works: Example Scenarios So, putting this all together, what does the user experience look like? Let’s look at three scenarios: original content, an exact copy of plagiarized content, and “patch written” content. When original content is submitted, the app responds with some possibly related articles, but the match scores are quite low. This is a good sign, as the content is not plagiarized, so we would expect low match scores. When an exact copy of plagiarized content is submitted, the app responds with a nearly perfect match score for a single article. That’s because the content is identical. Nice find, plagiarism checker! Now, for the third scenario, we should define what we mean by “patch written” content. Patch writing is a form of plagiarism in which someone copies and pastes stolen content but then attempts to mask the fact that they’ve plagiarized the work by changing some of the words here and there. If a sentence from the original article says, “He was overjoyed to find his lost dog,” someone might patch write the content to instead say, “He was happy to retrieve his missing dog.” This is somewhat different from paraphrasing because the main sentence structure of the content often stays the same throughout the entire plagiarized article. Here’s the fun part: Our plagiarism checker does really well in identifying “patch written” content too! If you were to copy and paste one of the articles in the database and then change some words here and there, and maybe even delete a few sentences or paragraphs, the match score will still come back as a nearly perfect match! When I attempted this with a copied and pasted article that had a 99.99% match score, the “patch written” content still returned a 99.88% match score after my revisions! Not too shabby! Our plagiarism checker looks like it’s working well. Conclusion and Next Steps We’ve now created a simple Python app to solve a real-world problem. Imitation may be the highest form of flattery, but no one likes having their work stolen. In a growing world of content, a plagiarism checker like this would be highly useful to authors and teachers alike. This demo app does have some limitations, as it is just a demo after all. The database of articles loaded into our index only contains 20,000 articles from 15 major news publications. However, there are millions or even billions of articles and blog posts out there. A plagiarism checker like this is only useful if it is checking your input against all the places where your work may have been plagiarized. This app would be better if our index had more articles in it and if we were continuously adding to it. Regardless, at this point we’ve demonstrated a solid proof of concept. Pinecone, as a managed similarity search service, did the heavy lifting for us when it came to the machine learning aspect. With it, we were able to build a useful application that utilizes natural language processing and fairly easily, and now we have peace of mind knowing our work isn’t being plagiarized. semantic search

How to Build a Plagiarism Checker Using Machine Learning

Content platforms thrive on suggesting related content to their users. The more relevant items the platform can provide, the longer the user will stay on the site, which often translates to increased ad revenue for the company. If you’ve ever visited a news website, online publication, or blogging platform, you’ve likely been exposed to a recommendation engine. Each of these takes input based on your reading history and then suggests more content you might like. As a simple solution, a platform might implement a tag-based recommendation engine — you read a “Business” article, so here are five more articles tagged “Business.” However, an even better approach to building a recommendation engine is to use . similarity search and a machine learning algorithm In this article, we’ll build a Python Flask app that uses — a similarity search service — to create our very own article recommendation engine. Pinecone Demo App Overview Below, you can see a brief animation of how our demo app works. Ten articles are initially displayed on the page. The user can choose any combination of those 10 articles to represent their reading history. When the user clicks the Submit button, the reading history is used as input to query the article database, and then 10 more related articles are displayed to the user. As you can see, the related articles returned are exceptionally accurate! There are 1,024 possible combinations of reading history that can be used as input in this example, and every combination produces meaningful results. So, how did we do it? In building the app, we first found a from Kaggle. This dataset contains 143,000 news articles from 15 major publications, but we’re just using the first 20,000. (The full dataset that this one is derived from contains over two million articles!) dataset of news articles We then cleaned up the dataset by renaming a couple of columns and dropping those that are unnecessary. Next, we ran the articles through an embedding model to create vector embeddings — that’s, metadata for machine learning algorithms to determine similarities between various inputs. We used the . We then inserted these vector embeddings into a vector index managed by Pinecone. Average Word Embeddings Model With the vector embeddings added to the index, we’re ready to start finding related content. When users submit their reading history, a request is made to an API endpoint that uses Pinecone’s SDK to query the index of vector embeddings. The endpoint returns 10 similar news articles and displays them in the app’s UI. That’s it! Simple enough, right? If you’d like to try it out for yourself, you can . The contains instructions for how to run the app locally on your own machine. find the code for this app on GitHub README Demo App Code Walkthrough We’ve gone through the inner workings of the app, but how did we actually build it? As noted earlier, this is a Python Flask app that utilizes the Pinecone SDK. The HTML uses a template file, and the rest of the frontend is built using static CSS and JS assets. To keep things simple, all of the backend code is found in the file, which we’ve reproduced in full below: app.py from dotenv import load_dotenv
from flask import Flask
from flask import render_template
from flask import request
from flask import url_for
import json
import os
import pandas as pd
import pinecone
import re
import requests
from sentence_transformers import SentenceTransformer
from statistics import mean
import swifter

app = Flask(__name__)

PINECONE_INDEX_NAME = "article-recommendation-service"
DATA_FILE = "articles.csv"
NROWS = 20000

def initialize_pinecone():
    load_dotenv()
    PINECONE_API_KEY = os.environ["PINECONE_API_KEY"]
    pinecone.init(api_key=PINECONE_API_KEY)

def delete_existing_pinecone_index():
    if PINECONE_INDEX_NAME in pinecone.list_indexes():
        pinecone.delete_index(PINECONE_INDEX_NAME)

def create_pinecone_index():
    pinecone.create_index(name=PINECONE_INDEX_NAME, metric="cosine", shards=1)
    pinecone_index = pinecone.Index(name=PINECONE_INDEX_NAME)

    return pinecone_index

def create_model():
    model = SentenceTransformer('average_word_embeddings_komninos')

    return model

def prepare_data(data):
    # rename id column and remove unnecessary columns
    data.rename(columns={"Unnamed: 0": "article_id"}, inplace = True)
    data.drop(columns=['date'], inplace = True)

    # extract only first few sentences of each article for quicker vector calculations
    data['content'] = data['content'].fillna('')
    data['content'] = data.content.swifter.apply(lambda x: ' '.join(re.split(r'(?<=[.:;])\s', x)[:4]))
    data['title_and_content'] = data['title'] + ' ' + data['content']

    # create a vector embedding based on title and article columns
    encoded_articles = model.encode(data['title_and_content'], show_progress_bar=True)
    data['article_vector'] = pd.Series(encoded_articles.tolist())

    return data

def upload_items(data):
    items_to_upload = [(row.id, row.article_vector) for i, row in data.iterrows()]
    pinecone_index.upsert(items=items_to_upload)

def process_file(filename):
    data = pd.read_csv(filename, nrows=NROWS)
    data = prepare_data(data)
    upload_items(data)
    pinecone_index.info()

    return data

def map_titles(data):
    return dict(zip(uploaded_data.id, uploaded_data.title))

def map_publications(data):
    return dict(zip(uploaded_data.id, uploaded_data.publication))

def query_pinecone(reading_history_ids):
    reading_history_ids_list = list(map(int, reading_history_ids.split(',')))
    reading_history_articles = uploaded_data.loc[uploaded_data['id'].isin(reading_history_ids_list)]

    article_vectors = reading_history_articles['article_vector']
    reading_history_vector = [*map(mean, zip(*article_vectors))]

    query_results = pinecone_index.query(queries=[reading_history_vector], top_k=10)
    res = query_results[0]

    results_list = []

    for idx, _id in enumerate(res.ids):
        results_list.append({
            "id": _id,
            "title": titles_mapped[int(_id)],
            "publication": publications_mapped[int(_id)],
            "score": res.scores[idx],
        })

    return json.dumps(results_list)

initialize_pinecone()
delete_existing_pinecone_index()
pinecone_index = create_pinecone_index()
model = create_model()
uploaded_data = process_file(filename=DATA_FILE)
titles_mapped = map_titles(uploaded_data)
publications_mapped = map_publications(uploaded_data)

@app.route("/")
def index():
    return render_template("index.html")

@app.route("/api/search", methods=["POST", "GET"])
def search():
    if request.method == "POST":
        return query_pinecone(request.form.history)
    if request.method == "GET":
        return query_pinecone(request.args.get("history", ""))
    return "Only GET and POST methods are allowed for this endpoint" Let’s go over the important parts of the file so that we understand it. app.py On lines 1–14, we import our app’s dependencies. Our app relies on the following: for reading environment variables from the file dotenv .env for the web application setup flask for working with JSON json also, for getting environment variables os for working with the dataset pandas for working with the Pinecone SDK pinecone for working with regular expressions (RegEx) re for making API requests to download our dataset requests for some handy stats methods statistics for our embedding model sentence_transformers for working with the pandas dataframe swifter On line 16, we provide some boilerplate code to tell Flask the name of our app. On lines 18–20, we define some constants that will be used in the app. These include the name of our Pinecone index, the file name of the dataset, and the number of rows to read from the CSV file. On lines 22–25, our method gets our API key from the file and uses it to initialize Pinecone. initialize_pinecone .env On lines 27–29, our method searches our Pinecone instance for indexes with the same name as the one we’re using (“article-recommendation-service”). If an existing index is found, we delete it. delete_existing_pinecone_index On lines 31–35, our method creates a new index using the name we chose (“article-recommendation-service”), the “cosine” proximity metric, and only one shard. create_pinecone_index On lines 37–40, our method uses the library to work with the Average Word Embeddings Model. We’ll encode our vector embeddings using this model later. create_model sentence_transformers On lines 62–68, our method reads the CSV file and then calls the and methods on it. Those two methods are described next. process_file prepare_data upload_items On lines 42–56, our method adjusts the dataset by renaming the first “id” column and dropping the “date” column. It then grabs the first four lines of each article and combines them with the article title to create a new field that serves as the data to encode. We could create vector embeddings based on the entire body of the article, but four lines will suffice in order to speed up the encoding process. prepare_data On lines 58–60, our method creates a vector embedding for each article by encoding it using our model. The vector embeddings are then inserted into the Pinecone index. upload_items On lines 70–74, our and methods create some dictionaries of the titles and publication names to make it easier to find articles by their IDs later. map_titles map_publications Each of the methods we’ve described so far is called on lines 98–104 when the backend app is started. This work prepares us for the final step of actually querying the Pinecone index based on user input. On lines 106–116, we define two routes for our app: one for the home page and one for the API endpoint. The home page serves up the template file along with the JS and CSS assets, and the API endpoint provides the search functionality for querying the Pinecone index. index.html Finally, on lines 76–96, our method takes the user’s reading history input, converts it into a vector embedding, and then queries the Pinecone index to find similar articles. This method is called when the endpoint is hit, which occurs whenever the user submits a new search query. query_pinecone /api/search For the visual learners out there, here’s a diagram outlining how the app works: Example Scenarios So, putting this all together, what does the user experience look like? Let’s look at three scenarios: a user interested in sports, a user interested in technology, and a user interested in politics. The sports user selects the first two articles about Serena Williams and Andy Murray, two famous tennis players, to use their reading history. After submitting their choices, the app responds with articles about Wimbledon, the US Open, Roger Federer, and Rafael Nadal. Spot on! The technology user selects articles about Samsung and Apple. After they submit their choices, the app responds with articles about Samsung, Apple, Google, Intel, and iPhones. Great recommendations again! The politics user selects a single article about voter fraud. After they submit their choice, the app responds with articles about voter ID, the US 2020 election, voter turnout, and claims of illegal voting (and why they don’t hold up). Three for three! Our recommendation engine is proving to be incredibly useful. Conclusion We’ve now created a simple Python app to solve a real-world problem. If content sites can recommend relevant content to their users, users will enjoy the content more and will spend more time on the site, resulting in more revenue for the company. Everyone wins! Similarity search helps provide better suggestions for your users. And Pinecone, as a similarity search service, makes it easy for you to provide recommendations to your users so that you can focus on what you do best — building an engaging platform filled with content worth reading.

Build an Article Recommendation Engine Using Machine Learning

Feature Selection in python is the process where you automatically or manually select the features in the dataset that contribute most to your prediction variable or output in which you are interested. One major reason is that machine learning follows the rule of “garbage in garbage out” and that is why you need to be very concerned about the features that are being fed to the model. Keep in mind that not all features presented in your dataset are important to give you the best model performance. Top 4 Reasons to Apply Feature Selection in Python: It improves the accuracy of a model if the right subset is chosen. It reduces overfitting. It enables the machine learning algorithm to train faster. It reduces the complexity of a model and makes it easier to interpret. “I prepared a model by selecting all the features and I got an accuracy of around 65% which is not good for a predictive model and after doing some feature selection and feature engineering without doing any logical changes in my model code my accuracy jumped to 81% which is quite impressive”- By . Raheel Shaikh In this article, you will learn how to automatically select important features by using an open-source python package called featurewiz. What is Featurewiz? Featurewiz is a new open-source python package for automatically creating and selecting important features in your dataset that will create the best model with higher performance. It also uses advanced feature engineering strategies to create new features before selecting the best set of features with a single line of code. Note: Featurewiz can automatically detect if the problem is regression or classification. How does it work? Featurewiz uses the SULOV algorithm and Recursive XGBoost to reduce features to select the best features for the model. SULOV means Searching for Uncorrelated List of Variables. The algorithm works in the following steps. (a) SULOV First step: find all the pairs of highly correlated variables exceeding a correlation threshold (say absolute(0.8)). Second step: find their Mutual Information Score to the target variable. Mutual Information Score is a non-parametric scoring method. So it's suitable for all kinds of variables and target. Third step: take each pair of correlated variables, then knock off the one with the lower Mutual Information Score. Final step: Collect the ones with the highest Information scores and least correlation with each other. After selecting the features with less correlation and high mutual information score, the Recursive XGBoost is used to find the best features among the remaining features. Here is how it works. (b) Recursive XGBoost First step: Select all features in the dataset and split the dataset into train and valid sets. Second step: Find top X features on train using valid for early stopping (to prevent overfitting). Third step: Take the next set of features and find top X. Final step: Repeat this 5 times and finally combine all selected features and de-duplicate them. Installation The package requires xgboost, NumPy, pandas and matplotlib. In addition, tIt should run on most Python 3. You can install Featurewiz using PyPI. pip install featurewiz How to Use Featurewiz for Feature Selection in Python We will use the Mobile Price dataset to find the best features that can help to get good accuracy when predicting the price range. 0 (low cost) 1 (medium cost) 2 (high cost) 4 (very high cost) You can download the dataset . here Import python packages. pandas pd numpy np seaborn sns sklearn.model_selection train_test_split sklearn.preprocessing StandardScaler sklearn.ensemble RandomForestClassifier sklearn.linear_model LogisticRegression sklearn.metrics accuracy_score featurewiz featurewiz
np.random.seed( ) # import packages import as import as import as from import from import from import from import from import from import 1234 Load the Mobile Price dataset. data = pd.read_csv( )
 
data.shape '../data/train.csv' (2000, 21) The dataset contains 21 columns(20 features and 1 target) and luckily this dataset has no missing values. Split the data into independent features and target. X = data.drop([ ],axis= )
 
y = data.price_range.values 'price_range' 1 Then Standardize the features by using StandardScaler from scikit-learn. X_scaled =  StandardScaler().fit_transform(X) Split the data into train and validate sets. 20% of the data will be used for validation. X_train, X_valid, y_train, y_valid = train_test_split(X_scaled,y,test_size = ,stratify=y, random_state= ) 0.2 1 Create and train the on the train set. RandoForestclassifier classifier = RandomForestClassifier()
 
classifier.fit(X_train,y_train) Make a prediction on the validation set and then check model performance. preds = classifier.predict(X_valid) accuracy_score(preds,y_valid) # make prediction # check performance 0.88 The model accuracy is when we use all 20 features available in the dataset. 88% Now you can use Featurewiz to automatically select the best set of features that will give the best model performance. target = features, train = featurewiz(data, target, corr_limit= , verbose= , sep= ,
header= ,test_data= , feature_engg= , category_encoders= ) # automatic feature selection by using featurewiz package 'price_range' 0.7 2 "," 0 "" "" "" On the Featurewiz instance, we have added the dataset and the name of the target variable. You can also change the correlation limit by using corr_limit (by default is 7). During the selection process, it will show the following series of output. Skipping feature engineering since no feature_engg input...
Skipping category encoding since no category encoders specified input...
Loading train data...
Shape of your Data Set loaded: ( , )
Loading test data...
    Filename an empty string file able to be loaded Classifying variables data set... Predictors classified...
        No variables removed since no ID low-information variables found data set
No GPU active on this device
    Running XGBoost using CPU parameters
Removing columns further processing since ID low information variables
    columns removed: []
    After removing redundant variables further processing, features left = Searching highly correlated variables variables using SULOV method No highly correlated variables data set to remove. All selected...
    Adding categorical variables to reduced numeric variables  of Current number of predictors = Finding Important Features using Boosted Trees algorithm...
        using variables...
        using variables...
        using variables...
        using variables...
        using variables...
Selected important features your dataset
    Time taken ( seconds) = Returning list of important features dataframe. in 2000 21 is or not ############## C L A S S I F Y I N G  V A R I A B L E S  #################### in 20 or in 0 from or from 20 #### Single_Label Multi_Classification Feature Selection Started #### for from 20 #####  SULOV : Searching for Uncorrelated List Of Variables (takes time...) ############ in 0 20 ############## F E A T U R E   S E L E C T I O N  #################### 20 20 16 12 8 4 16 from in 19 16 and As you can see, Featurewiz selects 16 important features from the dataset. The Featurewiz instance returns two objects Features - a list of selected features One dataframe - This dataframe contains only selected features and the target variable. Now you can train the RandomForestClassifier again with selected features and see if the model performance will improve. Let see the list of selected features. print(features) ['ram', 'battery_power', 'px_height', 'px_width', 'touch_screen', 'mobile_wt', 'int_memory', 'three_g', 'sc_h', 'four_g', 'sc_w', 'n_cores', 'fc', 'pc', 'talk_time', 'wifi'] Split the one dataframe into selected features and the target. X_new = train.drop([ ],axis= )
 
y = train.price_range.values #split data into feature and target 'price_range' 1 Then, Standardize the selected features by using StandardScaler from scikit-learn. X_scaled =  StandardScaler().fit_transform(X_new) # preprocessing the features Split the data into train and validate sets. 20% of the data will be used for validation. X_train, X_valid, y_train, y_valid = train_test_split(X_scaled,y,test_size = ,stratify=y, random_state= ) #split data into train and validate 0.2 1 Create and train the RandoForestclassifier on the train set again. classifier = RandomForestClassifier()
 
classifier.fit(X_train,y_train) # create and train classifier Make a prediction on the validation set and then check model performance. preds = classifier.predict(X_valid) accuracy_score(preds,y_valid) # make prediction # check performance 0.905 The model accuracy has increased from to when we use the best-selected features (16 out of 20  features) from the dataset. 88% 90.5% Final Thoughts on Feature Selection in Python In this article, you have learned how you can automatically select important features by using the Featurewiz package. You can also use Featurewiz on any Multi-Class or Multi-Label Dataset. So you can have as many target labels as you want. There are more options available on the Featurewiz package. I recommend you read them here. If you learned something new or enjoyed reading this article, please share it so that others can see it. Until then, see you in the next post! You can also find me on Twitter . @Davis_McDavid And you can read more articles like this . here Want to keep up to date with all the latest in python? in the footer below. Subscribe to our newsletter

Automatic Feature Selection in Python: An Essential Guide

Selección automática de características en Python: una guía esencial

Feature Selection is the process where you automatically or manually select those features which contribute most to your prediction variable or output in which you are interested. One major reason is that machine learning follows the rule of and that is why you need to be very concerned about the features that are being fed to the model. “garbage in garbage out” Having irrelevant features in your data can both increase the computational cost of modeling and the of the models and make your model learn based on irrelevant features. This means you need to select only to be presented during model training. decrease accuracy important features Top reasons to apply feature selection: It enables the machine learning algorithm to train faster. It reduces the complexity of a model and makes it easier to interpret. It improves the accuracy of a model if the right subset is chosen. It reduces overfitting. “I prepared a model by selecting all the features and I got an accuracy of around which is not pretty good for a predictive model and after doing some feature selection and feature engineering without doing any logical changes in my model code my accuracy jumped to which is quite impressive”- By Raheel Shaikh 65% 81% are intended to reduce the number of features to those that are believed to be most useful/important to a model in order to predict the target feature. There are different feature selection methods available that can help you to select important features. The most popular methods are. Feature selection methods .Spearman’s rank coefficient. Pearson’s correlation coefficient ANOVA correlation coefficient. Chi-Squared test. Machine Learning Algorithms (Random Forest and Extra Tree). Correlation Matrix with Heatmap. Mutual Information. You can read more about feature selection methods from . Scikit learn library One of the challenges of these methods is to which method(s) should you apply in your dataset to select important features. Each method has its own way to identify the important features. For example, a certain feature can be selected as an important feature in one method and not selected as an important feature in another method. identify Xverse package can help you to solve this problem. “At the end of the day, some machine learning projects succeed and some fail.What makes the difference? Easily the most important factor is the features used.” — Pedro Domingos What is Xverse? Xverse stands for X Universe which is the python package for machine learning to assist Data Scientists with feature transformation and feature selection. Xverse is created by . Sundar Krishnan How does it work? Xverse applies a variety of techniques to select features. When an algorithm picks a feature, it gives a vote for that feature. In the end, Xverse calculates the total votes for each feature and then picks the best ones based on votes. This way, we end up picking the best variables with minimum effort in the features selection process. Xverse uses the following methods to select important features. Information Value using Weight of evidence. Variable Importance using Random Forest. Recursive Feature Elimination. Variable Importance using Extra trees classifier. Chi-Square best variables. L1-based feature selection. Installation The package requires Numpy, Pandas, Scikit-learn, Scipy and Statsmodels. In addition, the package is tested on Python version 3.5 and above. Run the following command to install Xverse. pip xverse install I will use the Loan dataset to find the best features that can help to get good accuracy when predicting if a customer deserves to get a loan or not. You can download the dataset . here Import important packages for this problem. pandas pd numpy np xverse.ensemble VotingSelector sklearn.preprocessing StandardScaler, MinMaxScaler warnings warnings.filterwarnings( ) import as import as from import from import import # To ignore any warnings "ignore" Load the Loan dataset. data = pd.read_csv( )

data.columns "data/loans_data.csv" Loan_ID
Gender
Married
Dependents
Education
Self_Employed
ApplicantIncome
CoapplicantIncome
LoanAmount
Loan_Amount_Term
Credit_History
Property_Area
Loan_Status We have 12 independent features and a target (Loan_Status). You can read the description of each feature here. I have created a simple python function to handle missing data and feature engineering. data[ ].replace( , ,inplace= )
	data[ ].replace( , ,inplace= )
	data[ ].replace( , ,inplace= ) data[ ].fillna(data[ ].mode()[ ], inplace= )
	data[ ].fillna(data[ ].mode()[ ], inplace= )
	data[ ].fillna(data[ ].mode()[ ], inplace= )
	data[ ].fillna(data[ ].mode()[ ], inplace= )
	data[ ].fillna(data[ ].mode()[ ], inplace= )
	data[ ].fillna(data[ ].mode()[ ], inplace= )
	data[ ].fillna(data[ ].median(), inplace= ) data = data.drop( ,axis= ) data[ ] = MinMaxScaler().fit_transform(data[ ].values.reshape( , ))
	data[ ] = MinMaxScaler().fit_transform(data[ ].values.reshape( , ))
	data[ ] = MinMaxScaler().fit_transform(data[ ].values.reshape( , ))
	data[ ] = MinMaxScaler().fit_transform(data[ ].values.reshape( , )) data : def preprocessing (data) # replace with numerical values 'Dependents' '3+' 3 True 'Loan_Status' 'N' 0 True 'Loan_Status' 'Y' 1 True # handle missing data 'Gender' 'Gender' 0 True 'Married' 'Married' 0 True 'Dependents' 'Dependents' 0 True 'Self_Employed' 'Self_Employed' 0 True 'Credit_History' 'Credit_History' 0 True 'Loan_Amount_Term' 'Loan_Amount_Term' 0 True 'LoanAmount' 'LoanAmount' True # drop ID column 'Loan_ID' 1 #scale the data "ApplicantIncome" "ApplicantIncome" -1 1 "LoanAmount" "LoanAmount" -1 1 "CoapplicantIncome" "CoapplicantIncome" -1 1 "Loan_Amount_Term" "Loan_Amount_Term" -1 1 return Let’s preprocess the loan dataset. data = prepocessing(data) Split into independent features and target. X = data.drop( ,axis = )
y = data.Loan_Status 'Loan_Status' 1 Now is time to call VotingSelector from Xverse and train it to find the best features by using the voting approach. clf = VotingSelector(minimum_votes= )
clf.fit(X, y) #train to find the best features 2 I have set , this means the feature to be selected must have at least a total of from the six feature selection methods presented in Xverse. Features below a total of will be neglected. minimum_votes= 2 2 votes 2 votes After training, we can see the feature importance in each feature selection method used during the training. clf.feature_importances_ #show important features The output shows all the features and their values of importance in each method. Now let’s observe the votes from these feature selection methods. clf.feature_votes_ # votes The output shows the and at last, it shows for each feature. It starts by show features with many votes to the features with low or zero votes. variable name, list of feature selection methods and their votes total votes You can see that has a total of which means credit_History is a very important feature for this Loan problem. But both and features have which means we can neglect these two features because of the very low contribution to the prediction if a customer deserves to get a loan or not. Credit_History 6 votes, Gender Self_employed 0 votes, Now we can transform our data to remain with only important selected features. X = clf.transform(X) # transform your data into important features Conclusion Xverse is under active development. Currently, xverse package handles only binary targets. The code for this post is available on . Github If you learned something new or enjoyed reading this article, please share it so that others can see it. Feel free to leave a comment too. Till then, see you in the next post! I can also be reached on Twitter @Davis_McDavid Previously published . here

Features Selection by Using Xverse Package

Stories, News & Mentions about DAREDROP

Daredrop’s Stories on HackerNoon