Feature Selection in python is the process where you automatically or manually select the features in the dataset that contribute most to your prediction variable or output in which you are interested. One major reason is that machine learning follows the rule of “garbage in garbage out” and that is why you need to be very concerned about the features that are being fed to the model. Keep in mind that not all features presented in your dataset are important to give you the best model performance. Top 4 Reasons to Apply Feature Selection in Python: It improves the accuracy of a model if the right subset is chosen. It reduces overfitting. It enables the machine learning algorithm to train faster. It reduces the complexity of a model and makes it easier to interpret. “I prepared a model by selecting all the features and I got an accuracy of around 65% which is not good for a predictive model and after doing some feature selection and feature engineering without doing any logical changes in my model code my accuracy jumped to 81% which is quite impressive”- By . Raheel Shaikh In this article, you will learn how to automatically select important features by using an open-source python package called featurewiz. What is Featurewiz? Featurewiz is a new open-source python package for automatically creating and selecting important features in your dataset that will create the best model with higher performance. It also uses advanced feature engineering strategies to create new features before selecting the best set of features with a single line of code. Note: Featurewiz can automatically detect if the problem is regression or classification. How does it work? Featurewiz uses the SULOV algorithm and Recursive XGBoost to reduce features to select the best features for the model. SULOV means Searching for Uncorrelated List of Variables. The algorithm works in the following steps. (a) SULOV First step: find all the pairs of highly correlated variables exceeding a correlation threshold (say absolute(0.8)). Second step: find their Mutual Information Score to the target variable. Mutual Information Score is a non-parametric scoring method. So it's suitable for all kinds of variables and target. Third step: take each pair of correlated variables, then knock off the one with the lower Mutual Information Score. Final step: Collect the ones with the highest Information scores and least correlation with each other. After selecting the features with less correlation and high mutual information score, the Recursive XGBoost is used to find the best features among the remaining features. Here is how it works. (b) Recursive XGBoost First step: Select all features in the dataset and split the dataset into train and valid sets. Second step: Find top X features on train using valid for early stopping (to prevent overfitting). Third step: Take the next set of features and find top X. Final step: Repeat this 5 times and finally combine all selected features and de-duplicate them. Installation The package requires xgboost, NumPy, pandas and matplotlib. In addition, tIt should run on most Python 3. You can install Featurewiz using PyPI. pip install featurewiz How to Use Featurewiz for Feature Selection in Python We will use the Mobile Price dataset to find the best features that can help to get good accuracy when predicting the price range. 0 (low cost) 1 (medium cost) 2 (high cost) 4 (very high cost) You can download the dataset . here Import python packages. pandas pd numpy np seaborn sns sklearn.model_selection train_test_split sklearn.preprocessing StandardScaler sklearn.ensemble RandomForestClassifier sklearn.linear_model LogisticRegression sklearn.metrics accuracy_score featurewiz featurewiz
np.random.seed( ) # import packages import as import as import as from import from import from import from import from import from import 1234 Load the Mobile Price dataset. data = pd.read_csv( )
 
data.shape '../data/train.csv' (2000, 21) The dataset contains 21 columns(20 features and 1 target) and luckily this dataset has no missing values. Split the data into independent features and target. X = data.drop([ ],axis= )
 
y = data.price_range.values 'price_range' 1 Then Standardize the features by using StandardScaler from scikit-learn. X_scaled =  StandardScaler().fit_transform(X) Split the data into train and validate sets. 20% of the data will be used for validation. X_train, X_valid, y_train, y_valid = train_test_split(X_scaled,y,test_size = ,stratify=y, random_state= ) 0.2 1 Create and train the on the train set. RandoForestclassifier classifier = RandomForestClassifier()
 
classifier.fit(X_train,y_train) Make a prediction on the validation set and then check model performance. preds = classifier.predict(X_valid) accuracy_score(preds,y_valid) # make prediction # check performance 0.88 The model accuracy is when we use all 20 features available in the dataset. 88% Now you can use Featurewiz to automatically select the best set of features that will give the best model performance. target = features, train = featurewiz(data, target, corr_limit= , verbose= , sep= ,
header= ,test_data= , feature_engg= , category_encoders= ) # automatic feature selection by using featurewiz package 'price_range' 0.7 2 "," 0 "" "" "" On the Featurewiz instance, we have added the dataset and the name of the target variable. You can also change the correlation limit by using corr_limit (by default is 7). During the selection process, it will show the following series of output. Skipping feature engineering since no feature_engg input...
Skipping category encoding since no category encoders specified input...
Loading train data...
Shape of your Data Set loaded: ( , )
Loading test data...
    Filename an empty string file able to be loaded Classifying variables data set... Predictors classified...
        No variables removed since no ID low-information variables found data set
No GPU active on this device
    Running XGBoost using CPU parameters
Removing columns further processing since ID low information variables
    columns removed: []
    After removing redundant variables further processing, features left = Searching highly correlated variables variables using SULOV method No highly correlated variables data set to remove. All selected...
    Adding categorical variables to reduced numeric variables  of Current number of predictors = Finding Important Features using Boosted Trees algorithm...
        using variables...
        using variables...
        using variables...
        using variables...
        using variables...
Selected important features your dataset
    Time taken ( seconds) = Returning list of important features dataframe. in 2000 21 is or not ############## C L A S S I F Y I N G  V A R I A B L E S  #################### in 20 or in 0 from or from 20 #### Single_Label Multi_Classification Feature Selection Started #### for from 20 #####  SULOV : Searching for Uncorrelated List Of Variables (takes time...) ############ in 0 20 ############## F E A T U R E   S E L E C T I O N  #################### 20 20 16 12 8 4 16 from in 19 16 and As you can see, Featurewiz selects 16 important features from the dataset. The Featurewiz instance returns two objects Features - a list of selected features One dataframe - This dataframe contains only selected features and the target variable. Now you can train the RandomForestClassifier again with selected features and see if the model performance will improve. Let see the list of selected features. print(features) ['ram', 'battery_power', 'px_height', 'px_width', 'touch_screen', 'mobile_wt', 'int_memory', 'three_g', 'sc_h', 'four_g', 'sc_w', 'n_cores', 'fc', 'pc', 'talk_time', 'wifi'] Split the one dataframe into selected features and the target. X_new = train.drop([ ],axis= )
 
y = train.price_range.values #split data into feature and target 'price_range' 1 Then, Standardize the selected features by using StandardScaler from scikit-learn. X_scaled =  StandardScaler().fit_transform(X_new) # preprocessing the features Split the data into train and validate sets. 20% of the data will be used for validation. X_train, X_valid, y_train, y_valid = train_test_split(X_scaled,y,test_size = ,stratify=y, random_state= ) #split data into train and validate 0.2 1 Create and train the RandoForestclassifier on the train set again. classifier = RandomForestClassifier()
 
classifier.fit(X_train,y_train) # create and train classifier Make a prediction on the validation set and then check model performance. preds = classifier.predict(X_valid) accuracy_score(preds,y_valid) # make prediction # check performance 0.905 The model accuracy has increased from to when we use the best-selected features (16 out of 20  features) from the dataset. 88% 90.5% Final Thoughts on Feature Selection in Python In this article, you have learned how you can automatically select important features by using the Featurewiz package. You can also use Featurewiz on any Multi-Class or Multi-Label Dataset. So you can have as many target labels as you want. There are more options available on the Featurewiz package. I recommend you read them here. If you learned something new or enjoyed reading this article, please share it so that others can see it. Until then, see you in the next post! You can also find me on Twitter . @Davis_McDavid And you can read more articles like this . here Want to keep up to date with all the latest in python? in the footer below. Subscribe to our newsletter

Target

Twitter

23 Common Data Science Interview Questions for Beginners

Automatic Feature Selection in Python: An Essential Guide

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

10 Best African Language Datasets for Data Science Projects

10 Pieces of Sci-Fi Armor We All Wish Were Real

10 Movie Tie-In Games That Were Actually Good

10 Most Anticipated Sci-fi Games of 2022 You Can’t Miss

10 Female Video Game Characters You Need to Know

10 Cutest Pink Pokémon of All Time

10 Best African Language Datasets for Data Science Projects

10 Pieces of Sci-Fi Armor We All Wish Were Real

10 Movie Tie-In Games That Were Actually Good

10 Most Anticipated Sci-fi Games of 2022 You Can’t Miss

10 Female Video Game Characters You Need to Know

10 Cutest Pink Pokémon of All Time

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps