Features Selection by Using Xverse Package

Feature Selection is the process where you automatically or manually select those features which contribute most to your prediction variable or output in which you are interested. One major reason is that machine learning follows the rule of and that is why you need to be very concerned about the features that are being fed to the model. “garbage in garbage out” Having irrelevant features in your data can both increase the computational cost of modeling and the of the models and make your model learn based on irrelevant features. This means you need to select only to be presented during model training. decrease accuracy important features Top reasons to apply feature selection: It enables the machine learning algorithm to train faster. It reduces the complexity of a model and makes it easier to interpret. It improves the accuracy of a model if the right subset is chosen. It reduces overfitting. “I prepared a model by selecting all the features and I got an accuracy of around which is not pretty good for a predictive model and after doing some feature selection and feature engineering without doing any logical changes in my model code my accuracy jumped to which is quite impressive”- By Raheel Shaikh 65% 81% are intended to reduce the number of features to those that are believed to be most useful/important to a model in order to predict the target feature. There are different feature selection methods available that can help you to select important features. The most popular methods are. Feature selection methods .Spearman’s rank coefficient. Pearson’s correlation coefficient ANOVA correlation coefficient. Chi-Squared test. Machine Learning Algorithms (Random Forest and Extra Tree). Correlation Matrix with Heatmap. Mutual Information. You can read more about feature selection methods from . Scikit learn library One of the challenges of these methods is to which method(s) should you apply in your dataset to select important features. Each method has its own way to identify the important features. For example, a certain feature can be selected as an important feature in one method and not selected as an important feature in another method. identify Xverse package can help you to solve this problem. “At the end of the day, some machine learning projects succeed and some fail.What makes the difference? Easily the most important factor is the features used.” — Pedro Domingos What is Xverse? Xverse stands for X Universe which is the python package for machine learning to assist Data Scientists with feature transformation and feature selection. Xverse is created by . Sundar Krishnan How does it work? Xverse applies a variety of techniques to select features. When an algorithm picks a feature, it gives a vote for that feature. In the end, Xverse calculates the total votes for each feature and then picks the best ones based on votes. This way, we end up picking the best variables with minimum effort in the features selection process. Xverse uses the following methods to select important features. Information Value using Weight of evidence. Variable Importance using Random Forest. Recursive Feature Elimination. Variable Importance using Extra trees classifier. Chi-Square best variables. L1-based feature selection. Installation The package requires Numpy, Pandas, Scikit-learn, Scipy and Statsmodels. In addition, the package is tested on Python version 3.5 and above. Run the following command to install Xverse. pip xverse install I will use the Loan dataset to find the best features that can help to get good accuracy when predicting if a customer deserves to get a loan or not. You can download the dataset . here Import important packages for this problem. pandas pd numpy np xverse.ensemble VotingSelector sklearn.preprocessing StandardScaler, MinMaxScaler warnings warnings.filterwarnings( ) import as import as from import from import import # To ignore any warnings "ignore" Load the Loan dataset. data = pd.read_csv( ) data.columns "data/loans_data.csv" Loan_ID Gender Married Dependents Education Self_Employed ApplicantIncome CoapplicantIncome LoanAmount Loan_Amount_Term Credit_History Property_Area Loan_Status We have 12 independent features and a target (Loan_Status). You can read the description of each feature here. I have created a simple python function to handle missing data and feature engineering. data[ ].replace( , ,inplace= ) data[ ].replace( , ,inplace= ) data[ ].replace( , ,inplace= ) data[ ].fillna(data[ ].mode()[ ], inplace= ) data[ ].fillna(data[ ].mode()[ ], inplace= ) data[ ].fillna(data[ ].mode()[ ], inplace= ) data[ ].fillna(data[ ].mode()[ ], inplace= ) data[ ].fillna(data[ ].mode()[ ], inplace= ) data[ ].fillna(data[ ].mode()[ ], inplace= ) data[ ].fillna(data[ ].median(), inplace= ) data = data.drop( ,axis= ) data[ ] = MinMaxScaler().fit_transform(data[ ].values.reshape( , )) data[ ] = MinMaxScaler().fit_transform(data[ ].values.reshape( , )) data[ ] = MinMaxScaler().fit_transform(data[ ].values.reshape( , )) data[ ] = MinMaxScaler().fit_transform(data[ ].values.reshape( , )) data : def preprocessing (data) # replace with numerical values 'Dependents' '3+' 3 True 'Loan_Status' 'N' 0 True 'Loan_Status' 'Y' 1 True # handle missing data 'Gender' 'Gender' 0 True 'Married' 'Married' 0 True 'Dependents' 'Dependents' 0 True 'Self_Employed' 'Self_Employed' 0 True 'Credit_History' 'Credit_History' 0 True 'Loan_Amount_Term' 'Loan_Amount_Term' 0 True 'LoanAmount' 'LoanAmount' True # drop ID column 'Loan_ID' 1 #scale the data "ApplicantIncome" "ApplicantIncome" -1 1 "LoanAmount" "LoanAmount" -1 1 "CoapplicantIncome" "CoapplicantIncome" -1 1 "Loan_Amount_Term" "Loan_Amount_Term" -1 1 return Let’s preprocess the loan dataset. data = prepocessing(data) Split into independent features and target. X = data.drop( ,axis = ) y = data.Loan_Status 'Loan_Status' 1 Now is time to call VotingSelector from Xverse and train it to find the best features by using the voting approach. clf = VotingSelector(minimum_votes= ) clf.fit(X, y) #train to find the best features 2 I have set , this means the feature to be selected must have at least a total of from the six feature selection methods presented in Xverse. Features below a total of will be neglected. minimum_votes= 2 2 votes 2 votes After training, we can see the feature importance in each feature selection method used during the training. clf.feature_importances_ #show important features The output shows all the features and their values of importance in each method. Now let’s observe the votes from these feature selection methods. clf.feature_votes_ # votes The output shows the and at last, it shows for each feature. It starts by show features with many votes to the features with low or zero votes. variable name, list of feature selection methods and their votes total votes You can see that has a total of which means credit_History is a very important feature for this Loan problem. But both and features have which means we can neglect these two features because of the very low contribution to the prediction if a customer deserves to get a loan or not. Credit_History 6 votes, Gender Self_employed 0 votes, Now we can transform our data to remain with only important selected features. X = clf.transform(X) # transform your data into important features Conclusion Xverse is under active development. Currently, xverse package handles only binary targets. The code for this post is available on . Github If you learned something new or enjoyed reading this article, please share it so that others can see it. Feel free to leave a comment too. Till then, see you in the next post! I can also be reached on Twitter @Davis_McDavid Previously published . here