When you training a machine learning model, you can have some features in your dataset that represent categorical values. Categorical features are types of data that may be divided into groups. There are three common categorical data types which are: - This has a set of orders. Example: rating happiness on a scale of 1-10 Ordinal - This has only two values. Example: Male or Female Binary - This does not have any set of orders. Example: Countries Nominal Most machine learning algorithms require numerical input and output variables. Therefore you will have to transform categorical features in your dataset into integers or floats to be utilized by machine learning algorithms. You can either use for the binary features or the method for nominal features. LabelEncoding One-hot-encoding In this article, you will learn how combining categorical features can improve your machine learning model performance. So let’s get started.🚀 Combining Categorical Features in Machine Learning Models You can create a new feature that is a combination of the other two categorical features. You can also combine more than three or four or even more categorical features. df[ ] = (
	df.feature_1.astype(str)
	 + + df.feature_2.astype(str)
	) "new_feature" "_" In the above code, you can see how you can combine two categorical features by using pandas and form a new feature in your dataset. So which categorical features should you combine? Well, there isn't an easy answer to that. It depends on your data and the types of features. Some domain knowledge might be useful for creating new features like this. To illustrate the whole process, we are going to use the from the , which has many categorical features that we can combine some of these features and see if we can improve the model performance. Financial Inclusion in Africa dataset Zindi competition page The objective of this dataset is to predict who is most likely to have a bank account. So this is a classification problem. 1.Load the dataset Our first step is to make sure that we have downloaded the dataset provided in the competition. You can download the dataset . here Import important python packages. pandas pd numpy np matplotlib.pyplot plt seaborn sns warnings
np.random.seed( )
warnings.filterwarnings( )
%matplotlib inline import as import as import as import as import 123 'ignore' Load the dataset. data = pd.read_csv( ) # Import data 'data/Train_v2.csv' Let’s observe the shape of our dataset. print( , data.shape)

data shape : ( , ) # print shape 'data shape :' 23524 13 The above output shows the number of rows and columns in the dataset. We have 13 variables in the dataset, 12 independent variables, and 1 dependent variable. We can observe the first five rows from our data set by using the method from the pandas library. head() data.head() # inspect data It is important to understand the meaning of each feature so you can really understand the dataset. You can read the file to understand the meaning of each variable presented in the dataset. VariableDefinition.csv 2.Understand The Dataset We can get more information about the features presented by using the method from pandas. info() print(train_data.info()) #show Some information about the dataset The output shows the list of variables/features, sizes if it contains missing values, and data type for each variable. From the dataset, we don’t have any missing values and we have 3 features of integer data type and 10 features of the object data type(most are categorical features). 3. Data preparation for machine learning models The next step is to separate the independent variables and target(bank_account) from the data. Then transform the target values from the object data type into numerical by using . LabelEncoder sklearn.preprocessing LabelEncoder sklearn.preprocessing MinMaxScaler le = LabelEncoder()
data[ ] = le.fit_transform(data[ ]) X = data.drop([ ], axis= )
y = data[ ]

print(y) #import preprocessing module from import from import # Convert target label to numerical Data 'bank_account' 'bank_account' #Separate training features from target 'bank_account' 1 'bank_account' The target values have been transformed into numerical data types, 1 represents ‘Yes’ and 0 represents ‘No’. I have created a simple preprocessing function to: Handle conversion of data types. Convert categorical features to numerical features by using . One-hot Encoder and/or Label Encoder Drop uniqueid variable. Perform . feature scaling float_array = data[[ , , ]].values.astype(float
    ) categ = [ , , , , ,
    ] data = pd.get_dummies(data, prefix_sep= , columns=categ) data[ ] = le.fit_transform(data[ ])
    data[ ] = le.fit_transform(data[ ])
    data[ ] = le.fit_transform(data[ ]) data = data.drop([ ]), axis= ) scaler = StandardScaler()
    data = scaler.fit_transform(data) data # function to preprocess our data : def preprocessing_data (data) # Convert the following numerical labels from interger to float "household_size" "age_of_respondent" "year" # categorical features to be converted to One Hot Encoding "relationship_with_head" "marital_status" "education_level" "job_type" "country" # One Hot Encoding conversion "_" # Label Encoder conversion "location_type" "location_type" "cellphone_access" "cellphone_access" "gender_of_respondent" "gender_of_respondent" # drop uniquid column "uniquid" 1 # scale our data return Let’s preprocess our dataset. processed_test_data = preprocessing_data(X_train) # preprocess the train data 4. Model Building and Experiments A portion of the data set will be used to evaluate our models. # Split train_data sklearn.model_selection train_test_spilt
X_Train, X_val, y_Train, y_val = train_test_split(processed_train_data, y_train, stratify = y, test_size = , random_state= ) from import 0.1 42 Only of the dataset will be used for evaluating the machine learning models. The parameter will ensure an equal balance of values from both classes (‘yes’ and ‘no’) for both train and validation set. 10% stratify = y We will use the for this classification problem to train and predict who is most likely to have a bank account. Logistic Regression algorithm sklearn.linear_model LogisticRegression lg_model = LogisticRegression() lg_model.fit(X_Train,y_Train) #import classifier algorithm here from import # create classifier #Training the classifier After training the classifier, let’s use the trained model to predict our evaluation set and see how it performs. We will use Accuracy as our evaluation metric. # evaluation metrics sklearn.metrics confusion_matrix, accuracy_score

# evaluate the model
y_pred = lg_model.predict(X_val)

# Get the accuracy
print( , .format(accuracy_score(y_val, lg_y_pred))) import from import "Accuracy Score of Logistic Regression classifier: " "{:.4f}" Accuracy Score of Logistic Regression classifier: 0.8874 1st Experiment: Combine education_level and job_type features. Now that we know the basic model performance, let’s see if we can improve it by combining the and features. education_level job_type What we need to do in our first experiment is to update the preprocessing function we have created and then run the rest of the code. float_array = data[[ , , ]].values.astype(float) data[ ] = (data.education_level.astype(str) + + data.job_type.astype(str) ) data = data.drop([ , ], axis= ) categ = [ , , , ] data = pd.get_dummies(data, prefix_sep= , columns=categ) data[ ] = le.fit_transform(data[ ])
    data[ ] = le.fit_transform(data[ ])
    data[ ] = le.fit_transform(data[ ]) data = data.drop([ ], axis= ) scaler = StandardScaler()
    data = scaler.fit_transform(data) data # function to preprocess our data : def preprocessing_data (data) # Convert the following numerical labels from integer to float "household_size" "age_of_respondent" "year" # combine some cat features "features_combination" "_" # remove individual features that are combined together 'education_level' 'job_type' 1 # categorical features to be converted by One Hot Encoding "relationship_with_head" "marital_status" "features_combination" "country" # One Hot Encoding conversion "_" # Label Encoder conversion "location_type" "location_type" "cellphone_access" "cellphone_access" "gender_of_respondent" "gender_of_respondent" # drop uniquid column "uniqueid" 1 # scale our data return In the above preprocessing function I have updated the code by Combine educaion_level and job_type to create a new feature called “ ”. features_combination Remove individual features (education_level and job_type) from the dataset. Add a new feature called “ ” in the list of categorical features to be converted by . feature_combinaton One Hot Encoding I have selected only Nominal categorical features (have more than 2 unique values). Note: After retraining the logistic regression classifier for the 1st experiment, the model performance increased from to . This show that combining categorical feature can improve the model performance. Keep in mind that we did not change anything such as hyper-parameters in your machine learning classifier. 0.8874 0.8882 2nd Experiment: Combine relation_with_head and marital_status features In our second experiment, we are going to combine the other two categorical features which are and . relationship_with_head marital_status We just need to update the preprocessing function (like in the 1st experiment) and then run the rest of the code. float_array = data[[ , , ]].values.astype(
        float
    ) data[ ] = (data.relationship_with_head.astype(str) + + data.marital_status.astype(str) 
                      ) data = data.drop([ , ], axis= ) categ = [ , , , ,
    ] data = pd.get_dummies(data, prefix_sep= , columns=categ) data[ ] = le.fit_transform(data[ ])
    data[ ] = le.fit_transform(data[ ])
    data[ ] = le.fit_transform(data[ ]) data = data.drop([ ], axis= ) scaler = StandardScaler()
    data = scaler.fit_transform(data) data # function to preprocess our data : def preprocessing_data (data) # Convert the following numerical labels from integer to float "household_size" "age_of_respondent" "year" # combine some cat features "features_combination" "_" # remove individual features that are combined together 'relationship_with_head' 'marital_status' 1 # categorical features to be converted by One Hot Encoding "features_combination" "education_level" "job_type" "country" # One Hot Encoding conversion "_" # Label Encoder conversion "location_type" "location_type" "cellphone_access" "cellphone_access" "gender_of_respondent" "gender_of_respondent" # drop uniquid column "uniqueid" 1 # scale our data return In the above preprocessing function I have updated the code by Combine relation_with_head and marital_status to create a new feature called “ ”. features_combination Remove individual features (relation_with_head and marital_status) from the dataset. Add a new feature called “ ” in the list of categorical features to be converted by . feature_combination One Hot Encoding After retraining the logistic regression classifier for the 2nd experiment, the model performance decreased from to . This shows that sometimes when you combine categorical features your machine learning model will not improve as you expected. Therefore you will need to run a lot of experiments until you get satisfactory performance from your machine learning model. 0.8874 0.8865 Wrapping Up In this article, you have learned how to combine categorical features in your dataset in order to improve the performance of your machine learning model. As I have said to get satisfactory performance for your model, you need to have domain knowledge about the problem you are solving. Also, you need to run a lot of experiments that require more computational resources. Congratulations 👏👏, you have made it to the end of this article! I hope you have learned something new that will help you on your next machine learning or data science project. If you learned something new or enjoyed reading this article, please share it so that others can see it. Until then, see you in the next post! You can also find me on Twitter . @Davis_McDavid And you can read more articles like this . here

Target

Twitter

Features Selection by Using Xverse Package

Improve Machine Learning Model Performance by Combining Categorical Features

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

A Developer’s Guide: How to Automate a Financial Statement Analysis for Microfinance

The Noonification: How Often Do NFTs Pass The Howey Test? (1/13/2023)

Darwin's Hybrid Intelligence to Align AI & Human Goals for Startups & VCs

The Noonification: White Man (11/26/2022)

The Noonification: The Metaverse is a Sh*tshow (11/2/2022)

A Developer’s Guide: How to Automate a Financial Statement Analysis for Microfinance

The Noonification: How Often Do NFTs Pass The Howey Test? (1/13/2023)

Darwin's Hybrid Intelligence to Align AI & Human Goals for Startups & VCs

The Noonification: White Man (11/26/2022)

The Noonification: The Metaverse is a Sh*tshow (11/2/2022)

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps