When it comes to uplift modeling, traditional performance metrics commonly used for other machine learning tasks may fall short. The standard machine learning algorithms / business cases learn on the training data, predict the target on the test data and compare it to the ground truth. However, in uplift modeling, the concept of ground truth becomes elusive since we cannot observe the impact of being treated and not treated on an individual simultaneously. How to choose validation dataset ? The choice of data for training and testing an uplift model depends on the available information and the specific context. Uplift models are commonly used for marketing campaigns. Let’s illustrate how validation data is chosen from this perspective. If we have a single campaign, we can divide the customers within that campaign into training and validation sets. However, if there are multiple campaigns available, we can utilize some campaigns for training the model and reserve others for validation. This strategy allows the model to learn from a broader range of scenarios and potentially improves its generalization capabilities. Data used for uplift modeling should have sufficient information about the treatment, control groups, and relevant covariates. Data used for uplift modeling should have sufficient information about the treatment, control groups, and relevant covariates. Without these essential components, accurately capturing uplift becomes challenging. The main approaches There are two main ways to assess the performance of an uplift model: Cumulative Gain and Qini. Let’s explore them: Cumulative Gain : The cumulative gain illustrates the incremental response rate or outcome achieved by targeting a specific percentage of the population. To calculate cumulative gain, the individuals are ranked based on their uplift scores, and the sorted list is divided into a series of equal-sized deciles or percentile groups. The cumulative gain is then computed by summing up the outcomes or responses of individuals within each group. N : number of clients for control (C) and treatment (T) groups for the first p% of the clients N Y : Sum of our uplift in a metric we chose for control (C) and treatment (T) groups for the first p% of the clients Y For instance, CG at 20% of population targeted corresponds to the total incremental gain if we treat only the instances with top 20% highest scores. highest scores In the example provided below, we observe that targeting the top 20% of clients with the highest scores yields a cumulative gain of 0.019. A steeper curve indicates a better model, as it shows that a higher proportion of individuals with the highest predicted uplift are being targeted. Qini Coefficient: The Qini coefficient works on the same idea as the Cumulative Gain, with one key distinction. The Qini coefficient aims to penalize models that overestimate treatment samples and underestimate control samples. The Qini coefficient aims to penalize models that overestimate treatment samples and underestimate control samples. The Qini coefficient aims to penalize models that overestimate treatment samples and underestimate control samples. The formula to calculate it: That’s great but how we are going to choose between different models ? Relying solely on these curves to choose between different models might not be the most data-driven approach. The quality metrics There are three the most useful metrics that can help us and all of them are applicable to both Qini and Cumulative Gain approaches. applicable to both Qini and Cumulative Gain approaches Area under Uplift (AUC-U): Similar to the area under the ROC curve (AUC-ROC) in traditional classification, the AUC-U measures the overall performance of an uplift model. It calculates the area under the uplift / Qini curve, which represents the cumulative uplift along individuals sorted by uplift model predictions. Uplift@K: Uplift@K focuses on identifying the top K% of the population with the highest predicted uplift. It measures the proportion of truly responsive individuals within this selected group. A higher uplift@K value indicates a better model at targeting the right individuals. In the example below Uplift@0.2 for the first model is roughly 0.16 and for the second model is 0.19 , and the choice of the best model is obvious. When this metric can help ? When it comes to the real business cases, we often encounter constraints imposed by our stakeholders. Sometimes ,we can only send notifications or campaigns to limited number of people , for example only 10k people.This is precisely when the Uplift@K metric comes to our rescue, making it the obvious choice. When it comes to the real business cases, we often encounter constraints imposed by our stakeholders. Sometimes ,we can only send notifications or campaigns to limited number of people , for example only 10k people.This is precisely when the Uplift@K metric comes to our rescue, making it the obvious choice. When it comes to the real business cases, we often encounter constraints imposed by our stakeholders. Sometimes ,we can only send notifications or campaigns to limited number of people , for example only 10k people.This is precisely when the Uplift@K metric comes to our rescue, making it the obvious choice. Uplift max: Uplift max refers to the maximum uplift achieved by the model. It represents the difference between the treated and control groups with the highest uplift scores. Conclusion We have witnessed that traditional classification and regression metrics may not adequately measure uplift models’ effectiveness. To overcome this, two primary approaches, CG and Qini, offer valuable metrics for evaluation. However, the choice of metrics ultimately depends on your specific business goals. However, the choice of metrics ultimately depends on your specific business goals. However, the choice of metrics ultimately depends on your specific business goals. It is crucial to continuously experiment with different variations and find the metrics that align best with your objectives. By exploring and refining your approach, you can effectively measure the impact of uplift models and optimize their performance.

Why “Accuracy” Fails for Uplift Models (and What to Use Instead)

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

How to Build Connections for A/B Testing and Linear Regression: An Essential Guide

10 Ways to Optimize Your Database

10 Best React Native Chart Libraries

10 Best Datasets for Time Series Analysis

12 Mistakes that Data Scientists Make and How to Avoid Them

13 Best Datasets for Power BI Practice

How to Build Connections for A/B Testing and Linear Regression: An Essential Guide

10 Ways to Optimize Your Database

10 Best React Native Chart Libraries

10 Best Datasets for Time Series Analysis

12 Mistakes that Data Scientists Make and How to Avoid Them

13 Best Datasets for Power BI Practice

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps