Evaluation metrics for classification models Building Machine Learning models is fun, making sure we build the best ones is what makes a difference! Evaluating ML models Regression models RMSE is a good measure to evaluate how a model is performing. machine learning If RMSE is significantly higher in test set than training-set — There is a good chance model is overfitting.(Make sure train and test set are from same/similar distribution) What about Classification models? Guess what, evaluating a Classification model is not that simple But why? You must be wondering ‘Can’t we just use accuracy of the model as the holy grail metric?’ Accuracy is very important, but it might not be the best metric all the time. Let’s look at why with an example -: Let’s say we are building a model which predicts if a bank loan will default or not (The S&P/Experian Consumer Credit Default Composite Index reported a default rate of 0.91%) Let’s have a dummy model which always predicts that a loan will not default. Guess what would be the accuracy of this model? ===> 99.10% Impressive, right? Well, the probability of a bank buying this model is absolute zero. 😆 While our model has a stunning accuracy, this is an apt example where accuracy is definitely not the right metric. If not accuracy, what else? Along with accuracy, there are a bunch of other methods to evaluate the performance of a classification model Confusion , matrix Precision, Recall ROC and AUC Before moving forward, we will look into some terms which will be constantly repeated and might make the whole thing an incomprehensible maze if not understood clearly. I clearly remember when I came across the concept of Confusion Matrix for the first time. I saw the word and I felt like this dude below False Positive Easy right? Well, not the same feeling after I saw all these 🤔 But then as they say — Every cloud has a silver lining Let’s understand it one by one, starting with the fundamental terms. The Positives and Negatives — TP, TN, FP, FN I use this hack to remember the meaning of each of these correctly. (Binary classification problem. Ex — Predicting if a bank loan will default) So what is the meaning of a ? True Negative : We were right when we predicted that a loan will not default. True Negative : We falsely predicted that a loan will default. False Positive Lets reinforce what we learnt Another picture which stamps it in my mind Confusion Matrix As now we are familiar with TP, TN, FP, FN — It will be very easy to understand what confusion matrix is. It is a summary table showing how good our model is at predicting examples of various classes. Axes here are predicted-lables vs actual-labels. Confusion matrix for a classification model predicting if a loan will default or not Precision and Recall Also called Positive predictive valueThe ratio of correct positive predictions to the total predicted positives Precision — Also called Sensitivity, Probability of Detection, True Positive Rate Recall — Ratio of correct positive predictions to the total positives examples To understand and , let’s take an example of Search. Think about the search box on Amazon home page. Understanding Precision Recall The is the proportion of relevant results in the list of all returned search results. The is the ratio of the relevant results returned by the search engine to the total number of the relevant results that could have been returned. precision recall In our case of predicting if a loan would be default — It would be better to have a high Recall as the banks don’t want to lose money and would be a good idea to alarm the bank even if there is a slight doubt about defaulter. Low precision in this case might be okay. : Mostly we have to pick one over other, it’s almost impossible to have both high Precision and Recall. Note Accuracy Talking about accuracy, our favourite metric! Accuracy is defined as the ratio of correctly predicted examples by the total examples. In terms of confusion matrix it is given by: Remember, accuracy is a very useful metric when all the classes are equally important. But this might not be the case if we are predicting if a patient has cancer. In this example, we can probably tolerate FPs but not FNs. ROC curve An ROC curve(receiver operating characteristic curve) graph shows the performance of a classification model at all classification thresholds. (Using thresholds : Say, if you want to compute TPR and FPR for the threshold equal to 0.7, you apply the model to each example, get the score, and, if the score if higher than or equal to 0.7, you predict the positive class; otherwise, you predict the negative class) It plots 2 parameters: (Recall) True positive rate False Positive rate Tells what % of people who were not defaulter were identified as defaulter A typical ROC curve Lowering the classification threshold classifies more items as positive, thus increasing both False Positives and True Positives. AUC stands for It provides an aggregate measure of performance across all possible classification thresholds. AUC Area under the ROC Curve. The higher the (AUC), the better the classifier. A perfect classifier would have an AUC of 1. Usually, if your model behaves well, you obtain a good classifier by selecting the value of the threshold that gives TPR close to 1 while keeping FPR near 0. area under the ROC curve In this post we saw how a classification model can be effectively evaluated, specially in the situations where looking at standalone accuracy is not enough. We understood concepts like TP, TN, FP, FN, Precision, Recall, Confusion matrix, ROC and AUC. Hope it made things clearer! Always up for a discussion and constructive feedback! Twitter @jnvipul
