In this article, I’ll explain how Precision, Recall, F-score, and ROC-AUC are calculated, with real examples. We’ve already figured out that many model-quality metrics are derived from the Confusion Matrix, while Accuracy is used rather rarely (even though it’s very popular among clients who don’t really understand machine learning). So, what else can we calculate based on this matrix? Confusion Matrix Precision & Recall Precision = TP / (TP + FP) Precision = TP / (TP + FP) Precision shows the share of objects the model marked as positive that are actually positive. Recall = TP / (TP + FN) Recall = TP / (TP + FN) Recall shows what portion of all true positive-class objects the algorithm managed to find. These metrics reflect model quality more accurately than Accuracy, but they also have some drawbacks. For example, you can change both of them simply by shifting the probability threshold for class assignment. F-Score Sometimes, all these metrics can get confusing, and machine-learning algorithms are usually built so they optimize just one specific quality metric. That’s why people came up with metrics that reflect both Precision and Recall at the same time — these are called F-scores. There are several types of F-scores, but here I’ll explain the most basic one, the F1 score: F1 = 2 * (Precision * Recall) / (Precision + Recall) F1 = 2 * (Precision * Recall) / (Precision + Recall) So, F1 shows both how well the model finds positive-class objects among all true positives, and what share of the objects the algorithm labeled as positive are actually positive. ROC-AUC Probably the most popular metric for binary classification is ROC-AUC. It doesn’t depend on choosing a probability threshold and works well on imbalanced datasets. Let’s break down how it’s computed step by step using a simple example about buying an elephant. Say we have 10 people; 4 of them will buy an elephant, and 6 will not. Our model predicts the probability of buying an elephant for each person. We now have each person’s buying probability and the actual outcome. By counting the number of positive outcomes (there are 4) and negative ones (6), we draw a grid that is 4 cells high and 6 cells wide. Its total area is 24. Next, we sort the probabilities from highest to lowest. Then we move through them one by one: if the person actually bought the elephant, we take one step up; if not, we take one step right. After finishing the actual sequence, we continue placing the remaining steps to the right until the grid is filled. Here’s what happens: up right First step: Upward — the model predicted correctly (low probability of buying an elephant, and the guy actually didn’t buy it). First step: Second step: Upward again — another correct prediction (again, low probability of buying an elephant, and the guy actually didn’t buy it). Second step: Third step: This time the model was wrong — we moved right (we have a low probability of purchase, but the man actually bought an elephant). Third step: Fourth step: correct — we move up. Following this logic, we fill the entire grid. After placing all steps, we fill all cells below with 1. Then we count them: we get 19. Now, we divide 19 by the total number of cells (4 * 6 = 24): ROC-AUC = 19 / 24 = 79% ROC-AUC = 19 / 24 = 79% Conclusion In the next article, I’ll continue with AI metrics and move on to regression metrics such as MSE, RMSE, MAR, R-squared, MAPE, and SMAPE. Follow me — check my profile for links!

Part of HackerNoon's growing list of open-source research papers, promoting free access to academic material.

ApeCoin

AI Metrics Made Simple: Precision, Recall, F-Score, and ROC-AUC

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Confusion Matrix Explained: The Real Foundation of Model Evaluation

The Noonification: How Often Do NFTs Pass The Howey Test? (1/13/2023)

Darwin's Hybrid Intelligence to Align AI & Human Goals for Startups & VCs

The Noonification: White Man (11/26/2022)

The Noonification: The Metaverse is a Sh*tshow (11/2/2022)

100 Days of AI Day 1: From Newsletter to Podcast, Leveraging AI for Audio Transformation

Confusion Matrix Explained: The Real Foundation of Model Evaluation

The Noonification: How Often Do NFTs Pass The Howey Test? (1/13/2023)

Darwin's Hybrid Intelligence to Align AI & Human Goals for Startups & VCs

The Noonification: White Man (11/26/2022)

The Noonification: The Metaverse is a Sh*tshow (11/2/2022)

100 Days of AI Day 1: From Newsletter to Podcast, Leveraging AI for Audio Transformation

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps