A cutting-edge data science model can only be created if the impact is measured properly. Pinterest upgraded everyone's preferred impact measurement metric, CTR, after carefully analyzing the problems. Every Data person adores the CTR measure as it is: Simple to explain to leadership, product managers do not question it, and measuring it is straightforward: Divide the number of clicks by the total number of impressions. CTR data is easily available, so it will be easy to calculate, track, and analyze. And, as a bonus, lots of advertisers want to have high CTRs too. So, you’ve aligned your advertiser and user interests! So, just throw a lot of advanced machine learning at the problem to predict and maximize CTRs and you’re done, right? Not so fast! CTR Suffers From Some Serious Shortcomings. “CTR Is Not the Key to User Engagement.” CTR just reports on clicks on a certain item, regardless of whether it is relevant or not. Some may claim that the user clicked because the content or ad was relevant, however, this is not always the case; it could be due to positional bias, clickbait, or other factors. These flaws will be discussed in further depth below. CTR is only useful for measuring short-term gains without consideration of user engagement. As a result, we require a metric to track user engagement. “You can fool all the people some of the time and some of the people all the time but you cannot fool all the people all the time” - Abraham Lincoln Problem 1 - Positional Bias Ads higher up on the page are more likely to be clicked, but this does not imply that the user finds them more valuable. If you fail to take this into account, you can conclude that users prefer the ads at the top of the page when, in reality, they are just more inclined to pay attention to them. Problem 2 - CTR Promotes Clickbait Items having a high click-through rate (CTR) may be "clickbaity" — they entice the user to click without providing anything helpful, instead sending them to ad-loaded slideshows that generate revenue for the advertiser but annoy the user. Problem 3 - CTR Ignores Other Signals CTR doesn't imply positive interaction. The person may still despise that specific piece of content. Hide Ads or drop-offs are powerful hate signals that CTR does not take into account. Above is a Google Ads example to capture explicit user dislike feedback. No point in showing a strong dislike signal ad with high CTR; it will eventually put the user off the platform someday. Problem 4 - Some Things Are Not Meant to Be Clicked Ads that do not require a click, such as a video, brand awareness, or pitch ads, such as an Independence Day discount at a nearby store or a short video of a new car launch. Okay..! We have now listed the problems with CTR but what is the Solution? Going Beyond CTR They adjusted their ads measurement metric to address the difficulties with CTR mentioned previously. A metric that incorporates engagement and more accurately captures user experience. User Metric: Let's discuss the Numerator first - Weighted Engagement on Ads Instead of just looking at CTR, we look at a weighted average of different actions — Clicks, Hides (Stop ads), Watch Time (on Video Ads), and Saves, among others — on the ads. Consider two ads: B is clicked more often than A, but it is hidden by a lot more people. One can readily conclude that ad type A is superior to ad type B because it receives more engagement and fewer users hide or disconnect with it while CTR alone shows the opposite picture. Conclusion - If we choose the actions and weights carefully, we can address concerns 2, 3, and 4 with CTR. Coming to the Denominator - Weighted Engagement on Neighbouring Organic Content To address Problem 1 (Position Bias) - We need to dig a bit deeper and understand that the CTRs drop on ads and organic content as users scroll through. One can verify this hypothesis of positional bias issue with A/B testing on the platform as well. LEARNING (This stands true in the majority of businesses) - If an ad or Feed Content on the 10th Position of the carousel gets a similar CTR to one on the 1st position, this means the ad on the 10th position is better. To normalize the effect of positional bias on ads content, we compare the average engagement rate on ads to that of the organic content in the spots before and after. This is where the denominator comes in. Experimenting with Above setup and New Ads Measuring Metric From the above table, we can infer that Treatment has lower CTR than Control but a lot better on our Weighted Engagement metric. Conclusion I understand that digesting and practically applying this new metric can be difficult because CTR has been used to gauge ads and content relevance for a long time, but change brings growth. If your ads monetization model is purely dependent on CTR, you can place a higher weight on Clicks when taking weighted engagement scores, but deleting engagement aspects and relying solely on CTR would make users unhappy in the long run. NOTE: Reference A better clickthrough rate: How Pinterest upgraded everyone’s favorite engagement metric by Philip Apps Is there anything you would like to add? Please feel free to share your views and feedback in the comments section below. Connect, Follow or Endorse me on if you found this read useful. To learn more about me: visit LinkedIn Here This newsletter is now read by more than 4500+ subscribers. If you are building an AI or a data product or service, you are invited to become a sponsor of one of the future newsletter issues. Feel free to reach out to shauryauppal97@gmail.com for more details on sponsorships. I am nominated for the HackerNoon 2022 Noonies, Vote for me: https://www.noonies.tech/2022/programming/2022-hackernoon-contributor-of-the-year-data Connect 1:1 Meeting here: https://topmate.io/shaurya Data Science Book Recommendations: [1] The Book of Why [2] Naked Statistics Also published here

Google

Naked

Setting Alerts With 2 Additional Lines

Using Data Science To Deal With RTOs

Master Data Science Skills

Nominated for 2022 - Best Data Science Newsletter

Nominated for 2022 - HackerNoon Contributor of the Year - Data

Nominated for 2022 - Data Science Demon

Nominated for 2022 - HackerNoon Contributor of the Year - Data Science

Nominated for 2022 - HackerNoon Contributor of the Year - India

Too Long; Didn't Read

Is the Pinterest Way to Measure Ads the Right Way?

Is the Pinterest Way to Measure Ads the Right Way?

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Beyond A/B Testing — Switchbacks and Synthetic Control Group

06/06/2018: Biggest Stories in the Cryptosphere

10 Ways Fake News is Killing The Economy : From Anti-Vaxxers To The Stock Market

10 Reasons to Use PPC Advertising for Your Independent Film

10 facts about Mobile Rewarded Surveys

1 Star Review? Make It Your Next Marketing Campaign.

Beyond A/B Testing — Switchbacks and Synthetic Control Group

06/06/2018: Biggest Stories in the Cryptosphere

10 Ways Fake News is Killing The Economy : From Anti-Vaxxers To The Stock Market

10 Reasons to Use PPC Advertising for Your Independent Film

10 facts about Mobile Rewarded Surveys

1 Star Review? Make It Your Next Marketing Campaign.

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps