We all know that loyalty programs and reward systems like cashback in e-commerce are great tools to encourage repeat purchases. We aimed to increase the frequency of purchases by paying our premium users cashback earlier than they normally do within 15 days. However, this well-intentioned move led to an unexpected result: Increased Fraud! 😫 In this article, I will talk about our A/B test that we ran and the unexpected result we observed in user behaviour. Why We Ran This A/B Test? 👀 Making data-driven decisions is crucial in e-commerce. We had been receiving user feedback stating that cashback processing times were too long and that customers often forgot to use their cashback due to the delayed payout period. These concerns led us to explore ways to optimise cashback frequency to make the program more effective. We aimed to evaluate whether adjusting cashback timing could positively influence shopping frequency without negatively impacting business outcomes. Our Hypothesis 📚 Faster cashback payments would encourage users to shop more frequently, ultimately increasing engagement and revenue. Objective of the Test: Understand the relationship between the timing of users’ cashback purchases and their purchase frequency


See if users who receive cashback more frequently use the platform


Increase sales volume Steps and Decision-Making Process:

Initial Data Analysis: We examined past cashback transaction data to identify patterns in user behaviour.
Considering Refund Policy: Since our company’s refund policy allowed returns within 15 days, cashback timing was aligned with this restriction to minimise refund exploitation.
Test Groups:

Control Group: Cashback payments every 15 days (standard practice).
Test Group 1: Cashback credited on the same day.
Test Group 2: Cashback credited every 7 days.


Test Duration: We set a time frame to observe changes in behaviour and measure key outcomes.
Key Metrics: We tracked user transaction frequency, average order value, refund rates, and fraud attempts to assess the impact of different cashback timing strategies However, the findings we obtained at the end of the test surprised us. Why Did Users Start Frauding? 🐍 When examining the results of the A/B test, we noticed the following: Some users in Test Group 1 and 2 began manipulating our system to earn cashback rewards illegitimately.
An increase in small, frequent purchases, order completion followed by cancellations, and refund exploitation.
Certain users would receive cashback and then cancel their transactions to extract additional money from the system.
A significant rise in refund rates indicated that users were attempting to game the system by returning purchases after receiving cashback.
An increase in daily transactions per user, likely driven by users attempting to maximise cashback payouts. The biggest fraud risk emerged in the group where cashback was deposited on the same day. Users saw the incentive offered by the system as an "open door" and began to abuse it. What Can We Do to Prevent Fraud? This unexpected situation showed that we need to consider fraud risks when planning the timing of cashback payments. We evaluated the following solutions: Cashback Limits: Determining the maximum cashback amount in a certain period.
Fraud Detection Systems: Using AI-based systems to identify anomalies in the user's shopping habits.
Strict Order and Return Controls: Examining users who make a large number of orders and cancellations in a short period of time.
Different Cashback Structure: Making cashback payments more balanced and spread over time. Learning from Unexpected Outcomes 🙃 This experience taught us that every user incentive should be designed with potential exploitation risks in mind. Before launching any program, we need to ask ourselves: “How could users manipulate this system for unintended gains?” Otherwise, a well-intentioned improvement could end up backfiring!  🚀 Thank you for your time; sharing is caring! 🌍 We all know that loyalty programs and reward systems like cashback in e-commerce are great tools to encourage repeat purchases. We aimed to increase the frequency of purchases by paying our premium users cashback earlier than they normally do within 15 days. However, this well-intentioned move led to an unexpected result: Increased Fraud! 😫 Increased Fraud! 😫 Increased Fraud! 😫 In this article, I will talk about our A/B test that we ran and the unexpected result we observed in user behaviour. Why We Ran This A/B Test? 👀 Why We Ran This A/B Test? 👀 Making data-driven decisions is crucial in e-commerce. We had been receiving user feedback stating that cashback processing times were too long and that customers often forgot to use their cashback due to the delayed payout period . These concerns led us to explore ways to optimise cashback frequency to make the program more effective. cashback processing times were too long forgot to use their cashback due to the delayed payout period We aimed to evaluate whether adjusting cashback timing could positively influence shopping frequency without negatively impacting business outcomes. Our Hypothesis 📚 Our Hypothesis 📚 Faster cashback payments would encourage users to shop more frequently, ultimately increasing engagement and revenue. Faster cashback payments would encourage users to shop more frequently, ultimately increasing engagement and revenue. Faster cashback payments would encourage users to shop more frequently, ultimately increasing engagement and revenue. Objective of the Test: Objective of the Test: Objective of the Test: Understand the relationship between the timing of users’ cashback purchases and their purchase frequency See if users who receive cashback more frequently use the platform Increase sales volume Understand the relationship between the timing of users’ cashback purchases and their purchase frequency Understand the relationship between the timing of users’ cashback purchases and their purchase frequency See if users who receive cashback more frequently use the platform See if users who receive cashback more frequently use the platform Increase sales volume Increase sales volume Steps and Decision-Making Process:

Initial Data Analysis: We examined past cashback transaction data to identify patterns in user behaviour.
Considering Refund Policy: Since our company’s refund policy allowed returns within 15 days, cashback timing was aligned with this restriction to minimise refund exploitation.
Test Groups:

Control Group: Cashback payments every 15 days (standard practice).
Test Group 1: Cashback credited on the same day.
Test Group 2: Cashback credited every 7 days.


Test Duration: We set a time frame to observe changes in behaviour and measure key outcomes.
Key Metrics: We tracked user transaction frequency, average order value, refund rates, and fraud attempts to assess the impact of different cashback timing strategies Steps and Decision-Making Process: Initial Data Analysis: We examined past cashback transaction data to identify patterns in user behaviour.
Considering Refund Policy: Since our company’s refund policy allowed returns within 15 days, cashback timing was aligned with this restriction to minimise refund exploitation.
Test Groups:

Control Group: Cashback payments every 15 days (standard practice).
Test Group 1: Cashback credited on the same day.
Test Group 2: Cashback credited every 7 days.


Test Duration: We set a time frame to observe changes in behaviour and measure key outcomes.
Key Metrics: We tracked user transaction frequency, average order value, refund rates, and fraud attempts to assess the impact of different cashback timing strategies Steps and Decision-Making Process: Steps and Decision-Making Process: Initial Data Analysis: We examined past cashback transaction data to identify patterns in user behaviour. Considering Refund Policy: Since our company’s refund policy allowed returns within 15 days, cashback timing was aligned with this restriction to minimise refund exploitation. Test Groups:

Control Group: Cashback payments every 15 days (standard practice).
Test Group 1: Cashback credited on the same day.
Test Group 2: Cashback credited every 7 days. Test Duration: We set a time frame to observe changes in behaviour and measure key outcomes. Key Metrics: We tracked user transaction frequency, average order value, refund rates, and fraud attempts to assess the impact of different cashback timing strategies Initial Data Analysis: We examined past cashback transaction data to identify patterns in user behaviour. Initial Data Analysis: Considering Refund Policy: Since our company’s refund policy allowed returns within 15 days, cashback timing was aligned with this restriction to minimise refund exploitation. Considering Refund Policy: Test Groups: Control Group: Cashback payments every 15 days (standard practice).
Test Group 1: Cashback credited on the same day.
Test Group 2: Cashback credited every 7 days. Test Groups: Control Group: Cashback payments every 15 days (standard practice). Test Group 1: Cashback credited on the same day. Test Group 2: Cashback credited every 7 days. Control Group: Cashback payments every 15 days (standard practice). Control Group: Test Group 1: Cashback credited on the same day. Test Group 1: Test Group 2: Cashback credited every 7 days. Test Group 2: Test Duration: We set a time frame to observe changes in behaviour and measure key outcomes. Test Duration: Key Metrics: We tracked user transaction frequency, average order value, refund rates, and fraud attempts to assess the impact of different cashback timing strategies Key Metrics: However, the findings we obtained at the end of the test surprised us. Why Did Users Start Frauding? 🐍 Why Did Users Start Frauding? 🐍 When examining the results of the A/B test, we noticed the following: Some users in Test Group 1 and 2 began manipulating our system to earn cashback rewards illegitimately. An increase in small, frequent purchases, order completion followed by cancellations, and refund exploitation. Certain users would receive cashback and then cancel their transactions to extract additional money from the system. A significant rise in refund rates indicated that users were attempting to game the system by returning purchases after receiving cashback. An increase in daily transactions per user, likely driven by users attempting to maximise cashback payouts. Some users in Test Group 1 and 2 began manipulating our system to earn cashback rewards illegitimately. An increase in small, frequent purchases, order completion followed by cancellations, and refund exploitation. Certain users would receive cashback and then cancel their transactions to extract additional money from the system. A significant rise in refund rates indicated that users were attempting to game the system by returning purchases after receiving cashback. An increase in daily transactions per user, likely driven by users attempting to maximise cashback payouts. The biggest fraud risk emerged in the group where cashback was deposited on the same day. Users saw the incentive offered by the system as an "open door" and began to abuse it. The biggest fraud risk emerged in the group where cashback was deposited on the same day. Users saw the incentive offered by the system as an "open door" and began to abuse it. What Can We Do to Prevent Fraud? What Can We Do to Prevent Fraud? This unexpected situation showed that we need to consider fraud risks when planning the timing of cashback payments. We evaluated the following solutions: Cashback Limits: Determining the maximum cashback amount in a certain period. Fraud Detection Systems: Using AI-based systems to identify anomalies in the user's shopping habits. Strict Order and Return Controls: Examining users who make a large number of orders and cancellations in a short period of time. Different Cashback Structure: Making cashback payments more balanced and spread over time. Cashback Limits: Determining the maximum cashback amount in a certain period. Cashback Limits: Fraud Detection Systems: Using AI-based systems to identify anomalies in the user's shopping habits. Fraud Detection Systems: Strict Order and Return Controls: Examining users who make a large number of orders and cancellations in a short period of time. Strict Order and Return Controls: Different Cashback Structure: Making cashback payments more balanced and spread over time. Different Cashback Structure: Learning from Unexpected Outcomes 🙃 This experience taught us that every user incentive should be designed with potential exploitation risks in mind. Before launching any program, we need to ask ourselves: “How could users manipulate this system for unintended gains?” “How could users manipulate this system for unintended gains?” Otherwise, a well-intentioned improvement could end up backfiring!  🚀 Thank you for your time; sharing is caring! 🌍

This story contains new, firsthand information uncovered by the writer.

Walkthroughs, tutorials, guides, and tips. This story will teach you how to do something new or how to do something better.

This post provides insights into new product. 

A Well-intentioned Cashback Program Caused an Increase in Fraud—Here's What Happened

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

7 Open Source Projects Every Data Scientist/Analyst Needs to Bookmark 🚀

Website Conversion Rate Optimisation In 4 Quick Steps

For Effective A/B/n Testing, Try Running Them Simultaneously

4 Lessons We Learned in 2019 (and How Marketers Can Apply Them in 2020)

A/B Testing For Email Marketing — What Should You Test For?

A/B Testing Our Way to a Better Payment Gateway

7 Open Source Projects Every Data Scientist/Analyst Needs to Bookmark 🚀

Website Conversion Rate Optimisation In 4 Quick Steps

For Effective A/B/n Testing, Try Running Them Simultaneously

4 Lessons We Learned in 2019 (and How Marketers Can Apply Them in 2020)

A/B Testing For Email Marketing — What Should You Test For?

A/B Testing Our Way to a Better Payment Gateway

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps