Table of Links Abstract and 1 Introduction Abstract and 1 Introduction Literature Review Model Experiments Deployment Journey Future Directions and References Literature Review Literature Review Model Model Experiments Experiments Deployment Journey Deployment Journey Future Directions and References Future Directions and References 5 DEPLOYMENT JOURNEY In this section, we discuss several user-facing questions we addressed as well as our experience in deploying PCIC. 5.1 Deployment and Online Experience While offline metrics are informative and help us build competent models, the true test of a recommendations model is online, where we can measure impact on user behavior. We deployed PCIC to a production environment where recommendations are generated daily in our compute cluster on an Apache Spark ecosystem and exported to the cloud for real-time serving. When a user visits the site, these recommendations are then served to them, filtered on the item availability based on inventory and available shipment options selected by the user. 5.2 Human-in-the-loop feedback We first rolled out the results to a pool of internal team members for testing. This gave us some feedback as to having an exclusion list of some categories which users may not be very comfortable looking at, in their App (with friends and family or otherwise). Based on the feedback, we built an exclusion list of categories which are applied on top of recommendations as filters. Secondly, we found that users were sometimes recommended an item they’d recently purchased (e.g., a new flavor of yogurt) from a category where they repurchase, but not one they’d like to repurchase. We used a two step approach filter out such items from recommendations. Apart from the category being a repurchase category, we tried to ensure that the item was bought by the guest at least twice in the past n months (n=6). This helps the customer to identify the items in buy it again list as an item they have repeat purchased. Second, we identified items with low repurchase rates (similar to repurchase rate threshold in RCP [2]) and removed them. Several users also noted they typically buy more than one item from a specfic category (e.g., two or more flavors of yogurt) in a single trip. In the backend, we may have a ranked list of categories and ranked list of items within each category. Originally, the PCIC model would round robin among these lists to merge a new list which has first item in each category followed by second item on each category and so on till the list is finished. For a user who purchases more than one item per category every trip, this may be inconvenient. To resolve this, we calculate a variable 𝑁𝐼𝐵 which denotes the number of times the item was purchased by the user per trip. We tweaked the math used to combine the two lists by dividing item rank by NIB in and then taking a ceil function to create new item ranks. 5.3 Metrics To quantify the impact of the proposed PCIC algorithm, we performed A/B tests against existing online baselines. Each test was run for more than two weeks and stopped after ensuring that the samples are statistically significant. The metrics considered for tests are defined as follows: • CTR or Click Through Rate : Percentage of recommendation displays which were clicked by the guest. • Conversion Rate: Percentage of clicked recommendations which were purchased by the guest same day. • Units: The total number of units purchased by the users who were part of the treatment. 5.4 Testing against baseline When we introduced Buy It Again recommendation lists to the guest shopping experience, we A/B tested PCIC against a baseline of FBought. The results are given in Table 5. We can see that there is significant lift across all three metrics - 6% in CTR, 9% in Conversion and 27% in units purchased. 5.5 Testing Buy it Again on web search We tested adding a Buy It Again recommendation list to the search results of all users. For this, we filtered the Buy It Again results using the search query context, so if someone searched for paper towel, the BIA recommendation list would be filtered to show only paper towels. As a sizable fraction of user searches do not pertain to items the user has already purchased, most of the time this recommendation list would not be shown to the guest. We found that the user interaction with this recommendation list was significantly higher than existing search results (by over 20%). As we were testing a recommendation list against a non-existing one, we used visit level metrics to evaluate BIA. It was observed that the add-to-carts, average order values, and units per order went up by 0-2%. We also observed that the guests were able to directly add the items to cart from the recommendation list and subsequently browsed fewer items despite having higher add to carts. These metrics consolidate our belief that showing buy it again items helps the guest in their shopping experience. 5.6 Building virtual aisles Research studies and our internal surveys indicate that online grocery shopping experience for users is significantly different from the typical user store experience [4]. Shopping basket variety is significantly lower for online shopping trips, as measured by the number of unique categories and items purchased. Online grocery shopping environments may accelerate consumer inertia, leading to repurchase of essentials and reduction in purchase of fresh vegetables, impulse purchases such as candy or bakery desserts, and discretionary spending. After identifying these opportunities in online shopping experience development, in 2021, we rolled out BIA to guests by filtering recommendations by categories (Milk, Yogurt, Beauty, etc) to create a virtual aisles experience for online users. We use the personalized list of categories for each guest using PC model. For each category, we present a list of recommended items from IC model to form a virtual aisle. In each aisle, we first showed the BIA items of the guest followed by other relevant items (generated using other algorithms and personalized to each guest, not discussed here for sake of brevity). We rolled out the experience directly to the users and report the lift in experience of guests who used the experience against those who didn’t interact with it. Users who interacted with these recommendations had a significant increase in units per order (25-50%), and average order value (7-35%). Since the buy it again essentials are lower ticket items, they have a smaller dollar impact in order value than units per order. We saw higher engagement of guests with virtual aisles experience for frequency cateogories in the App than in the site. 6 FUTURE DIRECTIONS Buy It Again recommendations help users to quickly complete their shopping missions. Traditional approaches tend to model guest personalized behavior at item granularity. In this paper, we present the case for a coarse grained model which can capture the customer behavior at item category level. The proposed Personalized Category (PC) model combined with Items-within-Category (IC) model outperform existing BIA and NBR models on standard public datasets. The PCIC model also scales well for large retailers with millions sized product catalogs and millions of active guests. The A/B tests on the site show a significant improvement in guest shopping experience and guest spends by using the model. In the future, we would recommend that retailers explore models that combine the insights from Personalized Category features with Personalized Item features. Moreover, we would recommend considering mutual excitation among items and categories as simultaneous consumption has some inherent relationship with repeat consumption. REFERENCES [1] 1959. The Pattern of Consumer Purchases. Journal of the Royal Statistical Society. Series C (Applied Statistics) 8, 1 (1959), 26–41. http://www.jstor.org/stable/2985810 [2] Rahul Bhagat, Srevatsan Muralidharan, Alex Lobzhanidze, and Shankar Vishwanath. 2018. Buy it again: Modeling repeat purchase recommendations. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 62–70. [3] Chris Chatfield and Gerald Goodhardt. 1973. A Consumer Purchasing Model with Erlang Inter-Purchase Times. J. Amer. Statist. Assoc. 68 (1973), 828–835. [4] Sai Chand Chintala, Jura Liaukonyte, and Nathan Yang. 2022. Browsing the Aisles or Browsing the App? How Online Grocery Shopping Is Changing What We Buy. How Online Grocery Shopping is Changing What We Buy (August 8, 2022) (2022). [5] D. R. Cox. 1972. Regression Models and Life Tables. Journal of the Royal Statistical Society 34 (1972), 187–220. [6] Suvodip Dey, Pabitra Mitra, and Kratika Gupta. 2016. Recommending Repeat Purchases Using Product Segment Statistics. In Proceedings of the 10th ACM Conference on Recommender Systems (Boston, Massachusetts, USA) (RecSys ’16). Association for Computing Machinery, New York, NY, USA, 357–360. https: //doi.org/10.1145/2959100.2959145 [7] P S Fader, B G Hardie, and K Lee. 2009. Probability Models for Customer-Base Analysis. Journal of Interactive Marketing 23 (2009). [8] Sofia Gomes and João M Lopes. 2022. Evolution of the online grocery shopping experience during the COVID-19 Pandemic: Empiric study from Portugal. Journal of Theoretical and Applied Electronic Commerce Research 17, 3 (2022), 909–923. [9] Gary L. Grahn. 1969. NBD Model of Repeat-Purchase Loyalty: An Empirical Investigation. Journal of Marketing Research 6 (1969), 72 – 78. [10] Ruining He, Wang-Cheng Kang, Julian J McAuley, et al. 2018. Translation-based Recommendation: A Scalable Method for Modeling Sequential Behavior.. In IJCAI. 5264–5268. [11] Ruining He and Julian McAuley. 2016. Fusing similarity models with markov chains for sparse sequential recommendation. In 2016 IEEE 16th international conference on data mining (ICDM). IEEE, 191–200. [12] Haoji Hu and Xiangnan He. 2019. Sets2sets: Learning from sequential sets with neural networks. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 1491–1499. [13] Haoji Hu, Xiangnan He, Jinyang Gao, and Zhi-Li Zhang. 2020. Modeling personalized item frequency information for next-basket recommendation. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. 1071–1080. [14] Komal Kapoor, Karthik Subbian, Jaideep Srivastava, and Paul R. Schrater. 2015. Just in Time Recommendations: Modeling the Dynamics of Boredom in Activity Streams. Proceedings of the Eighth ACM International Conference on Web Search and Data Mining (2015). [15] Komal Kapoor, Mingxuan Sun, Jaideep Srivastava, and Tao Ye. 2014. A hazard based approach to user return time prediction. Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (08 2014). https://doi.org/10.1145/2623330.2623348 [16] Joseph A Konstan, Bradley N Miller, David Maltz, Jonathan L Herlocker, Lee R Gordon, and John Riedl. 1997. Grouplens: Applying collaborative filtering to usenet news. Commun. ACM 40, 3 (1997), 77–87. [17] Yehuda Koren. 2009. Collaborative filtering with temporal dynamics. In Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining. 447–456. [18] Pengjie Ren, Zhumin Chen, Jing Li, Zhaochun Ren, Jun Ma, and Maarten De Rijke. 2019. Repeatnet: A repeat aware neural recommendation machine for sessionbased recommendation. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33. 4806–4813. [19] Steffen Rendle, Christoph Freudenthaler, and Lars Schmidt-Thieme. 2010. Factorizing personalized markov chains for next-basket recommendation. In Proceedings of the 19th international conference on World wide web. 811–820. [20] Haochao Ying, Fuzhen Zhuang, Fuzheng Zhang, Yanchi Liu, Guandong Xu, Xing Xie, Hui Xiong, and Jian Wu. 2018. Sequential recommender system based on hierarchical attention network. In IJCAI International Joint Conference on Artificial Intelligence. [21] Feng Yu, Qiang Liu, Shu Wu, Liang Wang, and Tieniu Tan. 2016. A dynamic recurrent model for next basket recommendation. In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval. 729–732. Authors: (1) Amit Pande, Data Sciences, Target Corporation, Brooklyn Park, Minnesota, USA (amit.pande@target.com); (2) Kunal Ghosh, Data Sciences, Target Corporation, Brooklyn Park, Minnesota, USA (kunal.ghosh@target.com); (3) Rankyung Park, Data Sciences, Target Corporation, Brooklyn Park, Minnesota, USA (rankyung.park@target.com). Authors: Authors: (1) Amit Pande, Data Sciences, Target Corporation, Brooklyn Park, Minnesota, USA (amit.pande@target.com); (2) Kunal Ghosh, Data Sciences, Target Corporation, Brooklyn Park, Minnesota, USA (kunal.ghosh@target.com); (3) Rankyung Park, Data Sciences, Target Corporation, Brooklyn Park, Minnesota, USA (rankyung.park@target.com). This paper is available on arxiv under ATTRIBUTION-SHAREALIKE 4.0 INTERNATIONAL license. This paper is available on arxiv under ATTRIBUTION-SHAREALIKE 4.0 INTERNATIONAL license. available on arxiv available on arxiv
