The future of e-commerce lies in refined operation, as all industry players are leveraging data analytics to group their customers and provide tailored services. This is especially the case with Lifease, an e-commerce brand targeting middle-class consumers. In this post, I write about how we carry out refined operations, based on our own Data Management Platform (DMP). Let's start from the basics. As the raw material of our DMP, the data sources of Lifease include: Business logs from all sales ends； Sales data from third-party platforms; Basic data from within the company. These constitute our data assets, from which we derive a number of tags to describe the customers' age, address, preferred products, the device they use, etc. Using these tags as filters, we pick out a group of customers that match certain characteristics (the "Grouping" process). Then we observe the pattern of the target group. What is the DMP used for? From the data user's perspective, they mainly use the DMP for two purposes: tag query and grouping. : Sometimes they find a certain customer (or a group of customers) and check what tags they are attached to. Tag Query : After grouping, they might want to check to see if a certain customer is among a specified group, in order to find grounding for their marketing decisions. They might also pull the grouping result set from the DMP to their own business system for further development. Most of the time, they will start analyzing the shopping mode of the target group directly. Grouping How does the DMP work? Firstly, we, as data platform engineers, define the tags and the rules for grouping. Next, we define the domain-specific language ( ) that describes what we want to do, so we can submit computation tasks to Apache Spark. DSL Then, computing results will be stored in Apache Hive and Apache Doris. Lastly, data users can perform whatever queries they need in Hive and Doris. An abstraction of the DMP can be seen as a four-layered architecture: : All meta-information about the tags is stored in the source data tables; Metadata Management : This layer is supported by Spark, Hive, Doris, and Redis; Computation & Storage : This is like a command center of the DMP. It arranges the tasks throughout the whole platform, such as aggregating data into basic tags, converting data of basic tags into SQL semantics for queries based on the DSL rules, and passing computing results from Spark to Hive and Doris; Scheduling : This is where data users conduct grouping, profile analysis, and check the tags. Service Tags Tags are the most important elements of our DMP, so I'm going to take a whole chapter to introduce them. Every tag in our DMP goes through a five-phased lifecycle: : Data users put forward what kind of tags they need. Demand : Data developers sort out the data and produce the tags. Production : Tags are put into use for grouping. Grouping : Operators launch precision marketing practices on the targeted groups. Marketing : All relevant parties evaluate the utilization rate and effectiveness of tags, and then plan for subsequent improvement. Evaluation Every day, the grouping results will be updated and the outdated data will be cleaned. Both processes are automatic. We have also partially automated the tag production process. Our next priority is to realize 100% automation of it. How Tags are Produced? To derive tags from data, we need to turn our raw data into more organized and structured ones first: : This layer contains the user login history, event tracking logs, transaction data, and the binlogs from various databases. ODS (Operational Data Store) : Data processed by the ODS layer will be sent here to form user login tables, user event tables, and order information sheets. DWD (Data Warehouse Details) : The DM layer stores the aggregated data from DWD. DM (Data Market) Data in DM will be paired by “and”, “or”, and “xor” logic to produce tags that are more comprehensible for data users. Types of Tags Tags can be categorized differently based on different metrics: Where Do We Store Our Data? We made a list of what would look like an ideal data warehouse for us: Support high-performance queries so as to handle massive consumer end traffic; Support SQL for data analysis; Support data update; Be able to store huge amounts of data; Support extension functions to deal with user-defined data structures; Be closely integrated into the big data ecosystem. It was hard to find a single tool that met all these needs, so we tried a mixture of multiple tools. We stored part of our offline and real-time data in Hbase for basic tag queries, most of the offline data in Hive, and double-wrote the rest of our real-time data into Kudu and ES for real-time grouping and data query. The grouping results would be produced by Impala and then cached in Redis. As you can imagine, complicated storage could lead to tricky maintenance. Also, double writing adds to the risk of data inconsistency since one of them might fail. So, in Storage Architecture 2.0, we introduced and Apache Spark. The whole data pipeline was a Y-shaped diagram. Apache Doris We stored offline data in Hive, and the basic tags and real-time data Doris. Then, based on Spark, we conducted federated queries across Hive and Spark. The query results would be stored in Redis. With this new architecture, we compromised a little bit of performance for much lower maintenance costs. P.S. For your reference, here is a summary of the applicable scenarios for the various engines that we've used or investigated. What High-Performance Queries? Some of our data queries demand high performance, such as grouping checks and customer group analysis. Grouping Check Grouping check means to see if certain users are categorized into one or several given groups. It is a two-step check: : perform pre-computations and store the results in Redis; use Lua script for bulk check and thus increase performance. Check-in static group packets : extract data from contexts, API, and Apache Doris for rule-based judgment. Meanwhile, we have tried to increase performance by asynchronous checks, quick short-circuit calculation, query statement optimization, and join table quantity control. Check-in real-time behavior groups Customer Group Analysis Customer group analysis is done to figure out the behavioral path of consumers. It entails joining queries across the group packets and multiple tables. Apache Doris does not support path analysis functions so far, but its computing model is friendly to user-defined function development, so we have built a UDF for this purpose and it works well. Our Gains After Architectural Simplification The newly introduced data warehouse, Apache Doris, can be applied in multiple scenarios, including point queries, batch queries, behavioral path analysis, and grouping. In-pointf queries and small-scale join queries. It delivers a high performance of over 10,000 QPS and an RT99 of less than 50ms. Apart from its strong scalability and easy maintenance, we also benefit from the much simpler tag models brought by the integration of real-time and offline data. Conclusion In this post, I zoomed in on various parts of our DMP, and explained how the data tags, storage and query worked. I believe that a good tag system, faster query speeds, and more targeted consumer operation are the recipe for successful refined operations. So our follow-up efforts will be put into these aspects. Tag system: Build a comprehensive and accurate tag evaluation system; Improve quality and production efficiency of tags; Increase the coverage rate of tags. Storage & computation performance: Gradually move all our workloads from Hive and Spark to Apache Doris. Precision marketing: Develop a rich set of user analysis models; Enhance our user insight system; Enable general profile analysis to empower intelligent decisions. I'm writing this piece to share our practice in the data engineering community and hopefully, collect some valuable suggestions, so if you've got any ideas, meet me in the comment section.
