Further proof of why we don&#x27;t deserve nice things - Despite the love it received from core gamers and critics alike, Remedy&#x27;s recent effort, Alan Wake 2, has...

Alan Wake 2 Yet to Recoup Dev, Marketing Expenses as Tencent Raises Stake | Push Square

Tencent has nearly tripled its investment into Alan Wake developer Remedy Entertainment, Chinese juggernaut now owns nearly 15% of the company&#x27;s shares.

Tencent buys 14.8% of Remedy's shares, Remedy CEO: 'We don't see this changing anything'

In its quarterly earnings presentation, Remedy Entertainment has shared an update on its remake of Max Payne 1 and 2, as well as the news that Tencent has nearly tripled its investment in the studio.

Remedy Shares Max Payne Remake Update as Tencent Nearly Triples its Investment - IGN

On a related note, as reported by GamesIndustry, Tencent has increased its shareholding investment in the Alan Wake II developer, increasing its stake from 5% to 14%. The Chinese conglomerate is also co-financing Remedy&#x27;s free-to-play co-op multiplayer game Vanguard, so the increase in investment ...

Alan Wake II Has yet to Recoup Development and Marketing Expenses; Tencent Raised Stakes in Remedy to 14%

BEIJING -- China on Wednesday implemented a revised State Secrets Law, requiring internet companies such as Tencent and Weibo to delete leaked in

China enlists Tencent, Weibo and Douyin to protect state secrets - Nikkei Asia

Tencent has increased its shareholding investment in Remedy Entertainment to 14%.The conglomerate previously held 5% of…

Tencent raises its stake in Remedy to 14% | GamesIndustry.biz

Tencent quietly upped its stake in the Finnish studio, which is gearing up to begin production on at least three projects.

Tencent now owns almost 15 percent of Remedy

Tencent, the Chinese gaming publisher, has increased its stake in the Alan Wake and Control developer, Remedy Entertainment, [...]

Tencent Increases Stake in Remedy to Nearly 15%

Tencent Cloud, Metavision to jointly develop AI-based 3D visual interactive services

With all these games in the cooker, existing stakeholder Tencent has increased its interest in the studio. Previously, Tencent acquired a minority (3.8%) stake in Remedy that it later upped to 5%. Today, Remedy announced that Tencent increased its stake again to 14.8%, nearly triple what it ...

Remedy Shares Max Payne Remake Update as Tencent Nearly Triples its Investment

TENCENT

tencent

This article is co-written by me and my colleague Kai Dai. We are both data platform engineers at Tencent Music (NYSE: TME), a music streaming service provider with a whopping 800 million monthly active users. To drop the number here is not to brag but to give a hint of the sea of data that my poor coworkers and I have to deal with everyday. What We Use ClickHouse For The music library of Tencent Music contains data of all forms and types: recorded music, live music, audios, videos, etc. As data platform engineers, our job is to distill information from the data, based on which our teammates can make better decisions to support our users and musical partners. Specifically, we do all-round analysis of the songs, lyrics, melodies, albums, and artists, turn all this information into data assets, and pass them to our internal data users for inventory counting, user profiling, metrics analysis, and group targeting. We stored and processed most of our data in Tencent Data Warehouse (TDW), an offline data platform where we put the data into various tag and metric systems and then created flat tables centering each object (songs, artists, etc.). Then we imported the flat tables into ClickHouse for analysis and Elasticsearch for data searching and group targeting. After that, our data analysts used the data under the tags and metrics they needed to form datasets for different usage scenarios, during which they could create their own tags and metrics. The data processing pipeline looked like this: The Problems with ClickHouse When working with the above pipeline, we encountered a few difficulties: : Partial update of columns was not supported. Therefore, any latency from any one of the data sources could delay the creation of flat tables, and thus undermine data timeliness. Partial Update : Data under different tags and metrics was updated at different frequencies. As much as ClickHouse excelled in dealing with flat tables, it was a huge waste of storage resources to just pour all data into a flat table and partition it by day, not to mention the maintenance cost coming with it. High storage cost : Architecturally speaking, ClickHouse was characterized by the strong coupling of storage nodes and compute nodes. Its components were heavily interdependent, adding to the risks of cluster instability. Plus, for federated queries across ClickHouse and Elasticsearch, we had to take care of a huge amount of connection issues. That was just tedious. High maintainence cost Transition to Apache Doris , a real-time analytical database, boasts a few features that are exactly what we needed in solving our problems: Apache Doris : Doris supports a wide variety of data models, among which the Aggregate Model supports real-time partial update of columns. Building on this, we can directly ingest raw data into Doris and create flat tables there. The ingestion goes like this: Firstly, we use Spark to load data into Kafka; then, any incremental data will be updated to Doris and Elasticsearch via Flink. Meanwhile, Flink will pre-aggregate the data so as to release burden on Doris and Elasticsearch. Partial update : Doris supports multi-table join queries and federated queries across Hive, Iceberg, Hudi, MySQL, and Elasticsearch. This allows us to split the large flat tables into smaller ones and partition them by update frequency. The benefits of doing so include a relief of storage burden and an increase of query throughput. Storage cost : Doris is of simple architecture and is compatible with MySQL protocol. Deploying Doris only involves two processes (FE and BE) with no dependency on other systems, making it easy to operate and maintain. Also, Doris supports querying external ES data tables. It can easily interface with the metadata in ES and automatically map the table schema from ES so we can conduct queries on Elasticsearch data via Doris without grappling with complex connections. Maintenance cost What's more, Doris supports multiple data ingestion methods, including batch import from remote storage such as HDFS and S3, data reads from MySQL binlog and Kafka, and real-time data synchronization or batch import from MySQL, Oracle, and PostgreSQL. It ensures service availability and data reliability through a consistency protocol and is capable of auto debugging. This is great news for our operators and maintainers. Statistically speaking, these features have cut our storage cost by 42% and development cost by 40%. During our usage of Doris, we have received lots of support from the open source Apache Doris community and timely help from the SelectDB team, which is now running a commercial version of Apache Doris. Further Improvement to Serve Our Needs Introduce a Semantic Layer Speaking of the datasets, on the bright side, our data analysts are given the liberty of redefining and combining the tags and metrics at their convenience. But on the dark side, high heterogeneity of the tag and metric systems leads to more difficulty in their usage and management. Our solution is to introduce a semantic layer in our data processing pipeline. The semantic layer is where all the technical terms are translated into more comprehensible concepts for our internal data users. In other words, we are turning the tags and metrics into first-class citizens for data definement and management. Why would this help? For data analysts, all tags and metrics will be created and shared at the semantic layer so there will be less confusion and higher efficiency. For data users, they no longer need to create their own datasets or figure out which one is applicable for each scenario but can simply conduct queries on their specified tagset and metricset. Upgrade the Semantic Layer Explicitly defining the tags and metrics at the semantic layer was not enough. In order to build a standardized data processing system, our next goal was to ensure consistent definition of tags and metrics throughout the whole data processing pipeline. For this sake, we made the semantic layer the heart of our data management system: How does it work? All computing logics in TDW will be defined at the semantic layer in the form of a single tag or metric. The semantic layer receives logic queries from the application side, selects an engine accordingly, and generates SQL. Then it sends the SQL command to TDW for execution. Meanwhile, it might also send configuration and data ingestion tasks to Doris and decide which metrics and tags should be accelerated. In this way, we have made the tags and metrics more manageable. A fly in the ointment is that since each tag and metric is individually defined, we are struggling with automating the generation of a valid SQL statement for the queries. If you have any idea about this, you are more than welcome to talk to us. Give Full Play to Apache Doris As you can see, Apache Doris has played a pivotal role in our solution. Optimizing the usage of Doris can largely improve our overall data processing efficiency. So in this part, we are going to share with you what we do with Doris to accelerate data ingestion and queries and reduce costs. What We Want? Currently, we have 800+ tags and 1300+ metrics derived from the 80+ source tables in TDW. When importing data from TDW to Doris, we hope to achieve: : In addition to the traditional T+1 offline data ingestion, we require real-time tagging. Real-time availability : Each source table generates data through its own ETL task at various paces and involves only part of the tags and metrics, so we require the support for partial update of columns. Partial update : We need a response time of only a few seconds in group targeting, analysis and reporting scenarios. High performance : We hope to reduce costs as much as possible. Low costs What We Do? Generate Flat Tables in Flink Instead of TDW Generating flat tables in TDW has a few downsides: : TDW has to maintain an extra flat table apart from the discrete 80+ source tables. That's huge redundancy. High storage cost : Any delay in the source tables will be augmented and retard the whole data link. Low real-timeliness : To achieve real-timeliness would require extra development efforts and resources. High development cost On the contrary, generating flat tables in Doris is much easier and less expensive. The process is as follows: Use Spark to import new data into Kafka in an offline manner. Use Flink to consume Kafka data. Create a flat table via the primary key ID. Import the flat table into Doris. As is shown below, Flink has aggregated the five lines of data, of which "ID"=1, into one line in Doris, reducing the data writing pressure on Doris. This can largely reduce storage costs since TDW no long has to maintain two copies of data and KafKa only needs to store the new data pending for ingestion. What's more, we can add whatever ETL logic we want into Flink and reuse lots of development logic for offline and real-time data ingestion. Name the Columns Smartly As we mentioned, the Aggregate Model of Doris allows partial update of columns. Here we provide a simple introduction to other data models in Doris for your reference: : This is applicable for scenarios requiring primary key uniqueness. It only keeps the latest data of the same primary key ID. (As far as we know, the Apache Doris community is planning to include partial update of columns in the Unique Model, too.) Unique Model : This model stores all original data exactly as it is without any pre-aggregation or deduplication. Duplicate Model After determining the data model, we had to think about how to name the columns. Using the tags or metrics as column names was not a choice because: Ⅰ. Our internal data users might need to rename the metrics or tags, but Doris 1.1.3 does not support modification of column names. Ⅱ. Tags might be taken online and offline frequently. If that involves the adding and dropping of columns, it will be not only time-consuming but also detrimental to query performance. Instead, we do the following: For flexible renaming of tags and metrics, we use MySQL tables to store the metadata (name, globally unique ID, status, etc.). Any change to the names will only happen in the metadata but will not affect the table schema in Doris. For example, if a is given an ID of 4, it will be stored with the column name of a4 in Doris. Then if the is involved in a query, it will be converted to a4 in SQL. song_name song_name For the onlining and offlining of tags, we sort out the tags based on how frequently they are being used. The least used ones will be given an offline mark in their metadata. No new data will be put under the offline tags but the existing data under those tags will still be available. For real-time availability of newly added tags and metrics, we prebuild a few ID columns in Doris tables based on the mapping of name IDs. These reserved ID columns will be allocated to the newly added tags and metrics. Thus, we can avoid table schema change and the consequent overheads. Our experience shows that only 10 minutes after the tags and metrics are added, the data under them can be available. Noteworthily, the recently released Doris 1.2.0 supports Light Schema Change, which means that to add or remove columns, you only need to modify the metadata in FE. Also, you can rename the columns in data tables as long as you have enabled Light Schema Change for the tables. This is a big trouble saver for us. Optimize Date Writing Here are a few practices that have reduced our daily offline data ingestion time by 75% and our CUMU compaction score from 600+ to 100. Flink pre-aggregation: as is mentioned above. Auto-sizing of writing batch: To reduce Flink resource usage, we enable the data in one Kafka Topic to be written into various Doris tables and realize the automatic alteration of batch size based on the data amount. Optimization of Doris data writing: fine-tune the the sizes of tablets and buckets as well as the compaction parameters for each scenario: max_XXXX_compaction_thread
max_cumulative_compaction_num_singleton_deltas Optimization of the BE commit logic: conduct regular caching of BE lists, commit them to the BE nodes batch by batch, and use finer load balancing granularity. Use Dori-on-ES in Queries About 60% of our data queries involve group targeting. Group targeting is to find our target data by using a set of tags as filters. It poses a few requirements for our data processing architecture: Group targeting related to APP users can involve very complicated logic. That means the system must support hundreds of tags as filters simultaneously. Most group targeting scenarios only require the latest tag data. However, metric queries need to support historical data. Data users might need to perform further aggregated analysis of metric data after group targeting. Data users might also need to perform detailed queries on tags and metrics after group targeting. After consideration, we decided to adopt Doris-on-ES. Doris is where we store the metric data for each scenario as a partition table, while Elasticsearch stores all tag data. The Doris-on-ES solution combines the distributed query planning capability of Doris and the full-text search capability of Elasticsearch. The query pattern is as follows: SELECT tag, agg(metric) 
   FROM Doris 
   WHERE id in (select id from Es where tagFilter)
   GROUP BY tag As is shown, the ID data located in Elasticsearch will be used in the sub-query in Doris for metric analysis. In practice, we find that the query response time is related to the size of the target group. If the target group contains over one million objects, the query will take up to 60 seconds. If it is even larger, a timeout error might occur. After investigation, we identified our two biggest time wasters: I. When Doris BE pulls data from Elasticsearch (1024 lines at a time by default), for a target group of over one million objects, the network I/O overhead can be huge. II. After the data pulling, Doris BE needs to conduct Join operations with local metric tables via SHUFFLE/BROADCAST, which can cost a lot. Thus, we make the following optimizations: Add a query session variable that specifies whether to enable optimization. es_optimize In data writing into ES, add a BK column to store the bucket number after the primary key ID is hashed. The algorithm is the same as the bucketing algorithm in Doris (CRC32). Use Doris BE to generate a Bucket Join execution plan, dispatch the bucket number to BE ScanNode and push it down to ES. Use ES to compress the queried data; turn multiple data fetch into one and reduce network I/O overhead. Make sure that Doris BE only pulls the data of buckets related to the local metric tables and conducts local Join operations directly to avoid data shuffling between Doris BEs. As a result, we reduce the query response time for large group targeting from 60 seconds to a surprising 3.7 seconds. Community information shows that Doris is going to support inverted indexing since version 2.0.0, which is soon to be released. With this new version, we will be able to conduct full-text search on text types, equivalence or range filtering of texts, numbers, and datetime, and conveniently combine AND, OR, NOT logic in filtering since the inverted indexing supports array types. This new feature of Doris is expected to deliver 3~5 times better performance than Elasticsearch on the same task. Refine the Management of Data Doris' capability of cold and hot data separation provides the foundation of our cost reduction strategies in data processing. Based on the TTL mechanism of Doris, we only store data of the current year in Doris and put the historical data before that in TDW for lower storage cost. We vary the numbers of copies for different data partitions. For example, we set three copies for data of the recent three months, which is used frequently, one copy for data older than six months, and two copies for data in between. Doris supports turning hot data into cold data so we only store data of the past seven days in SSD and transfer data older than that to HDD for less expensive storage. Conclusion Thank you for scrolling all the way down here and finishing this long read. We've shared our cheers and tears, lessons learned, and a few practices that might be of some value to you during our transition from ClickHouse to Doris. We really appreciate the help from the Apache Doris community and the SelectDB team, but we might still be chasing them around for a while since we attempt to realize auto-identification of cold and hot data, pre-computation of frequently used tags/metrics, simplification of code logic using Materialized Views, and so on and so forth.

Tencent Music Transitions from ClickHouse to Apache Doris

ক্লিকহাউস থেকে অ্যাপাচি ডরিসে টেনসেন্ট মিউজিক ট্রানজিশন

Tencent Music wechselt von ClickHouse zu Apache Doris

Transiciones de Tencent Music de ClickHouse a Apache Doris

Tencent Music passe de ClickHouse à Apache Doris

क्लिकहाउस से अपाचे डोरिस तक टेनसेंट म्यूजिक ट्रांजिशन

Tencent Music の ClickHouse から Apache Doris への移行

ClickHouse에서 Apache Doris로 Tencent Music 전환

Tencent Music Transições de ClickHouse para Apache Doris

ClickHouse'dan Apache Doris'e Tencent Müzik Geçişleri

Tencent Music chuyển đổi từ ClickHouse sang Apache Doris

腾讯音乐从 ClickHouse 过渡到 Apache Doris

Alluxio provides a comprehensive metrics system for monitoring the status of the Alluxio master, worker and client. Alluxio’s metrics system supports various kinds of metrics, e.g. an embedded JSON sink and the Prometheus sink. Users and developers can easily create a custom sink of Alluxio by implementing the Sink interface. In addition, Alluxio provides a metrics page in web UI showing some key information of Alluxio, such as bytes throughput and storage space. However, if you want more flexible and universal monitoring, additional work is required. This blog will introduce how Tencent uses Prometheus and Grafana to set up a monitoring system for Alluxio in 10 minutes. How Alluxio Metrics System Works The following framework depicts Alluxio’s metrics system. Types of Metrics Alluxio metrics are generated by different sources and consumed by sinks. The metric system polls sources periodically and passes metric reports to sinks. There are two types of metrics. Cluster metrics: aggregated from workers and clients, then calculated by the leading master. Cluster metrics provide snapshots of Alluxio cluster. Process metrics: collected by each Alluxio process, including master metrics, worker metrics, and client metrics. Metrics Naming Patterns In general, metrics of Alluxio follow two naming patterns. The first is for master metrics, which consist of Master, metricName, and some tags (optional). For example, Master.GetFileInfoOps. The second is for non-master metrics, which consists of processType, metricName, tags, and hostnName. For example, below is openExistingFile metrics for worker. The processType is worker, the metricName is openExistingFile, followed by three tags: user, UFS location, and UFS type. The last part is the hostName of worker. The Flow of Metrics from Worker to Master Since the master needs to collect metrics from workers and clients to calculate the cluster metrics, the clients and workers need to report their metrics to the master. Below is a diagram of the flow of metrics from the worker to the master. Workers and clients send the metrics to Alluxio master through the heartbeat. The block on the right is the worker process, and the left is the master process. It goes from the right to the left. The heartbeat process is under the control of BlockMasterSync, and the metrics and other information about the worker are passed to the RpcClient, then the Rpc server receives the heartbeat and processes the metrics from workers by MetricMaster. The metrics are stored in the MetricsStore and later will be aggregated and calculated. Client metrics are handled in a similar manner. Refer to the Alluxio source code for more details. How to Implement a Custom Alluxio Sink Alluxio supports many types of sinks. A sink specifies where the metrics are delivered to. For example, Alluxio has Http JSON sink, which is enabled by default, so we can view dashboards of Alluxio metrics in Alluxio web UI in the JSON format. In addition, Alluxio provides convenient and flexible sink interfaces for developers and users to implement their own sink. Passive Sink and Active Sink Sinks that Alluxio supports are listed below. In general, these sinks can be divided into two categories: passive sink and active sink. The difference between these two kinds of sinks is the way we get the metrics: Active: the left screenshot is the consolesink. If we enable the consolesink, the metrics will be printed into a standard output file, for example, the master.out or worker.out. The consolesink reports the metrics periodically, so you can get the metrics any time you want. From Alluxio’s perspective, it is active sink. Passive: the right screenshot is the HTTP JSON sink, which is the passive sink. A passive sink works as a service. We can get the metrics only when we send the requests to the server. For example, if we access the metrics/json location in the Alluxio worker server, we can get the following metrics in JSON format. How ConsoleSink and Prometheus Sink Work The ConsoleSink structure is shown below. We can find the ConsoleReporter in the ConsoleSink. It schedules the report that reports the metrics on a regular basis. We can see methods of the ScheduledReporter, in the report, that different types of metrics are printed in a specific format. The Prometheus sink operates in a completely different way. There is no ScheduledReporter to report the metrics, so we need to get metrics by ourselves. The passive sink needs to implement the getHandler method, so the master and worker process can start. For example, in the StartServingWebServer method of master process, the Prometheus handler is added to the webserver. We can access the specific location of master web server, then we can get the metrics in the format of Prometheus. E.g. MasterProcess Implement the Sink The first thing we need to do is to determine which kind of sink we want to implement, passive or active. For example, XmlSink of Alluxio prints the metrics to a specific path, so we know that it is an active sink. Just like the ConsoleSink, we need to implement the sink interface, and also construct a XmlReporter to report metrics in XML format. This gives us a custom sink. After that, we need to enable our sink by adding XmlSink class we just defined to metrics.properties. In addition, we can add some properties to our sink in this format. conf/metrics.properties Similarly, if you want to implement a passive sink, you can refer to the JSON sink or Prometheus sink. How to Set Up Monitoring of Alluxio in 10 Minutes Alluxio Web UI Monitor On the metrics tab for Alluxio web UI, we can monitor the state of the Alluxio cluster and get some key information about Alluxio. Even though the dashboard is convenient, it can sometimes be confusing (the screenshot on the right). In addition, if we want more information about Alluxio or a dashboard with more visuals, we will need another monitoring solution, which requires additional effort. We use Prometheus sink to bridge Alluxio and Grafana since both Alluxio and Grafana have good support for Prometheus. First, Prometheus scrapes metrics from Alluxio servers and transform into time-series data and store them. Then Grafana server gets this time series data from Prometheus, after which Grafana web UI will display metrics in dashboards. How to Set Up the Monitoring System It only takes four steps to set up this monitoring: Install and start Prometheus and Grafana server. It is quite simple to follow the tutorials using the default configuration. [ ] Add Alluxio Jobs to Prometheus, which can find Alluxio server and scrape the metrics automatically. In the Grafana part, we provide Grafana dashboard template so we can reuse it to quickly set up the Alluxio dashboard. [ ] Download and import the dashboard template. [ ] Modify variables of the template to make it our own dashboard. [ ] (16:03) is the demo video on how to import the dashboard template. Here We need to add Alluxio Jobs to Prometheus first (refer to Alluxio in the metrics section). Now we add two jobs in Prometheus. One is Alluxio master, and the other is Alluxio worker. Here we need to record the job_name, which is used to import Grafana dashboard. documentation The versions of components in the demo video are as follows: Prometheus Version: 2.22.2 Grafana Version: 7.5.6 Alluxio Version: 2.5.0-3 The following are the services set up in the demo video. There are two nodes in our cluster, worker node and master node: Prometheus Server @ master:9090 Grafana Server @ worker:3000 Alluxio Master @ master Alluxio Worker @ worker As can be seen in the screenshots below, there are 8 live workers in the cluster, and the monitoring system works well, with metrics in different panels. In the dashboard, we also list the labels of different rows. Adding or removing panels can be easily done on the dashboard. Alluxio IO Key Metrics Read Local & Read Remote & Alive workers … Storage Space Used & UFS Space Used … Workers Blocks Cached Blocks & Evicted Blocks … Logical Operations Mount Operations & File Pinned … Alluxio Metadata Operations Block Heartbeat Cost & Get status Cost … AsyncCache Blocks & Operations AsyncCacheSuccessedBlocks & AysncCacheFailedBlocks … Master JVM Memory Master Heap Memory & Total Memory … Summary By introducing Prometheus and Grafana, we have built a more flexible, agile and easy-to-use monitoring system for Alluxio. References Alluxio Documentation: Metrics System Grafana dashboards About the Author Pan Liu Big Data Engineer at Tencent Responsible for distributed computing with a focus on open source projects such as Alluxio and Presto Also published here: https://www.alluxio.io/blog/how-to-set-up-monitoring-system-for-alluxio-with-prometheus-and-grafana-in-10-minutes/

How Tencent uses Prometheus and Grafana to Set Up a Monitoring System in 10 Minutes

Regulation is not a sexy topic, unless we are discussing crypto regulations. In the last few weeks, many countries have voiced their unique ways to welcome ( or not so much) the crypto community. Korea has decided to as banks, while to be traded and exchanges to be opened. Even the SEC made remarks on BTC and ETH not being securities (horay!). China certainly has voiced their opinions too, this time through their permitting of e-commerce company JD regulate crypto exchanges Thailand has permitted certain cryptocurrencies to build a blockchain-backed token. This is particularly important, for 2 reasons. For one, we now know that China doesn’t just care about blockchain, it cares about tokens too. Past announcements only suggested the former, such as President Xi’s recent speech and the country’s unfavorable stance towards ICOs and crypto exchanges. But here we are, with an asset-backed-security token being issued by JD in collaboration with Huatai Securities, a state-owned investment bank. Secondly, and more importantly, the announcement divulges a glimpse of China’s crypto strategy. Tokens can be issued, but only if it’s issued by the right party. touting the benefits of blockchain technology, TL:DR: to really play and win the crypto game in China as an investor, you should actually be investing and buying into a basket of the top established tech companies such as Tencent, Alibaba, Baidu and JD, instead of the Chinese tokens that are currently ranked in coinmarketcap.com . Very few of the current top market cap tokens can succeed in China (but that doesn’t mean they won’t succeed.) Some more details about the news per Coindesk: “According to a from the Securities Times, an outlet of the China Securities Regulatory Commission, JD Finance established the trial in partnership with Huatai Securities, a brokerage firm that will underwrite the issuance, as well as Xingye Bank, which will act as a trust. JD Finance aims to issue the securities using a consortium blockchain that will see each party act as a node, recording the transactions in a transparent manner. The experiment sets out to ascertain if the blockchain can meet the demands of the various parties involved in the asset securitization process, such as issuers, underwriters and buyers.” report Upon first glance, this partnership is quite unique. First, a Chinese bank is collaborating with a Chinese tech company to jointly develop a blockchain product. This is equivalent to the US if Amazon worked together with Morgan Stanley to develop a blockchain product. That is unheard of, and unlikely to happen given the sheer number of legal issues that would be involved. However, in China, this is not out of the ordinary. In China, for example, tech companies can take a significant ownership of banks through equity. Mr. Moritz shared a great data point in the aforementioned Financial Times article about how Tencent and Alibaba’s investments totaled in over a thousand companies. Many of those investments also include public-listed companies such as banks and state owned enterprises (or commonly known as SOEs). Tencent, for example, (China International Capital Corporation) last year, the first investment bank in China born out of a joint venture between China Construction Bank and Morgan Stanley. Such large type of stake in a company naturally fosters closer relationship and collaboration. And what to better collaborate on than blockchain? acquired 5% of CICC Now, I don’t believe JD owns significant or any equity in Huatai Securities, but with this collaboration announcement, I promise you that it won’t be the last. Other new token creations between tech companies and banks may already be in the works. Note: You may ask, why is this important? Now in the US, we have a situation where the finance companies are trying to invest and build out their technology platform and possibly crypto trading I.e Goldman Sachs and JP Morgan, and tech companies like Coinbase are fighting daily over legal battles and clearing through gray areas with the SEC. And then you have an alternative situation in China, where the tech and finance companies are working together to build new products with the support of regulators. Who do you think is going to move faster in getting the cryptocurrency recognized by the masses, US or China? I believe each of these country’s approaches will have its benefits and flaws, opportunities and problems. But the fact that China’s regulators permit such possibility of collaboration is worth learning from. Now, the Chinese securities commission granted JD the right to issue a financial token in collaboration with Huatai. They did not grant this opportunity to any of the top market cap crypto companies such as NEO, EOS nor TRON. is primarily an . Despite the company’s recent , it’s not a pure, crypto-focused company like the others that I’ve mentioned. Note a tech company has been blessed with issuing the first blockchain token, after almost a year since the country banned cryptocurrency trading and ICOs. Even the recent Ministry of Industry & Information Technology’s (MIIT) crypto ranking did not give away this clue. Despite of Ethereum and NEO in its recent technology evaluation, it still opted to let a traditional tech company to issue tokens. JD.com e-commerce company with multiple product lines including food delivery developments on blockchain MIIT’s recent high ranks I believe JD was chosen because the Chinese regulators already had a close working relationships with them, along with Tencent, Alibaba and Baidu. When it comes to working with a “foreign” technology like blockchain, the regulators trust the existing technology leaders to test out these products more than the new kids on the block. A China expert tweeted once- “the purpose of a blockchain is to decentralize, which goes completely against what the Chinese .” Indeed, decentralization undermines the , and that’s exactly the opposite of what China under . Regulators will not support companies that attempt to decentralize the party’s power or put power into people’s hands in ways they don’t them want to. So Crypto companies that have touted decentralization, even if they were Chinese and may arguably have a leg up to non-Chinese Crypto companies when doing business in China, are at direct opposition with the regulators’ primary objectives. This is something Binance knew early on, and subsequently established their businesses elsewhere (they also didn’t have a choice, but they were also strategic and established a long-term relationship with Malta). Meanwhile, with it’s main objectives to the Chinese government continued to work towards transforming China with the help of its technology companies. There is a reason why Tencent and Alibaba’s products such as WeChat and Alipay are so ingrained into people’s lives in China (if you don’t know what those apps are, you need to. Better yet, go to Beijing for your next vacation and experience first hand). These tech companies had the blessings of the federal and local government, which permitted them to build amazing tools for their citizens, with recent example such as the and ”. Nonetheless, this is also the same reason why all the and there is no privacy on these platforms. The exact reason that tech companies have collaborated so closely with the government is why these same tech companies will get to first try new shiny toys. When it comes to new token issuance, the government is going to default to working with the technology companies they have worked with before than working with the newcomers in the industry. government wants Central Party President Xi is trying to do become the world’s dominant power, mobile “metro cards” mobile “passport WeChat messages are monitored The Chinese government also has a habit of fostering healthy competition between the large tech conglomerates, so not just one has all the leverage. To me, the signs of allowing JD to work with Huatai means that not soon after we’ll see Tencent or Alibaba announcing another token. All of this bodes well for the tech companies as well as China’s crypto space. If the initial blockchain+token experiments goes well, these technology companies will potentially take over as the backbone of China’s finance infrastructure. It’ll be a win for regulators as they then would have better control and monitor of banks and all of the country’s money supply. It’ll also be a win for the tech companies, as they will be entrusted with more innovation and potential future businesses. Note: Now, once the value of tokens is validated, I believe the Chinese government may feel more comfortable to allow companies like NEO to build private chains for businesses and permit it to flourish. Nonetheless, for crypto companies that are based in China or founded by Chinese teams, but don’t have the support of the government will either have try to find a business case elsewhere or be absorbed by the large tech companies who are looking for talent. This is why, for a long time, I’m going to be very bullish on the future of the top tech companies in China — Tencent, , JD and Baidu. I don’t believe that one can just do traditional financial analysis and derive these companys’ valuations by forecasting each of their products’ line revenues. For these companies, their products are not just what you may see on the roadmap, or what’s already in their apps. Instead, their products are influenced and created by whatever the Chinese government wants them to build and whichever technological direction the Chinese government wants to turn to. So, the possibilities are limitless. And for now, it’s blockchain. Alibaba

China’s True Crypto Giants- Tencent, Baidu, Alibaba and JD

By Jassem Osseiran, Co-Founder of PlaysOut In the rapidly evolving landscape of gaming, industry veterans are aiming to bridge the gap between traditional web2 games and the burgeoning world of web3. Co-founded by industry veterans with extensive experience in both gaming and blockchain technology, PlaysOut's mission is to integrate mini-games into major traffic platforms, providing access to Web3 tools and making them available through popular messenger apps and other platforms with large user bases. This approach seeks to create a seamless gaming experience that leverages the strengths of both web2 and web3 environments. A Serious Technological Edge PlaysOut leverages a distinct technological framework that sets it apart from traditional platforms, providing easy access to games outside of H5/webview and standalone mobile games. Using a mini-game container technology, we’ve created a closed runtime environment. This approach enhances security through sandbox technology, ensuring that games run in an isolated and protected environment, enhancing security, and protecting both game developers and players from potential vulnerabilities and threats. By optimizing game resource preloading, PlaysOut significantly improves game loading speeds and enhances native capabilities. This results in a more seamless and responsive gaming experience. PlaysOut's focus on game performance optimization also leads to smoother graphics and a more enjoyable gameplay experience. This optimization allows mini-games to achieve near-native fluidity while maintaining the convenience and same accessibility of H5. The technological advancements enabled by PlaysOut allows mini-games to support medium to heavy games with rich content, providing a gaming experience that rivals standalone mobile games while retaining the user-friendliness of H5/webview platforms. It’s All About Your People The strength of PlaysOut lies in the combined expertise of its core team, which encompasses deep knowledge of both gaming and web3. The core team is highly skilled in Web3, having been active in the field since 2014 with over 10 years of industry experience. Our expertise covers a wide range of domains, including underlying public chains, DEX, cross-chain bridges, DeFi, GameFi, and NFTs. This team has both the technological and operational capabilities needed to navigate the complex Web3 landscape. The synergy of gaming and Web3 expertise at an S-tier level provides a solid foundation for PlaysOut's mission to create a world-class mini-game platform that bridges Web2 and Web3. With a diverse background in Technology, Venture Building, Finance, Web3, and Tencent technology, the technical team lead brings 20 years of experience in the gaming industry. Prior to joining Playsout, our team was responsible for the overall technical operations of Tencent's independent game studio and possesses technical capabilities at the level of all major game engines. Our profound understanding of the technical principles and architecture of WeChat mini-games, combined with a decade of experience in the Web3 environment, is a significant asset. In complete synergy with the technical team, the product operations team comes from major game companies such as Perfect World and SNK. The team product lead has 20 years of experience in the gaming industry and has served as Vice President of Distribution, launching over 25 international games. Partnerships Give the Ultimate Advantage PlaysOut's unique collaboration advantages further bolster its position in the market. With Tencent as a partner for mini-programs, PlaysOut secured status as the only overseas company to achieve this collaboration at several exclusive levels. China's vast market of over 400,000 WeChat mini-game developers, producing tens of thousands of mini-game products annually, presents significant opportunities for the space. The primary collaboration advantages for these developers are: Efficient Integration Advantage Independently developed mini-game container technology allows for seamless integration similar to WeChat mini-games. This ensures that mini-game products from the Chinese and Asian markets can be easily adapted and integrated into the PlaysOut platform, localized and deployed for global traffic access.


Developer Trust Relationships Establishing strong relationships with developers sets the stage for seamless global positioning. Some notable groups in the space include  4399, Huaxia Leyou, Beijing Haibi, Stars Interactive Entertainment, Shanghai Coolreal, Chengdu MUMU, and Beijing Aodun Network Technology.


Web3 Deployment Expertise
Creating and facilitating an environment for developers and games entering the Web3 space through access toessential tools such as wallet, swap, and token infrastructures enables seamless transitions and optimization of traffic performance and behavior across various game titles. One of the core challenges faced by web3 games is integrating into the broader gaming industry. The vision centers on creating a self-sustaining ecosystem where all participants — game developers, players, investors, and gaming platforms — are driven by more than just profit. However, it is impossible to ignore profit as a driver, and the gaming industry is among the top 10 most profitable industries globally, serving as a strong monetization terminal for the internet industry. Ideally, the next WeChat of web3 gaming would become a true global mini-games publisher, and combined with the intrinsic profitability of games with web3, they would ultimately elevate the overall web3 ecosystem and bring entertaining and profitable games and players into the space. PlaysOut’s platform and SDK solution tackles the profitability challenges faced by game developers and traffic providers worldwide, ensuring that everyone has the opportunity to earn rewards for their effort and engagement with games. The profitability of games serves as the underlying support for PlaysOut, upon which web3 empowerment is built, amplifying the value of excellent products. PlaysOut is poised to revolutionize web3 gaming by fostering a thriving community through innovative developer and gamer tools. This focus on a healthy ecosystem will attract high-quality games and players while also engaging better with the existing audience. By setting a new standard for transitioning web2 games and building a unified platform, PlaysOut is on track to become a central force in the future of gaming, where the best of both worlds collide. About Jassem Osseiran Jassem ("J") is an experienced entrepreneur and operational investor based in London. With a deep understanding of global finance gained through his Economics degree at the University of San Francisco, Jassem began his career as a tech venture builder with Rocket Internet in New York. He then shifted his focus to the Middle East and North Africa (MENA), managing media portfolios and founding startups tailored to the region. A sought-after advisor for high-growth brands and cutting-edge technology platforms, Jassem has orchestrated multiple successful exits, including a notable IPO on the London Stock Exchange. In 2020, he founded 611 Capital Investments, concentrating on investments, incubation, and market scalability across the EU, the Middle East, and Asia. Currently, Jassem is the founder and chief strategy officer of PlaysOut Technologies, spearheading the commercial framework for seamlessly integrating mini-games within super apps. This initiative represents a significant shift in the digital ecosystem, leveraging evolving Web3 technologies to unlock immense potential for mini-games in the digital space. Jassem remains a close advisor and partner to multiple global institutions and key opinion leaders. About PlaysOut PlaysOut is a globally oriented open platform for mini-programs that fully aligns with the Weixin Mini-Program framework. It provides SDK interfaces for super apps, offering a rich array of mini-program and mini-game content. Additionally, PlaysOut streamlines the integration process for developers, providing convenient tools and seamless access to mini-program and mini-game products. This story was distributed under HackerNoon’s Business Blogging Program. Learn more about the program here. By Jassem Osseiran, Co-Founder of PlaysOut By Jassem Osseiran, Co-Founder of PlaysOut PlaysOut PlaysOut In the rapidly evolving landscape of gaming, industry veterans are aiming to bridge the gap between traditional web2 games and the burgeoning world of web3. Co-founded by industry veterans with extensive experience in both gaming and blockchain technology, PlaysOut's mission is to integrate mini-games into major traffic platforms, providing access to Web3 tools and making them available through popular messenger apps and other platforms with large user bases. This approach seeks to create a seamless gaming experience that leverages the strengths of both web2 and web3 environments. rapidly evolving landscape rapidly evolving landscape A Serious Technological Edge A Serious Technological Edge A Serious Technological Edge PlaysOut leverages a distinct technological framework that sets it apart from traditional platforms, providing easy access to games outside of H5/webview and standalone mobile games. Using a mini-game container technology, we’ve created a closed runtime environment. This approach enhances security through sandbox technology, ensuring that games run in an isolated and protected environment, enhancing security, and protecting both game developers and players from potential vulnerabilities and threats. By optimizing game resource preloading, PlaysOut significantly improves game loading speeds and enhances native capabilities. This results in a more seamless and responsive gaming experience. PlaysOut's focus on game performance optimization also leads to smoother graphics and a more enjoyable gameplay experience. This optimization allows mini-games to achieve near-native fluidity while maintaining the convenience and same accessibility of H5. The technological advancements enabled by PlaysOut allows mini-games to support medium to heavy games with rich content, providing a gaming experience that rivals standalone mobile games while retaining the user-friendliness of H5/webview platforms. It’s All About Your People It’s All About Your People It’s All About Your People The strength of PlaysOut lies in the combined expertise of its core team, which encompasses deep knowledge of both gaming and web3. The core team is highly skilled in Web3, having been active in the field since 2014 with over 10 years of industry experience. Our expertise covers a wide range of domains, including underlying public chains, DEX, cross-chain bridges, DeFi, GameFi, and NFTs. This team has both the technological and operational capabilities needed to navigate the complex Web3 landscape. The synergy of gaming and Web3 expertise at an S-tier level provides a solid foundation for PlaysOut's mission to create a world-class mini-game platform that bridges Web2 and Web3. With a diverse background in Technology, Venture Building, Finance, Web3, and Tencent technology, the technical team lead brings 20 years of experience in the gaming industry. Prior to joining Playsout, our team was responsible for the overall technical operations of Tencent's independent game studio and possesses technical capabilities at the level of all major game engines. Our profound understanding of the technical principles and architecture of WeChat mini-games, combined with a decade of experience in the Web3 environment, is a significant asset. In complete synergy with the technical team, the product operations team comes from major game companies such as Perfect World and SNK. The team product lead has 20 years of experience in the gaming industry and has served as Vice President of Distribution, launching over 25 international games. Partnerships Give the Ultimate Advantage Partnerships Give the Ultimate Advantage Partnerships Give the Ultimate Advantage PlaysOut's unique collaboration advantages further bolster its position in the market. With Tencent as a partner for mini-programs, PlaysOut secured status as the only overseas company to achieve this collaboration at several exclusive levels. China's vast market of over 400,000 WeChat mini-game developers, producing tens of thousands of mini-game products annually, presents significant opportunities for the space. The primary collaboration advantages for these developers are: Efficient Integration Advantage Independently developed mini-game container technology allows for seamless integration similar to WeChat mini-games. This ensures that mini-game products from the Chinese and Asian markets can be easily adapted and integrated into the PlaysOut platform, localized and deployed for global traffic access. Developer Trust Relationships Establishing strong relationships with developers sets the stage for seamless global positioning. Some notable groups in the space include  4399, Huaxia Leyou, Beijing Haibi, Stars Interactive Entertainment, Shanghai Coolreal, Chengdu MUMU, and Beijing Aodun Network Technology. Web3 Deployment Expertise
Creating and facilitating an environment for developers and games entering the Web3 space through access toessential tools such as wallet, swap, and token infrastructures enables seamless transitions and optimization of traffic performance and behavior across various game titles. Efficient Integration Advantage Independently developed mini-game container technology allows for seamless integration similar to WeChat mini-games. This ensures that mini-game products from the Chinese and Asian markets can be easily adapted and integrated into the PlaysOut platform, localized and deployed for global traffic access. Efficient Integration Advantage Independently developed mini-game container technology allows for seamless integration similar to WeChat mini-games. This ensures that mini-game products from the Chinese and Asian markets can be easily adapted and integrated into the PlaysOut platform, localized and deployed for global traffic access. Developer Trust Relationships Establishing strong relationships with developers sets the stage for seamless global positioning. Some notable groups in the space include  4399, Huaxia Leyou, Beijing Haibi, Stars Interactive Entertainment, Shanghai Coolreal, Chengdu MUMU, and Beijing Aodun Network Technology. Developer Trust Relationships Establishing strong relationships with developers sets the stage for seamless global positioning. Some notable groups in the space include  4399, Huaxia Leyou, Beijing Haibi, Stars Interactive Entertainment, Shanghai Coolreal, Chengdu MUMU, and Beijing Aodun Network Technology. Web3 Deployment Expertise
Creating and facilitating an environment for developers and games entering the Web3 space through access toessential tools such as wallet, swap, and token infrastructures enables seamless transitions and optimization of traffic performance and behavior across various game titles. Web3 Deployment Expertise
Creating and facilitating an environment for developers and games entering the Web3 space through access toessential tools such as wallet, swap, and token infrastructures enables seamless transitions and optimization of traffic performance and behavior across various game titles. One of the core challenges faced by web3 games is integrating into the broader gaming industry. The vision centers on creating a self-sustaining ecosystem where all participants — game developers, players, investors, and gaming platforms — are driven by more than just profit. However, it is impossible to ignore profit as a driver, and the gaming industry is among the top 10 most profitable industries globally, serving as a strong monetization terminal for the internet industry. Ideally, the next WeChat of web3 gaming would become a true global mini-games publisher, and combined with the intrinsic profitability of games with web3, they would ultimately elevate the overall web3 ecosystem and bring entertaining and profitable games and players into the space. PlaysOut’s platform and SDK solution tackles the profitability challenges faced by game developers and traffic providers worldwide, ensuring that everyone has the opportunity to earn rewards for their effort and engagement with games. The profitability of games serves as the underlying support for PlaysOut, upon which web3 empowerment is built, amplifying the value of excellent products. PlaysOut is poised to revolutionize web3 gaming by fostering a thriving community through innovative developer and gamer tools. This focus on a healthy ecosystem will attract high-quality games and players while also engaging better with the existing audience. By setting a new standard for transitioning web2 games and building a unified platform, PlaysOut is on track to become a central force in the future of gaming, where the best of both worlds collide. About Jassem Osseiran About Jassem Osseiran About Jassem Osseiran Jassem ("J") is an experienced entrepreneur and operational investor based in London. With a deep understanding of global finance gained through his Economics degree at the University of San Francisco, Jassem began his career as a tech venture builder with Rocket Internet in New York. He then shifted his focus to the Middle East and North Africa (MENA), managing media portfolios and founding startups tailored to the region. A sought-after advisor for high-growth brands and cutting-edge technology platforms, Jassem has orchestrated multiple successful exits, including a notable IPO on the London Stock Exchange. In 2020, he founded 611 Capital Investments, concentrating on investments, incubation, and market scalability across the EU, the Middle East, and Asia. Currently, Jassem is the founder and chief strategy officer of PlaysOut Technologies, spearheading the commercial framework for seamlessly integrating mini-games within super apps. This initiative represents a significant shift in the digital ecosystem, leveraging evolving Web3 technologies to unlock immense potential for mini-games in the digital space. Jassem remains a close advisor and partner to multiple global institutions and key opinion leaders. About PlaysOut About PlaysOut About PlaysOut PlaysOut is a globally oriented open platform for mini-programs that fully aligns with the Weixin Mini-Program framework. It provides SDK interfaces for super apps, offering a rich array of mini-program and mini-game content. Additionally, PlaysOut streamlines the integration process for developers, providing convenient tools and seamless access to mini-program and mini-game products. PlaysOut PlaysOut This story was distributed under HackerNoon’s Business Blogging Program. Learn more about the program here. This story was distributed under HackerNoon’s Business Blogging Program. Learn more about the program here . here here

PlaysOut: Bridging Worlds, One Mini-Game at a Time

প্লেসআউট: ব্রিজিং ওয়ার্ল্ডস, এক সময়ে একটি মিনি-গেম

PlaysOut: Welten verbinden, ein Minispiel nach dem anderen

PlaysOut: Uniendo mundos, un minijuego a la vez

PlaysOut : Bridging Worlds, un mini-jeu à la fois

प्लेज़आउट: ब्रिजिंग वर्ल्ड्स, एक समय में एक मिनी-गेम

PlaysOut: ミニゲームで世界をつなぐ

PlaysOut: 세계 연결, 한 번에 하나의 미니 게임

PlaysOut: unindo mundos, um minijogo de cada vez

PlaysOut: соединяя миры, по одной мини-игре за раз

PlaysOut: Dünyalar Arasında Köprü Kurmak, Her Seferinde Bir Mini Oyun

PlaysOut: Cầu nối các thế giới, mỗi lần một trò chơi nhỏ

PlaysOut：连接世界，每次玩一款迷你游戏

Attributing for one-fifth of the planet's population, China is undeniably an economic superpower that's empowered to become a global blockchain leader in terms of adoption, utility, and overall innovation. The People's Republic of China is not fooling around when it comes to innovation. When President Xi Jinping is assured of something is important for China's succession as a global economic and technopolitical center in the foreseeable future, it becomes instantly important for Chinese government bodies, domestic enterprises, and even individual citizens. Have in mind, we're talking about a single Asian country that attributes for near 1/5 of our current planet's population. That's right. So, why do I think it's too late to compete with China's urge to spearhead the blockchain revolution? The answer is quite complex and based on several factors that most of the time are unparalleled with each other, making it even harder to spot and include as an influential factor when measuring China's overall rise to global power. A Brief Take On China's Metamorphosis Into A Hi-Tech Conglomerate Without a doubt, China is one of the fastest-growing economies on the map. Take Shenzhen for example. A former village that was mainly occupied by local rice farmers, turned into the Asian alternative to Silicon Valley in less than a couple of decades. In Shenzhen, money is the same as the numbers behind your smart hand-held device's screen. Although China is the world's first cashless society, and on a national scale, one could say that , where exotic tech innovations are being developed and tested as an essential part of the norm, before spreading to the rest of mainland China. Shenzhen is China's beta server Pit Stop: Chinese “Silicon Valley”, Shenzhen to launch its own Cryptocurrency That is mostly due to China's long-year experience in consumer electronics production, software engineering, and mass marketing strategies that helped most of the western titans including but not limited to Apple, Microsoft, BOSCH, and pretty much every major international-grade brand operating in the computing field, to become what they are today. Of course, cheap Chinese hands, and low-quality components are a thing of the past now, with the majority of local manufacturers that used to work for Apple, Nike, and HP launching their own ventures citing that . producing iPhones is just "not enough" The Chinese not only learned to copy-paste western patents, but they improved them over time, managing to create twice as good products, for half the price. Some of them like Huawei, Xiaomi, Li-Ning, and DJI, are taking over the global scene by the hour. Furthermore, simplified Chinese language (Mandarin, Pinyin) is a collective communication protocol instead of individualistic as happens in most western societies. For example, we would say something like , while Chinese would wrap the same ideological concept with something like . *Similar to native American names, Chinese names represent a deeper meaning and are not self-centered and individualistic. Sure, that plays a major role in China's governing model, analogizing they are running a modern form of communism, yet it is also the driving force behind China's rise to power. "Bob did that yesterday" "This happened yesterday when the guy who runs fast* did that" In Shenzhen, every single action, whether natural, pre-designed, or humane is monitored, recorded, and evaluated by a network supported by AI, IoT, and Blockchain, in order to conceive and prevent malicious or harmful actions with minimum bias. For example, a trespasser in the hi-tech Metropolis, won't be stopped by a law enforcement officer in case the later spots the violation. Instead, high-quality real-time cameras powered by facial recognition software identify the actor and notify him with the sum of the fine he has to cover for his violation, directly to his hand-held device, and all that without human interference. Some would argue that this is a Black Mirror inspired dystopia. I'd say, , rather than a sci-fi scenario. Black Mirror is a documentary of the past for Shenzhen netizens Another stunning, yet practical example would allow Chinese beggars to carry a QR code on their neck instead of jars and hats, in order to accept donations and philanthropy, considering paper money is something Chinese can only see in the results of a relevant web search. Too far-fetched you might think? Efficient I'd correct. China Grabs Blockchain By Its Hand If you consider the above as something that's already taking place under physical circumstances, blockchain technology should come as a natural component for such complex and autonomous information flow systems. The Chinese are not into blockchain because of cryptos, neither they desperately seek to profit from digital currencies, having in mind, their economy is digital for over a decade now. Relevant Article: China Urges Its Citizens To Avoid Illegal Investment Opportunities Blockchain architecture is basically an essential pillar for any autonomous society, and China is not keen on losing not a single minute of reaching that reality point first. Recording, archiving, managing, and distributing data (which by-the-way is the new oil), is what distributed ledger networks do best, and on top of that in a decentralized manner. President Xi Jinping not only was the first government leader to blockchain adoption, urging local government agencies and organizations, as well as Chinese companies to integrate the new technology asap, but he is one of these guys who are more into Artificial General Intelligence, Machine Economies, Automated Public Administration and you can confirm that by taking a look at his library, listen, or read his interviews and experience the overall growth of Chinese hi-tech products and services even as overseas consumers. publically promote On the other hand, western leaders, including Trump and Johnson are subjects of adult entertainment magazines, and the only reason they are into hi-tech, and blockchain more specifically, would be the pressure they get from their advisory board and domestic enterprises in order to avoid being left in the dust by China's or any other country's speeding on the matter. It's one thing when a country's leader comprehends what blockchain is really capable of, and a totally alien thing when a country's leader is not even interested in learning blockchain, considering it too vague and complex to understand, too risky to invest, and too early to start digging it with high-priority. Thankfully, there's more room for China. Recent Blockchain Developments In China In Machine-Gun Speed The (BSN) is China's first state-backed DLT where any local or international company is welcome to build and deploy dapps, smart-contracts, and DeFi tools executed on-chain. Blockchain Service Network Domestic payments processing giants of the likes of AliPay, WeChat, and UnionPay are in accordance with the government's vision and are not only experimenting with blockchain technology, but most of them are already utilizing it in their internal services, apps, and products. Read More: AliPay, Tencent, and Union Pay among the first to receive China’s gov-backed cryptocurrency Beijing is working on a municipal governance system based on blockchain, where literally every single gov-affiliated organization, legal entity, and SMEs are called to implement DLTs using a framework directly by the government as guidance. provided During the annual (WAIC) that takes place in Shanghai and has speakers of the level of Elon Musk and Jack Ma, blockchain technology is a native topic and it is being considered as the backbone for truly autonomous AI systems. Wang Jianwei, deputy director of the Information Technology Division at the Ministry of Industry and Information Technology that most economic centers in China, including the capital Beijing, Shanghai, and Shenzhen are already working on local and national blockchain projects, while some of them are already empowering every-day life activities and procedures by utilizing some sort of a DLT. World Artificial Intelligence Conference said For example, the Shanghai High Court has recently recording, registering, and accessing court data on blockchain successfully, meaning not only the Shanghai High Court will adopt this method to use in future courts, but . tested over 250 Chinese courts are intrigued by the method and shall follow suit accordingly Last but not least, Chinese hi-tech companies such as JD.com, Tencent, and Alibaba account for near five thousand (5,000) patents filings/year subjecting the term "blockchain" within their respective citations. Related Article: A Brief History Of Tencent Holdings And Its Rise To The Blockchain Era Why China Leaves No Space For Competition Truth is, China is already a global blockchain leader, although since we're at the early stage of DLT advancement, one could say other countries could follow up if not surpass the Asian titan. Unfortunately, that's not the case with Western countries, where blockchain is still directly associated with Bitcoin and other volatile digital assets, while the real networking benefits a DLT offers is totally absent in regulators' minds. Most regulators, gov-leaders, and financial institutions in Europe and the U.S. are unsure about how they should approach blockchain technology, and in addition, they are unable to point out the vast amount of benefits DLTs offer to future societies, where everything tends to be digital and automated. That's not just lack of comprehension, but lack of incentive, and most importantly lack of will to elevate to the next level. The EU is run by old folks nobody knows their names, and their current social status, regardless of its deadline, is not helping them, and by extension domestic regulators, in seeing further what they've already achieved. Read More: What Lawmakers Make Of The Term ‘Blockchain’ In 2020? For that reason, I don't believe any country could beat China in blockchain advancement and development, and again, not because they are unable to, but simply because it is not a top priority for the time being. We have priorities like green energy and data privacy. Something completely irrelevant and outdated, considering our data has been sucked for over five decades straight. It is idiotic at best to pretend we care about data privacy in 2020 or assume that data privacy is more important than what's yet to come in the next five decades. The Chinese don't pretend they have data privacy. They don't. The gov knows it, the companies who suck the data know it, and people know it. Therefore they focus on what they can do with that data instead of subjecting the concept of data privacy as an important aspect, especially when we're far ahead from the point where we could still take another path, one less tech-oriented. Would you disagree? Feel free to leave a comment in the relevant section below, or hit me on . Which country you suggest is most likely to attempt overthrowing China as the global blockchain leader in the next decade? Twitter

Why China Leaves No Space For Competition in Blockchain

You may have noticed a new Chinese company in the tech news recently: JD.com aka Jingdong. Formerly known as 360Buy, the company has been growing rapidly in the past year, increasing its market cap from $36.93B to $66B YTD. In terms of traffic, JD is now the 21st most visited website in the world. . Currently, JD is the largest Chinese retailer by revenue JD started operations in 1998 in a four-square-meter unit in Beijing with just $1800. The company was founded by Liu Qiangdong (also known as Richard Liu) as a consumer electronics retailer. In 2007, . JD began focusing on their logistics network so they could control all aspects of their supply chain Most of JD’s current revenue comes from its business-to-consumer (B2C) operations. What this means is that JD is similar to Amazon; they (the business) sell to individuals (the consumers). Consumer electronics are still a large portion of JD’s sales, with 85% of the company’s revenue in 2014 coming from CE sales. Since 2016, JD has been working with an unlikely partner: Walmart. . . In June of 2016, Walmart handed over its Chinese e-commerce business to JD in exchange for a 5% equity stake And in August of 2017, the companies ran the first joint JD-Walmart shopping festival JD is also connected to another Chinese tech titan: . Tencent acquired 15% of JD in 2014 and . JD and Tencent have been cooperating through WeChat, Tencent’s massively popular messaging client. Tencent in 2016 became JD’s largest shareholder with a 21.25% equity stake JD‘s closest rival is TMall, Alibaba’s B2C marketplace. As noted before, JD is heavily B2C focused, with ; Alibaba is the opposite, being far more known for its business-to-business (B2B) enterprise. “According to L2’s newly released Insight Report on Alibaba, over 90% of Index CPG brands maintained official flagships on Tmall in 2014 but only 26% had an official JD.com shop. Now, in 2017, the percentage of CPG brands with an official shop on JD.com has surged to 98%.” only ~6% of its revenue coming from B2C sales Source When compared to Amazon, JD looks like a small player in the B2C sector. For example, in 2016, Amazon’s revenue was $136B, while JD’s was meager $39B. At the time of writing, Amazon stands at a market cap of $474B, while JD is at $66B. While headquartered in Beijing, JD has been expanding internationally. JD has a general international site, and and . https://www.joybuy.com/ has been working on expanding into the Thai and Russian markets the Southeast Asian market in general Considering JD is now of considerable size, what strategic investments has it been making? In June 2017, , an online luxury goods marketplace. This deal allows JD to gain credibility in the luxury e-commerce space, while its competitor Alibaba remains ridiculed for knock-offs. the company invested $397 million into Farfetch In May of 2017, . is a ride-hailing and courier startup based in Indonesia. This move could help JD with last-mile logistics in Indonesian deliveries, and perhaps deliveries in other countries as Go-Jek expands. JD made a $1.2B investment in Go-Jek as part of the firms move to tap into Southeast Asia Go-Jek And, in August of 2015, JD made a $630M investment in Ele.me, a customer-to-customer meal ordering service based in Shanghai. For a full lists of JD’s investments, see their . Crunchbase profile When analyzing Chinese-based companies, its important to understand their standing with the party. In , how well you’re viewed in the eyes of the party is first and foremost. China In just the past few weeks, . He wrote that “we are far from a communist society, but … the light of communism will always shine on us.” JD’s founder has stated that with technological advancement, China will be able to have full Communism, with all companies being state-owned In addition, as said so eloquently in : another Medium article about JD I have two company-external reasons to be super bullish on JD as an investment in the Chinese economy. These two reasons are affected by government decisions: The Chinese government wants their core online economy to be made up of Chinese-founded businesses The Chinese government … wants to protect their citizens in a way to drive economic growth It appears that Richard, and JD as a whole, is properly following the party lines and is positioned nicely for success. Full disclosure: I am currently employed by Amazon.com and all opinions in this piece are my own.

What is JD.com?

The Chinese tech user base has grown at an astronomical pace over the past decade. . In addition to growing at a neck-breaking pace, the industry has been becoming increasingly transparent to western audiences. In 2016, online spending in China was greater than that of the US and UK combined Currently, the Chinese tech landscape is dominated by 3 kingpins: (Market Cap $374B), Tencent ($327B), and Baidu ($64B). Alibaba The largest of the trio is Alibaba, led by Jack Ma. Starting off initially as a B2B (business-to-business) website for connecting manufacturers and potential buyers outside of , Alibaba has transformed into the second-largest e-commerce company in the world by revenue, behind Amazon.com. For more on Alibaba, see published in . China my article HackerNoon The next largest is Tencent, which also tends to be the lesser known of the trio in the US. The company is primarily mobile-focused, with popular apps for social media and gaming, with investments in just about every other aspect of tech. Tencent’s owns WeChat, one of the largest messaging apps in the world, with over 938M monthly users. But Tencent is also the largest gaming company in the world, with its Honor of Kings being the most downloaded app in the world for Q1 of 2017. . Tencent recently announced that it will be making its most successful game (Honor of Kings) available in the US and Europe in the fall The smallest of the trio is Baidu, commonly referred to as “the Google of China” in Western media. Baidu owns the most popular search engine in China, but also has operations like “Baidu Baike” (a Chinese Wikipedia) and Baidu Music. These companies don’t operate in strict silos; they’ve all expanded into most of the technology sub-sectors. See the below graphic from The Boston Consulting Group: Source: What China Reveals About the Future of Shopping . In the last two years, . Partnering with one of the Chinese giants is almost a necessity for Western tech companies who want to penetrate the Chinese market. And their investments do not stop at China’s borders these companies (and their follower JD.com) invested $5.6 billion into US tech companies In early 2016, and in 2014, they became an investor in Lyft. Alibaba has been investing heavily into Augmented Reality startups across the globe: they have equity in , , and even the mysterious (in the form of a ~$800M investment). See: . Alibaba became the 4th largest owner of Groupon Lumus WayRay Magic Leap Top Funded AR Startups in 2017 Tencent too is growing outside of the Great Wall. The company was an early investor in Snapchat and bought Supercell, the makers of the popular mobile game Clash of Clans, for a whopping $8.6 billion. Tencent even owns Riot Games, the makers of the competitive game League of Legends, which had ~67M monthly active users as of 2014 .They’ve also invested $1.2 billion into Flipkart and $1.8 billion into Tesla. Source: The Rise of China’s Tech Firms in Five Charts Even their investments into individual companies are intertwined. All of the aforementioned companies have investments in Didi Chuxing, the Uber of China. One can see how none of these companies are comfortable simply staying inside of their specialty; much like their US counterparts, they’ve each invested heavily in other aspects of tech. The Chinese tech industry has been largely ignored in US financial/tech media, but as the companies expand and grow, the days of this trend are numbered. Further Reading: China’s Internet Giants Go Global Chinacast: Alibaba The Rise of China’s Tech Firms in Five Charts Meet the 8 Chinese Tech Giants You Need to Know The Next Disney Will Come from China and Its Name Is Tencent Full Disclosure: I am employed by Amazon.com and the viewpoints in this article are entirely my own.

The Chinese Tech Trio And Their Holdings

Six months ago, I wrote about for our data management system. Back then, we were struggling with the auto-generation of SQL statements. As days pass, we have made progresses big enough to be references for you, so here I am again. why we replaced ClickHouse with Apache Doris as an OLAP engine We have adopted Large Language Models (LLM) to empower our Doris-based OLAP services. LLM + OLAP Our incentive was to save our internal staff from the steep learning curve of SQL writing. Thus, we used LLM as an intermediate. It transforms natural language questions into SQL statements and sends the SQLs to the OLAP engine for execution. Like every AI-related experience, we came across some friction: LLM does not understand data jargons, like "fields", "rows", "columns" and "tables". Instead, they can perfectly translate business terms like "corporate income" and "DAU", which are basically what the fields/rows/columns are about. That means it can work well only if the analysts use the exact right word to refer to the metric they need when typing their questions. The LLM we are using is slow in inference. It takes over 10 seconds to respond. As it charges fees by token, cost-effectiveness becomes a problem. Although the LLM is trained on a large collection of public datasets, it is under-informed of niche knowledge. In our case, the LLM is super unfamiliar with indie songs, so even if the songs are included in our database, the LLM will not able to identify them properly. Sometimes our input questions require adequate and latest legal, political, financial, and regulatory information, which is hard to be included in a training dataset or knowledge base. We need to connect the LLM to wider info bases in order to perform more diversified tasks. We knock these problems down one by one. 1. A semantic layer For problem No.1, we introduce a semantic layer between the LLM and the OLAP engine. This layer translates business terms into the corresponding data fields. It can identify data filtering conditions from the various natural language wordings, relate them to the metrics involved, and then generate SQL statements. Besides that, the semantic layer can optimize the computation logic. When analysts input a question that involves a complicated query, let's say, a multi-table join, the semantic layer can split that into multiple single-table queries to reduce semantic distortion. 2. LLM parsing rules To increase cost-effectiveness in using LLM, we evaluate the computation complexity of all scenarios, such as metric computation, detailed record retrieval, and user segmentation. Then, we create rules and dedicate the LLM-parsing step to only complicated tasks. That means for the simple computation tasks, it will skip the parsing. For example, when an analyst inputs "tell me the earnings of the major musical platforms", the LLM identifies that this question only entails several metrics or dimensions, so it will not further parse it but send it straight for SQL generation and execution. This can largely shorten query response time and reduce API expenses. 3. Schema Mapper and external knowledge base To empower the LLM with niche knowledge, we added a Schema Mapper upstream from the LLM. The Schema Mapper maps the input question to an external knowledge base, and then the LLM will do parsing. We are constantly testing and optimizing the Schema Mapper. We categorize and rate content in the external knowledge base, and do various levels of mapping (full-text mapping and fuzzy mapping) to enable better semantic parsing. 4. Plugins We used plugins to connect the LLM to more fields of information, and we have different integration methods for different types of plugins: : This is especially useful when we need to "teach" the LLM the latest regulatory policies, which are often text files. Firstly, the system vectorizes the local text file, executes semantic searches to find matching or similar terms in the local file, extracts the relevant contents and puts them into the LLM parsing window to generate output. Embedding local files : The marketplace is full of third-party plugins that are designed for all kinds of sectors. With them, the LLM is able to deal with wide-ranging topics. Each plugin has its own prompts and calling function. Once the input question hits a prompt, the relevant plugin will be called. Third-party plugins After we are done with above four optimizations, the SuperSonic framework comes into being. The SuperSonic framework Now let me walk you through this : framework An analyst inputs a question. The Schema Mapper maps the question to an external knowledge base. If there are matching fields in the external knowledge base, the question will not be parsed by the LLM. Instead, a metric computation formula will trigger the OLAP engine to start querying. If there is no matching field, the question will enter the LLM. Based on the pre-defined rules, the LLM rates the complexity level of the question. If it is a simple query, it will go directly to the OLAP engine; if it is a complicated query, it will be semantically parsed and converted to a DSL statement. At the Semantic Layer, the DSL statement will be split based on its query scenario. For example, if it is a multi-table join query, this layer will generate multiple single-table query SQL statements. If the question involves external knowledge, the LLM will call a third-party plugin. Example: To answer whether a certain song can be performed on variety shows, the system retrieves the OLAP data warehouse for details about the song, and presents it with results from the Commercial Use Query third-party plugin. OLAP Architecture As for the OLAP part of this framework, after several rounds of architectural evolution, this is what our current OLAP pipeline looks like. Raw data is sorted into tags and metrics, which are custom-defined by the analysts. The tags and metrics are under unified management in order to avoid inconsistent definitions. Then, they are combined into various tagsets and metricsets for various queries. We have drawn two main takeaways for you from our architectural optimization experience. 1. Streamline the links Before we adopted Apache Doris, we used to have ClickHouse to accelerate the computation of tags and metrics, and Elasticsearch to process dimensional data. That's two analytic engines and requires us to adapt the query statements to both of them. It was high-maintenance. Thus, we replaced ClickHouse with Apache Doris, and utilized the functionality to connect Elasticsearch data to Doris. In this way, we make Doris our unified query gateway. Elasticsearch Catalog 2. Split the flat tables In early versions of our OLAP architecture, we used to put data into flat tables, which made things tricky. For one thing, flat tables absorbed all the writing latency from upstreams, and that added up to considerable loss in data realtimeliness. For another, 50% of data in a flat table was dimensional data, which was rarely updated. With every new flat table came some bulky dimensional data that consumed lots of storage space. Therefore, we split the flat tables into metric tables and dimension tables. As they are updated in different paces, we put them into different data models. : We arrange metric data in the Aggregate Key model of Apache Doris, which means new data will be merged with the old data by way of SUM, MAX, MIN, etc. Metric tables : These tables are in the Unique Key model of Apache Doris, which means new data record will replace the old. This can greatly increase performance in our query scenarios. Dimension tables You might ask, does this cause trouble in queries, since most queries require data from both types of tables? Don't worry, we address that with the Rollup feature of Doris. On the basis of the base tables, we can select the dimensions we need to create Rollup views, which will automatically execute . This relieves us of the need to define tags for each Rollup view and largely speed up queries. GROUP BY Other Tricks In our experience with Apache Doris, we also find some other functionalities handy, so I list them here for you, too: 1. Materialized View A Materialized View is a pre-computed dataset. It is a way to accelerate queries when you frequently need to access data of certain dimensions. In these scenarios, we define derived tags and metrics based on the original ones. For example, we create a derived metric by combining Metric 1, Metric 2, and Metric 3: . Then, we can create a Materialized View for it. According to the Doris release schedule, version 2.1 will support multi-table Materialized Views, and we look forward to that. sum(m1+m2+m3) 2. Flink-Doris-Connector This is for Exactly-Once guarantee in data ingestion. The Flink-Doris-Connector implements a checkpoint mechanism and two-stage commit, and allows for auto data synchronization from relational databases to Doris. 3. Compaction When the number of aggregation tasks or data volume becomes overwhelming for Flink, there might be huge latency in data compaction. We solve that with Vertical Compaction and Segment Compaction. Vertical Compaction supports loading of only part of the columns, so it can reduce storage consumption when compacting flat tables. Segment Compaction can avoid generating too much segments during data writing, and allows for compaction while writing simultaneously. What's Next With an aim to reduce costs and increase service availability, we plan to test the newly released Storage-Compute Separation and Cross-Cluster Replication of Doris, and we embrace any ideas and inputs about the SuperSonic framework and the Apache Doris project.

LLM-Powered OLAP: the Tencent Experience with Apache Doris 

LLM-চালিত OLAP: Apache Doris-এর সাথে Tencent অভিজ্ঞতা

LLM-gestütztes OLAP: die Tencent-Erfahrung mit Apache Doris

OLAP impulsado por LLM: la experiencia Tencent con Apache Doris

OLAP basé sur LLM : l'expérience Tencent avec Apache Doris

एलएलएम-संचालित ओएलएपी: अपाचे डोरिस के साथ टेनसेंट अनुभव

LLM を利用した OLAP: Apache Doris を使用した Tencent エクスペリエンス

LLM 기반 OLAP: Apache Doris를 통한 Tencent 경험

OLAP baseado em LLM: a experiência da Tencent com Apache Doris

Yüksek Lisans Destekli OLAP: Apache Doris ile Tencent Deneyimi

OLAP do LLM hỗ trợ: Trải nghiệm Tencent với Apache Doris

LLM 支持的 OLAP：腾讯使用 Apache Doris 的经验

In this blog, guest writers Bing Zheng (Software Engineer at Tencent), Baolong Mao (Senior Software Engineer at Tencent), Zhizheng Pan (Senior Software Engineer at Tencent) shares the practice of running a 1000-node Alluxio cluster to speed up Ceph storage for game AI training at Tencent. The original content was published on Alluxio's Blog (Disclaimer: The author is a Founding Member @Alluxio). Overview Alluxio is an open-source data orchestration platform for AI and data analytics applications. With an increasing number of data analytics and AI applications being containerized, Alluxio is becoming the top choice for large organizations as an intermediate layer to accelerate analytics and model training. We have adopted Alluxio to solve the data challenges in distributed computing for our game AI offline training. Traditionally, this is solved by: Image packaging: Although resource isolation is better in this way, the capacity of the image caching is limited. Frequent data updates result in repackaging and re-deploying the image, forcing the container services to stop each time. Load on-demand: Although this architecture is simple, we have to pull data remotely, which results in poor performance. Deploy locally: This solution brings good data locality. However, authentication and DevOps are challenging. These solutions have their pros and cons. With Alluxio, we have solved the challenges in a better way. The implementation of Alluxio can significantly improve the concurrency limit of AI workloads without adding additional costs. Also, the business side is not impacted by the change of storage system because AI workloads are still using the original POSIX interface to access Alluxio. In this article, we discuss the implementation and optimization of Alluxio for our game AI platform. Business Background and Data Challenges The game AI offline training consists of supervised learning and reinforcement learning. Generally, supervised learning is divided into three stages: feature computation (feature extraction), model training, and model evaluation. Reinforcement learning requires feature computation as well. In the game AI offline training, feature computation requires reducing game match information to generate the feature data for model training through statistics and computation. The information recovery of the game match is made through the corresponding game dependencies (game ontology, game translator, game replay tool, etc.), which generally range from 100MB to 3GB in file size. Furthermore, due to the versioning nature, processing specific game match information requires a specific version of the game dependency. The game dependency on the storage side is called gamecore, which corresponds to a Linux client of a specific game version. Gamecore can be stored on local storage for better performance and stability, but it is expensive and requires authentication of local machines. Alternatively, gamecore can be stored in distributed storage, such as Ceph, which is faster and easier to deploy. Still, the disadvantage is that the MDS, metadata server daemon of cephfs, will become a bottleneck. Usually, thousands of containers are scheduled in feature computing jobs. Each container will start several processes. At the beginning of the job, tens of thousands of processes will access the same gamecore in parallel with hundreds of gigabytes of data reads per minute on the storage side. The metadata pressure will fall solely on MDS because these files are small. In addition, the latency between storage and compute is usually high, especially when they are remote, which may lead to a higher job failure rate. With careful consideration, we decided to introduce Alluxio on Ceph in order to address the current data challenges. In this implementation, we got strong support from the game AI team and the operation team. The game AI team gave us the overall business background and coordinated the production environments so that we could fully test the solution before productionalization. The Operations Management team also provided significant assistance with deployment architecture support and resource coordination. Architecture with Alluxio In the data analytics ecosystem, Alluxio sits between the data-driven compute framework and the storage systems. Alluxio unifies the data stored in these different storage systems, providing flexible APIs and a global namespace for data-driven applications. In our use case, the under storage is cephfs, and the data application is feature computing. Alluxio, as an intermediate layer, is well suited as a distributed caching solution that is ideal for optimizing reads-heavy, massive small files, and high concurrent access scenarios of feature computing business. We have gained several benefits from Alluxio: Alluxio provides good support on the cloud, making it easy to deploy and scale up and down Alluxio clusters on computing platforms. Alluxio can be deployed co-located with the computation side. By deploying Alluxio workers on the same node of the compute, we can cache data locally and improve I/O throughput. The Alluxio worker uses the memory disk of the computing node, which can provide sufficient cache space, and loads hot data from cephfs into the worker through distributedLoad. The following diagram shows the architecture of Alluxio in the game AI platform. In production, we want to support concurrent jobs on 4000 CPU cores to run stably. We configure quad-core CPU for each pod of game match feature extraction, providing 1000 pods concurrency on the application side. Each pod embeds an alluxio-fuse sidecar container as a client, and data read requests from the application can directly access data from Alluxio through POSIX with the path mounted by alluxio-fuse. The Alluixo cluster master node is configured in HA mode, and the worker number is 1000. We want to co-locate the application pods and worker pods to one node as much as possible so that we can use the domain socket to improve the read performance further. Before the application goes live, the hot data of gamecore in cephfs is preloaded to the Alluxio worker through distributedLoad to warm up. R&D and Performance Tuning The Alluxio cluster is currently supporting feature computing and is a large-scale deployment (1000+ worker nodes) in AI and machine learning platform. Such extensive concurrent access poses challenges to the master’s resilience. We have done tuning and added new features to achieve the best results during our practice. R&D For feature computing + Cephfs, we designed and implemented HCFS + cephfs-Hadoop + libcephfs and libcephfs directly. We worked with the Alluxio community to design and implement the feature of switching leader to a specified master node for HA. We abstracted the ratis-related functionality into a separate ratis-shell tool ( ), which can set the priority of each master by sending setConfiguration requests to the ratis server directly, followed by sending transferleader request to ensure that the leader has switched to the specific node we want. Ratis-shell works with Alluxio and Ozone, and all other applications that use ratis. https://github.com/opendataio/ratis-shell We added the ability to dynamically change the configuration to modify specific cluster parameters online to optimize the configuration with minimal impact on the application. As shown below, the updateConf API was added between the Client and the Master. Through the API, the configuration change requests can be sent to the Alluxio Master, and the internal config hash will be changed after the master updates the configuration. In addition, other services connected to Alluxio Master, such as Client and worker, will periodically synchronize their config hash with the master and perceive the configuration changes and thus synchronize the configuration changes. On the Alluxio FUSE client side, for the all-read scenario, we have turned on the kernel cache and the metadata cache on the Alluxio client side to further improve the read performance. We also optimized the metadata cache. For example, when the metadata of the under storage changes, we can actively invalidate the metadata cache and recache it, which is very flexible. We also fixed some bugs after Alluxio FUSE turns on the LocalCache. We added metrics of master busy status, ratis metrics, OS, JVM, GC, cache hit rate, and many other valuable metrics to enrich Alluxio's metrics system. We developed a feature to view stacks of Alluxio's key processes to track cluster status. We leveraged the warm-up feature of distributedLoad to fix an OOM issue in the Alluxio Job Service when executing many distributedLoads. Configuration Optimization Turn off passive cache. By default, Alluxio turns on the passive cache feature Allluxio.user.file.passive.cache.enabled=true, so if the client finds that the data block is not in the local worker, it will copy from the remote worker to the local worker, resulting in many copies of each worker. This configuration adds significant metadata pressure to the Alluxio master in our scenario with 1000 workers. Our tests have shown that the performance benefit of this locality is very small, so we turned it off to reduce the pressure on the master. Use internal kona jdk11 and tune the jvm parameters for master and worker.  By using konajdk11 + g1gc, there were no longer leader master switching issues due to fullgc. Turn off auditlogging. With the help of the jvm team, we identified that the enabled auditlog is the bottleneck of the all-read scenario. By setting alluxio.master.audit.logging.enabled=false to turn off auditlogging, the throughput improved by 7x. The flame chart captured by kona-profiler reveals that using the ROCKSDB metadata management has increased the performance overhead, and we plan to switch Alluxio to HEAP metadata management in the future. Benchmarking Test We use feature computing jobs of a moba (multi-player online battle arena) game to compare Alluxio (UFS is cephfs) and cephfs, respectively. The cluster information of Alluxio in the test is as follows: Alluxio master: 3 masters deployed in High Availability mode. Alluxio worker: 1000 workers, about 4TB of storage space. Application pods: 1000, each pod is in parallel with 4 cores. Benchmarking test: A moba game AI feature computing job (containing 250,000 matches).** ** Test results are as follows: Client Jobs Completion Failure Time Failure rate alluxio-fuse 250000 248152 1848 2h46min 0.73% ceph-fuse 250000 242930 7070 2h40min 2.8% Accessing Alluxio + cephfs has a failure rate of 0.73% and a job duration of 2h 46min. Accessing cephfs directly has a failure rate of 2.8% and a job duration of 2h 40min. Both solutions pass the tests above and the failure rate is within an acceptable range. With Alluxio + cephfs, failure rates are lower. The graph above shows the metadata stress metrics (rpc count and qps of mds) for Alluxio and cephfs. There is a pressure hit at the beginning of the job, and then the pressure of master metadata gradually decreases. With Alluxio, the qps of ceph mds is almost zero, which indicates that Alluxio is reducing most of the pressure. The Read Remote, Read UFS, and Read Domain metrics are used to observe the locality of the data. We can see that Remote Read and Read Domain account for most of the read traffic, with most of the read traffic being remote reads between workers and local domain socket reads, and very few from UFS. The above figure shows the heap memory change curve during job execution after using kona jdk11. While using the official version of kona jdk11 before, the master encountered a leader switching problem due to the long gc time. After replacing kona jdk11, this no longer happened, and master has been smoother. Future Work Increase Throughput Limit Currently, Alluxio master is under 7000 cores concurrency pressure, and we have found that the callqueue of master is backlogged, and the masterStressBench tool has measured the throughput capacity of nearly 210,000 rpc requests per second. To make the alluxio cluster support endless concurrent access, tuning a single cluster is not enough. We need to design the overall architecture to support higher concurrent access. Decouple Application and Alluxio FUSE with Alluxio CSI Currently, Alluxio-FUSE is located in the same pod as the application as a sidecar, so the application container and Alluxio-FUSE container must be managed together. By decoupling Application and Alluxio FUSE with Alluxio CSI, we can independently maintain the pod and yaml on the application side. Build the Alluxio Cluster Management System on Kubernetes We maintain a helm chart for operating and maintaining Alluxio clusters based on the helm chart template provided by Alluxio. Still, we want to go further and operate each pod and container based on the Kubernetes API and interactively execute the commands such as the mount and unmount of under storage, job service visualization, and load free services. Summary With Alluxio, we have been able to support 4000 concurrent CPU cores smoothly for the game AI feature extraction use case. From our practice, Alluxio has proven to offset most of the metadata pressure for the under storage and reduce the failure rate of jobs to a satisfactory range. Such large-scale, high-concurrency scenarios posed challenges to Alluxio, and we have contributed to the open-source community and improved the stability, usability, and DevOps ability of Alluxio, making it more adaptable in the future. Reference Kaiwu Game AI Offline Experiment Platform ratis-shell Alluxio Documentation Chinese Version of This Case Study About the Authors Bing Zheng Software Engineer at Tencent Responsible for the R&D of big data system and distributed storage. Baolong Mao Senior Software Engineer at Tencent, Alluxio PMC member, Apache Ozone committer, Alluxio OTeam open source collaborative team leader at Tencent Responsible for the development and implementation of Alluxio at Tencent, and the development of the Apache Ozone file system. Zhizheng Pan Senior Software Engineer in Kaiwu game AI platform team at Tencent Responsible for developing and operating the machine learning platform for game AI use cases. (Disclaimer: The author is a Founding Member @Alluxio)

Architecting a Thousand-Node Data Orchestration Platform to Accelerate Game AI Training at Tencent

This article describes how engineers in the Data Service Center (DSC) at Tencent PCG (Platform and Content Business Group) leverages Alluxio to optimize the analytics performance and minimize the operating costs in building , a real-time data analytics platform. Tencent Beacon Growing Specifically, by using Alluxio as a distributed caching layer, the HDFS scanning performance for Impala SQL queries is accelerated by 2x out of the box without further optimizations. The original article was published on by guest writer Honghan Tian. Alluxio's Engineering Blog Tencent’s Big Data Analytic App Beacon Growing is an analytics platform that provides analyzers insights of user behaviors in realtime. Beacon comes with both built-in and customized analytic models, such as RETENT, FUNNEL, PATH, etc. Combining the functions of A/B testing and user profiles, developers can select promotional plans more easily. Under the hood, Beacon is powered by a stack of open source technologies. Particularly for the context of this article, we used Impala to query data in Parquet format stored in HDFS. Here is the overview of the software stack currently being deployed by Beacon, to give the readers a better understanding of the system. Challenges Before In Tencent, we build systems to handle data at the Internet-scale. For example, the scan range of a typical query can be 100 billion rows, with hundreds of concurrent queries. To speed up the queries, we have applied optimizations like . In practice, there were still a few issues observed: page index in Parquet files to reduce the I/O cost in scan operations : When more than 100 queries are running concurrently and scanning HDFS, we observed that the usage of host disk was fixed at 100%. I/O Bottleneck under high concurrency :  Insert operations such as “SORT BY” during table creation are very slow due to files merge under HDFS-parquet sort. Note that in our , “SORT BY” helps meet query SLAs without over partitioning the table. Expensive writes application Why Alluxio Based on the above observations, we decided to build a caching layer using SSDs. However, HDFS is not designed to leverage heterogeneous storage. We deployed Alluxio to leverage the speed advantage of SSDs and the capacity advantage of HDDs. The new architecture is shown in the following figure: There are a few changes and improvements observed after integrating Alluxio into Tencent Beacon infrastructure: The range of the average load of HDFS was significantly reduced from  80% ~ 100% down to 50% ~ 70%. The I/O time is also reduced by 50% or more. Better Resource Utilization: : With faster data retrieval, I/O-intensive queries are sped up by 244% with a median improvement of 121% across all queries. Higher Query Performance Alluxio greatly reduces the I/O consumption of hotspot and large queries to the entire HDFS cluster, reducing the overall query failure rate by more than 5% Higher Cluster Stability: Faster execution also reduces the query time in waiting-queue, resulting in 29% reduction in the timeout query failure rate. Reduced Timeout Failure Rate: One of the most valuable features of Alluxio in our use case is . It helps share the same namespace across multiple clusters, making it easier for Impala to access the data resources of multiple HDFS clusters. Unified Namespace The following table is a comparison of some data： Next steps: Locality Groups There are a few on-going projects on the horizon to integrate Alluxio at Tencent PCG. First, we plan to further reduce the operating work by deploying a large Alluxio cluster with worker processes spanning multiple Impala clusters. In other words, Alluxio and Impala processes will be co-located at the same server to reduce network traffic. We are currently experiencing some and working with the core team to address them. issues Second, we want to leverage the Alluxio to further improve Impala performance and eliminate remote reads of Alluxio workers with improved data locality. tiered locality group Specifically, with one but much larger Alluxio cluster deployed, we will divide Alluxio workers nodes into different ‘locality groups’. As a result, when queries read data from different HDFS clusters, data will be cached into the nearest locality group to reduce I/O cost. This feature is an analogy of the in Clickhouse. “layer” configuration” Third, once locality grouping described above is achieved, locally deployed Impala computing groups can be used to elastically expand or shrink computing resources based on different Impala clusters for different business uses. Conclusion In our project “Beacon Growing”, we have deployed Alluxio to improve Impala performance by 2.44x for IO intensive queries and 1.20x for all queries. The query failure rate due to timeout is also reduced by 29%. In the future, we foresee it can reduce disk utilization by over 20% for our planned elastic computing on Impala.

Stories, News & Mentions about TENCENT

Tencent’s Stories on HackerNoon

Tencent's News & Mentions