A unified serverless stream processing platform to manage, secure and process all your event streams

DeltaStream

T3Go is China’s first platform for smart travel based on the Internet of Vehicles. In this article, Trevor Zhang and Vino Yang from T3Go describe the evolution of their data lake architecture, built on cloud-native or open-source technologies, including Alibaba OSS, Apache Hudi, and Alluxio. Today, their data lake stores petabytes of data, supporting hundreds of pipelines and tens of thousands of tasks daily. It is essential for business units at T3Go, including Data Warehouse, Internet of Vehicles, Order Dispatching, Machine Learning, and self-service query analysis. In this blog, you will see how we slashed data ingestion time by half using Hudi and Alluxio. Furthermore, data analysts using Presto, Hudi, and Alluxio saw the queries speed up by 10 times. We built our data lake based on data orchestration for multiple stages of our data pipeline, including ingestion and analytics. T3Go data lake Overview Prior to the data lake, different business units within T3Go managed their own data processing solutions, utilizing different storage systems, ETL tools, and data processing frameworks. Data for each became siloed from every other unit, significantly increasing cost and complexity. Due to the rapid business expansion of T3Go, this inefficiency became our engineering bottleneck. We moved to a unified data lake solution based on Alibaba OSS, an object store similar to AWS S3, to provide a centralized location to store structured and unstructured data, following the design principles of ; all the applications access OSS storage as the source of truth, as opposed to different data silos. This architecture allows us to store the data as-is without having to first structure the data and run different types of analytics to guide better decisions, building dashboards and visualizations from big data processing, real-time analytics, and machine learning. Multi-cluster Shared-data Architecture Efficient Near Real-time Analytics Using Hudi Our business in smart travel drives the need to process and analyze data in a near real-time manner. With a traditional data warehouse, we faced the following challenges: High overhead when updating due to long-tail latency. High cost of order analysis due to the long window of a business session Reduced query accuracy due to late or ad-hoc updates Unreliability in data ingestion pipeline. Data lost in the distributed data pipeline cannot be reconciled. High latency to access data storage. As a result, we adopted Apache Hudi on top of OSS to address these issues. The following diagram outlines the architecture: Enable near real-time data ingestion and analysis With Hudi, our data lake supports multiple data sources, including Kafka, MySQL binlog, GIS, and other business logs in near real-time. As a result, more than 60% of the company’s data is stored in the data lake, and this proportion continues to increase. We are also able to speed up the data ingestion time down to a few minutes by introducing Apache Hudi into the data pipeline. Combined with big data interactive query and analysis frameworks, such as Presto and SparkSQL, real-time data analysis and insights are achieved. Enable Incremental processing pipeline With the help of Hudi, it is possible to provide incremental changes to the downstream derived table when the upstream table updates frequently. Even with a large number of interdependent tables, we can quickly run partial data updates. This also effectively avoids updating the full partitions of cold tables in the traditional Hive data warehouse. Accessing Data using Hudi as a unified format Traditional data warehouses often deploy Hadoop to store data and provide batch analysis. Kafka is used separately to distribute Hadoop data to other data processing frameworks, resulting in duplicated data. Hudi helps effectively solve this problem; we always use Spark pipelines to insert new updates into the Hudi tables, then incrementally read the update of Hudi tables. In other words, Hudi tables are used as the unified storage format to access data. Efficient Data Caching Using Alluxio In the early version of our data lake without Alluxio, data received from Kafka in real-time is processed by Spark and then written to OSS data lake using Hudi DeltaStreamer tasks. With this architecture, Spark often suffered high network latency when writing to OSS directly. Since all data is in OSS storage, OLAP queries on Hudi data may also be slow due to a lack of data locality. To address the latency issue, we deployed Alluxio as a data orchestration layer, co-located with computing engines such as Spark and Presto, and used Alluxio to accelerate read and write on the data lake as shown in the following diagram: Data in formats such as Hudi, Parquet, ORC, and JSON are stored mostly on OSS, consisting of 95% of the data. Computing engines such as Flink, Spark, Kylin, and Presto are deployed in isolated clusters, respectively. When each engine accesses OSS, Alluxio acts as a virtual distributed storage system to accelerate data, being co-located with each of the computing clusters. Specifically, here are a few applications leveraging Alluxio in the T3Go data lake. Data lake ingestion We mount the corresponding OSS path to the Alluxio file system and set Hudi’s parameter value to use the scheme in place of scheme. Spark pipelines with Hudi continuously ingest data to Alluxio. After data is written to Alluxio, it is asynchronously persisted from the Alluxio cache to the remote OSS every minute. These modifications allow Spark to write to a local Alluxio node instead of writing to remote OSS, significantly reducing the time for the data to be available in the data lake after ingestion. target-base-path alluxio:// oss:// Data analysis on the lake We use Presto as an ad-hoc query engine to analyze the Hudi tables in the lake, co-locating Alluxio workers on each Presto worker node. When Presto and Alluxio services are co-located and running, Alluxio caches the input data locally in the Presto worker, which greatly benefits Presto for subsequent retrievals. On a cache hit, Presto can read from the local Alluxio worker storage at memory speed without any additional data transfer over the network. Concurrent accesses across multiple storage systems In order to ensure the accuracy of training samples, our machine learning team often synchronizes desensitized data in production to an offline machine learning environment. During synchronization, the data flows across multiple file systems, from production OSS to an offline HDFS followed by another offline Machine Learning HDFS. This data migration process is not only inefficient but also error-prone for modelers because multiple different storages with varying configurations are involved. Alluxio helps in this specific scenario by mounting the destination storage systems under the same filesystem to be accessed by their corresponding logical paths in Alluxio namespace. By decoupling the physical storage, allows applications with different APIs to access and transfer data seamlessly. This data access layout also improves performance. Microbenchmark Overall, we observed the following improvements with Alluxio: It supports a hierarchical and transparent caching mechanism. It supports cache promote mode when reading. It supports asynchronous writing mode. It supports LRU recycling strategy. It has pin and TTL features After comparison and verification, we choose to use Spark SQL as the query engine. Our performance testing queries the Hudi table, comparing Alluxio + OSS together against OSS directly as well as HDFS. In the stress test shown above, after the data volume is greater than a certain magnitude (2400W), the query speed using Alluxio+OSS surpasses the HDFS query speed of the hybrid deployment. After the data volume is greater than 1E, the query speed starts to double. After reaching 6E data, it is up to 12 times higher than querying native OSS and 8 times higher than querying native HDFS. The improvement depends on the machine configuration. Based on our performance benchmarking, we found that the performance can be improved by over 10 times with the help of Alluxio. Furthermore, the larger the data scale, the more prominent the performance improvement. Next Step As T3Go’s data lake ecosystem expands, we will continue facing the critical scenario of compute and storage segregation. With T3Go’s growing data processing needs, our team plans to deploy Alluxio on a larger scale to accelerate our data lake storage. In addition to the deployment of Alluxio on the data lake computing engine, which currently is mainly SparkSQL, we plan to add a layer of Alluxio to the OLAP cluster using Apache Kylin and an ad_hoc cluster using Presto. The goal is to have Alluxio cover all computing scenarios, with Alluxio interconnected between each scene to improve the read and write efficiency of the data lake and the surrounding lake ecology. Conclusion As mentioned earlier, Hudi and Alluxio cover all scenarios of Hudi’s near real-time ingestion, near real-time analysis, incremental processing, and data distribution on DFS, among many others, and play the role of a powerful accelerator on data ingestion and data analysis on the lake. With Hudi and Alluxio together, our R&D engineers shortened the time for data ingestion into the lake by up to a factor of 2. Data analysts using Presto, Hudi, and Alluxio in conjunction to query data on the lake saw their queries speed up by 10 times faster. Furthermore, the larger the data scale, the more prominent the performance improvement. Alluxio is an important part of T3Go’s plan to become a leading enterprise data lake in China. We look forward to seeing further integration with Alluxio in T3Go’s data lake ecosystem.

Building High-Performance Data Lake Using Apache Hudi and Alluxio 

Every application is built from the same type of components - buttons, inputs, dropdowns, etc., and modal windows are in this list. The reason is simple - modal windows allow us to communicate with a user and do not overwhelm a UI with additional information until it becomes needed. In this article, I will explore one of the approaches to managing modal windows in apps. Angular We use the primeng library for modals in the article's examples. Your case can be different. The general idea will be the same despite the libraries, but API can be different; be aware of it. Modal windows are just regular components - they have markup, style, and selector. But in contrast to regular components, they usually don't have to be shown on the page immediately. We need them only for the result of users' actions - confirm, discard, fill a form, etc. That's why we won't include them in our build - we will download them dynamically only if we require them. Let's see an example: public openModal(): void {
    const modalComponent = async (): Promise<any> => import('./path-to-your-modal');
    this.modalService.open(modalComponent());
} Here, we have the dynamic import approach. It means that our component will be loaded when we invoke the method. It's quite useful because we don't load the modal immediately in our chunk - only when the user needs it. This approach can help you to slightly reduce the bundle size, especially if your modals are full of logic and components. openModal In general, this is the working solution. But we can go further. Let's make a service that will handle the dynamic imports and return an instance of the modal: import { Injectable } from '@angular/core';
import { DialogService } from 'primeng/dynamicdialog';

@Injectable({ providedIn: 'root' })
export class CustomModalService {
    public modalWindowComponents: { [key: string]: () => Promise<any> } = {
        ConfirmModalComponent: () =>
            import('your-path/to-component').then((component) => component.ConfirmModalComponent),
    };

constructor(private modalService: DialogService) {}

    public async openDynamicDialogModal({
        componentName,
        data,
    }: {
        componentName: string;
        data?: { [key: string]: any };
    }): Promise<any> {
        const modalComponent = this.modalService.open(await this.modalWindowComponents[componentName](), { ...data });

        modalComponent.onClose.subscribe(() => {
            // some logic
        });
        return modalComponent;
    }
} Here, we can see several important things: We use an object to store modal windows imports where the key is the component name, and the value is an import function that returns Promise; Our method is taking the params object with the component's name (to take the import function from the object) and some additional data, which contains whatever you want for a component or its appearing configuration. Also, we have a subscription for close events. It is more convenient to handle window closing in one place; We return a Promise with the imported component. It allows us to handle the logic for each modal. Here is an example of the usage of this service: public async handleOpeningModalWindow(): Promise<void> {
    const dialogRef = await this.primengModalService.openDynamicDialogModal({
        componentName: 'DeleteStreamModalComponent',
        data: {
           // your params
        },
    });

    dialogRef.onClose
        .pipe(
            // your pipes
        )
        .subscribe(() => {
            // your logic
        });
} Conclusion By adopting the approach outlined above, Angular developers can achieve a leaner, more manageable modal window system. This strategy not only reduces bundle size but also offers a unified method for modal control and customization. Embracing dynamic imports and service-based architecture empowers developers to optimize user experiences while maintaining code scalability and efficiency.

How to Manage Modal Windows in Angular Apps

How to Build and Deploy a Real-time Traffic Monitoring App Using Quix's Computer Vision Template

কুইক্সের কম্পিউটার ভিশন টেমপ্লেট ব্যবহার করে কীভাবে একটি রিয়েল-টাইম ট্র্যাফিক মনিটরিং অ্যাপ তৈরি এবং স্থাপন করবেন

So erstellen und implementieren Sie eine Echtzeit-Verkehrsüberwachungs-App mithilfe der Computer Vision-Vorlage von Quix

Cómo crear e implementar una aplicación de monitoreo de tráfico en tiempo real utilizando la plantilla de visión por computadora de Quix

Comment créer et déployer une application de surveillance du trafic en temps réel à l'aide du modèle de vision par ordinateur de Quix

क्विक्स के कंप्यूटर विज़न टेम्पलेट का उपयोग करके रीयल-टाइम ट्रैफ़िक मॉनिटरिंग ऐप कैसे बनाएं और तैनात करें

Quix の Computer Vision テンプレートを使用してリアルタイム トラフィック監視アプリを構築および展開する方法

Quix의 컴퓨터 비전 템플릿을 사용하여 실시간 교통 모니터링 앱을 구축하고 배포하는 방법

Como construir e implantar um aplicativo de monitoramento de tráfego em tempo real usando o modelo de visão computacional do Quix

Как создать и развернуть приложение для мониторинга трафика в реальном времени с использованием шаблона компьютерного зрения Quix

Quix'in Görüntü İşleme Şablonunu Kullanarak Gerçek Zamanlı Trafik İzleme Uygulaması Nasıl Oluşturulur ve Dağıtılır

Cách xây dựng và triển khai ứng dụng giám sát lưu lượng truy cập thời gian thực bằng mẫu thị giác máy tính của Quix

如何使用 Quix 的计算机视觉模板构建和部署实时流量监控应用程序

Flink-Doris-Connector 1.4.0 allows users to ingest a whole database ( or ) that contains thousands of tables into , a real-time analytic database, . MySQL Oracle Apache Doris in one step With built-in Flink CDC, the Connector can directly synchronize the table schema and data from the upstream source to Apache Doris, which means users no long have to write a DataStream program or pre-create mapping tables in Doris. When a Flink job starts, the Connector automatically checks for data equivalence between the source database and Apache Doris. If the data source contains tables that do not exist in Doris, the Connector will automatically create those same tables in Doris, and utilize the side outputs of Flink to facilitate the ingestion of multiple tables at once; if there is a schema change in the source, it will automatically obtain the DDL statement and make the same schema change in Doris. Quick Start : For MySQL Download the JAR file: https://github.com/apache/doris-flink-connector/releases/tag/1.4.0 Maven: <dependency>
  <groupId>org.apache.doris</groupId>
  <artifactId>flink-doris-connector-1.15</artifactId>
  
  
  <version>1.4.0</version>
</dependency> : For Oracle Download JAR file: , , Flink 1.15 Flink 1.16 Flink 1.17 How to Use It For example, to ingest a whole MySQL into Doris (the MySQL table names start with or ), simply execute the following command (no need to create the tables in Doris in advance): database mysql_db tbl test <FLINK_HOME>/bin/flink run \
    -Dexecution.checkpointing.interval=10s \
    -Dparallelism.default=1 \
    -c org.apache.doris.flink.tools.cdc.CdcTools \
    lib/flink-doris-connector-1.16-1.4.0.jar \
    mysql-sync-database \
    --database test_db \
    --mysql-conf hostname=127.0.0.1 \
    --mysql-conf username=root \
    --mysql-conf password=123456 \
    --mysql-conf database-name=mysql_db \
    --including-tables "tbl|test.*" \
    --sink-conf fenodes=127.0.0.1:8030 \
    --sink-conf username=root \
    --sink-conf password=123456 \
    --sink-conf jdbc-url=jdbc:mysql://127.0.0.1:9030 \
    --sink-conf sink.label-prefix=label1 \
    --table-conf replication_num=1 To ingest an Oracle database: please refer to the . sample code How It Performs When it comes to synchronizing a whole database (containing hundreds or even thousands of tables, active or inactive), most users want it to be done within seconds. So we tested the Connector to see if it came up to scratch: 1000 MySQL tables, each having 100 fields. All tables were active (which meant they were continuously updated and each data writing involved over a hundred rows) Flink job checkpoint: 10s Under pressure test, the system showed high stability, with key metrics as follows: According to feedback from early adopters, the Connector has also delivered high performance and system stability in 10,000-table database synchronization in their production environment. This proves that the combination of Apache Doris and Flink CDC is capable of large-scale data synchronization with high efficiency and reliability. How It Benefits Data Engineers Engineers no longer have to worry about table creation or table schema maintenance, saving them days of tedious and error-prone work. Previously in Flink CDC, you need to create a Flink job for each table and build a log parsing link at the source end, but now with whole-database ingestion, resource consumption in the source database is largely reduced. It is also a unified solution for incremental updates and full updates. Other Features Joining dimension table and fact table The common practice is to put dimension tables in Doris and run join queries via the real-time stream of Flink. Based on the Async I/O of Flink, Flink-Doris-Connector 1.4.0 implements asynchronous Lookup Join, so the Flink real-time stream won't be blocked due to queries. Also, the Connector allows you to combine multiple queries into one big query, and send it to Doris at once for processing. This improves the efficiency and throughput of such join queries. Thrift SDK We introduced Thrift-Service SDK into the Connector so users no longer have to use Thrift plug-ins or configure a Thrift environment in the compilation. This makes the compilation process much simpler. On-Demand Stream Load During data synchronization, when there is no new data ingestion, no Stream Load requests will be issued. This avoids unnecessary consumption of cluster resources. Polling of Backend Nodes For data ingestion, Doris calls a frontend node to obtain a list of the backend nodes and randomly chooses one to launch an ingestion request. That backend node will be the Coordinator. Flink-Doris-Connector 1.4.0 allows users to enable the polling mechanism, which is to have a different backend node to be the Coordinator at each Flink checkpoint to avoid too much pressure on a single backend node for a long time. Support for More Data Types In addition to the common data types, Flink-Doris-Connector 1.4.0 supports DecimalV3/DateV2/DateTimev2/Array/JSON in Doris. Example Usage Read from Apache Doris: You can read data from Doris via DataStream or FlinkSQL (bounded stream). Predicate pushdown is supported. CREATE TABLE flink_doris_source (
    name STRING,
    age INT,
    score DECIMAL(5,2)
    ) 
    WITH (
      'connector' = 'doris',
      'fenodes' = '127.0.0.1:8030',
      'table.identifier' = 'database.table',
      'username' = 'root',
      'password' = 'password',
      'doris.filter.query' = 'age=18'
);

SELECT * FROM flink_doris_source; : Join dimension table and fact table CREATE TABLE fact_table (
  `id` BIGINT,
  `name` STRING,
  `city` STRING,
  `process_time` as proctime()
) WITH (
  'connector' = 'kafka',
  ...
);

create table dim_city(
  `city` STRING,
  `level` INT ,
  `province` STRING,
  `country` STRING
) WITH (
  'connector' = 'doris',
  'fenodes' = '127.0.0.1:8030',
  'jdbc-url' = 'jdbc:mysql://127.0.0.1:9030',
  'lookup.jdbc.async' = 'true',
  'table.identifier' = 'dim.dim_city',
  'username' = 'root',
  'password' = ''
);

SELECT a.id, a.name, a.city, c.province, c.country,c.level 
FROM fact_table a
LEFT JOIN dim_city FOR SYSTEM_TIME AS OF a.process_time AS c
ON a.city = c.city : Write to Apache Doris CREATE TABLE doris_sink (
    name STRING,
    age INT,
    score DECIMAL(5,2)
    ) 
    WITH (
      'connector' = 'doris',
      'fenodes' = '127.0.0.1:8030',
      'table.identifier' = 'database.table',
      'username' = 'root',
      'password' = '',
      'sink.label-prefix' = 'doris_label',
      //json write-in
      'sink.properties.format' = 'json',
      'sink.properties.read_json_by_line' = 'true'
);

Auto-Synchronization of an Entire MySQL Database for Data Analysis

This is a part of the digital transformation of a real estate giant. For the sake of confidentiality, I'm not going to reveal any business data, but you'll get a detailed view of our data warehouse and our optimization strategies. Now let's get started. Architecture Logically, our data architecture can be divided into four parts. : This is supported by Flink CDC, DataX, and the Multi-Catalog feature of Apache Doris. Data integration : We use Apache Dolphinscheduler for script lifecycle management, privileges in multi-tenancy management, and data quality monitoring. Data management : We use Grafana, Prometheus, and Loki to monitor component resources and logs. Alerting : This is where BI tools step in for user interaction, such as data queries and analysis. Data services 1. Tables We create our dimension tables and fact tables centering each operating entity in business, including customers, houses, etc. If there are a series of activities involving the same operating entity, they should be recorded by one field. (This is a lesson learned from our previous chaotic data management system.) 2. Layers Our data warehouse is divided into five conceptual layers. We use Apache Doris and Apache DolphinScheduler to schedule the DAG scripts between these layers. Every day, the layers go through an overall update besides incremental updates in case of changes in historical status fields or incomplete data synchronization of ODS tables. 3. Incremental Update Strategies (1) Set where >= "activity time -1 day or -1 hour" instead of where >= "activity time The reason for doing so is to prevent data drift caused by the time gap of scheduling scripts. Let's say, with the execution interval set to 10 min, suppose that the script is executed at 23:58:00 and a new piece of data arrives at 23:59:00, if we set , that piece of data of the day will be missed. where >= "activity time (2) Fetch the ID of the largest primary key of the table before every script execution, store the ID in the auxiliary table, and set where >= "ID in auxiliary table" This is to avoid data duplication. Data duplication might happen if you use the Unique Key model of Apache Doris and designate a set of primary keys, because if there are any changes in the primary keys in the source table, the changes will be recorded and the relevant data will be loaded. This method can fix that, but it is only applicable when the source tables have auto-increment primary keys. (3) Partition the tables As for time-based auto-increment data such as log tables, there might be less changes in historical data and status, but the data volume is large, so there could be huge computing pressure on overall updates and snapshot creation. Hence, it is better to partition such tables, so for each incremental update, we only need to replace one partition. (You might need to watch out for data drift, too.) 4. Overall Update Strategies (1) Truncate Table Clear out the table and then ingest all data from the source table into it. This is applicable for small tables and scenarios with no user activity in wee hours. (2) ALTER TABLE tbl1 REPLACE WITH TABLE tbl2 This is an atomic operation and it is advisable for large tables. Every time before executing a script, we create a temporary table with the same schema, load all data into it, and replace the original table with it. Application : every minute ETL job : 8 nodes, 2 frontends, 8 backends, hybrid deployment Configuration for first-time deployment : 32C * 60GB * 2TB SSD Node configuration This is our configuration for TBs of legacy data and GBs of incremental data. You can use it as a reference and scale your cluster on this basis. Deployment of Apache Doris is simple. You don't need other components. To integrate offline data and log data, we use DataX, which supports CSV format and readers of many relational databases, and Apache Doris provides a DataX-Doris-Writer. We use Flink CDC to synchronize data from source tables. Then we aggregate the real-time metrics utilizing the Materialized View or the Aggregate Model of Apache Doris. Since we only have to process part of the metrics in a real-time manner and we don't want to generate too many database connections, we use one Flink job to maintain multiple CDC source tables. This is realized by the multi-source merging and full database sync features of Dinky, or you can implement a Flink DataStream multi-source merging task yourself. It is noteworthy that Flink CDC and Apache Doris support Schema Change. EXECUTE CDCSOURCE demo_doris WITH (
  'connector' = 'mysql-cdc',
  'hostname' = '127.0.0.1',
  'port' = '3306',
  'username' = 'root',
  'password' = '123456',
  'checkpoint' = '10000',
  'scan.startup.mode' = 'initial',
  'parallelism' = '1',
  'table-name' = 'ods.ods_*,ods.ods_*',
  'sink.connector' = 'doris',
  'sink.fenodes' = '127.0.0.1:8030',
  'sink.username' = 'root',
  'sink.password' = '123456',
  'sink.doris.batch.size' = '1000',
  'sink.sink.max-retries' = '1',
  'sink.sink.batch.interval' = '60000',
  'sink.sink.db' = 'test',
  'sink.sink.properties.format' ='json',
  'sink.sink.properties.read_json_by_line' ='true',
  'sink.table.identifier' = '${schemaName}.${tableName}',
  'sink.sink.label-prefix' = '${schemaName}_${tableName}_1'
); We use SQL scripts or "Shell + SQL" scripts, and we perform script lifecycle management. At the ODS layer, we write a general DataX job file and pass parameters for each source table ingestion, instead of writing a DataX job for each source table. In this way, we make things much easier to maintain. We manage the ETL scripts of Apache Doris on DolphinScheduler, where we also conduct version control. In case of any errors in the production environment, we can always rollback. After ingesting data with ETL scripts, we create a page in our reporting tool. We assign different privileges to different accounts using SQL, including the privilege of modifying rows, fields, and global dictionary. Apache Doris supports privilege control over accounts, which works the same as that in MySQL. We also use Apache Doris data backup for disaster recovery, Apache Doris audit logs to monitor SQL execution efficiency, Grafana+Loki for cluster metric alerts, and Supervisor to monitor the daemon processes of node components. Optimization 1. Data Ingestion We use DataX to Stream Load offline data. It allows us to adjust the size of each batch. The Stream Load method returns results synchronously, which meets the needs of our architecture. If we execute asynchronous data import using DolphinScheduler, the system might assume that the script has been executed, and that can cause a messup. If you use a different method, we recommend that you execute in the shell script, and check the regex filtering status to see if the ingestion succeeds. show load 2. Data Model We adopt the Unique Key model of Apache Doris for most of our tables. The Unique Key model ensures idempotence of data scripts and effectively avoids upstream data duplication. 3. Reading External Data We use the Multi-Catalog feature of Apache Doris to connect to external data sources. It allows us to create mappings of external data at the Catalog level. 4. Query Optimization We suggest that you put the most frequently used fields of non-character types (such as int and where clauses) in the first 36 bytes, so you can filter these fields within milliseconds in point queries. 5. Data Dictionary For us, it is important to create a data dictionary because it largely reduces personnel communication costs, which can be a headache when you have a big team. We use the in Apache Doris to generate a data dictionary. With it, we can quickly grasp the whole picture of the tables and fields and thus increase development efficiency. information_schema Performance : Within minutes Offline data ingestion time : For tables containing over 100 million rows, Apache Doris responds to ad-hoc queries within one second, and complicated queries in five seconds. Query latency : It only takes up a small number of servers to build this data warehouse. The 70% compression ratio of Apache Doris saves us lots of storage resources. Resource consumption Experience and Conclusion Actually, before we evolved into our current data architecture, we tried Hive, Spark and Hadoop to build an offline data warehouse. It turned out that Hadoop was overkill for a traditional company like us since we didn't have too much data to process. It is important to find the component that suits you most. On the other hand, to smoothen our  big data transition, we need to make our data platform as simple as possible in terms of usage and maintenance. That's why we landed on Apache Doris. It is compatible with MySQL protocol and provides a rich collection of functions so we don't have to develop our own UDFs. Also, it is composed of only two types of processes: frontends and backends, so it is easy to scale and track.

Building a Data Warehouse for Traditional Industry: Hadoop is Overkill

In Vue, data is typically passed from parent components to their children in a uni-directional fashion. This is transferred with , which are the properties or attributes we give to . props components For example, if we call a component , which has a prop called , that property would become available within the component itself, letting us do what we want to do with it. That way, the data is passed down to the child component when we declare it in the parent component or page: PageOne name name PageOne In most scenarios, props allow us to do everything we need to do with data. - from a child component to its parent. For this, we use , which lets us send data upwards, and then trigger an event in the parent component should an event be fired. Sometimes, however, we need to emit data upwards $emit $emit How works in Vue? $emit There are three ways to fire in Vue, depending on if you're using the Options API, Composition API, or inlining your events. $emit $emit within the Options API. this.$emit if used in your HTML template. $emit and if used in the Composition API. defineEmits emit . Let's say we have a counter component, which looks like this: Let's take a look at how this works, through a silly example <template>
    <button @click="$emit('counterEvent')">Click Me</button>
</template> This component is stored in a file called . Our component can't be changed as it's used in other places, but it does have an event fired any time it is clicked. This is perfect since we can use this in our parent component. Counter.vue $emit So what if we want to add this component somewhere - for example, in our file - and use it to display the value of our counter. Let's try doing that now: App.vue <template>
    <h1>{{ counter }}</h1>
  	<Counter @counter-event="incrCounter"/>
</template>

<script>
import Counter from './Counter.vue'
export default {
    // Add our components
  	components: {
      Counter
    },
    // Store our data
    data() {
        return {
            counter: 0
        }
    },
    methods: {
        incrCounter: function() {
            this.counter += 1;
        }
    }
}
</script> Let's break this down - first of all, we include our . Since there is an event called , we can attach that to our HTML. Whenever fires, it'll fire the , and thus the function within that property. anytime fires. Counter $emit counterEvent Counter $emit counterEvent Here, we run incrCounter counterEvent By doing that, we can also increase our data by 1, since that is what does. As such, we've emitted the click event upwards to our parent component. counter incrCounter Why did we use Kebab Case? You may notice that when we defined our event, we used camel case ( ), but when tracking the event, we used kebab case ( ). $emit counterEvent counter-event In , it is fine to use and interchangably since Vue 3 automatically converts to . In , this functionality does not exist, so just stick with for both. Vue 3 counterEvent counter-event counterEvent counter-event Vue 2 counter-event Passing Data with $emit Let's say instead, we want our components to define how much the should increase. If we want to do that, we can pass a second argument to the function, which is the value: counterEvent $emit <template>
    <button @click="$emit('counterEvent', 2)">Click Me</button>
</template> , we are passing the value to our . Let's go back to our file. To leverage this value in , we need to write it as a function. Below, is the value: Here 2 counterEvent App.vue counterEvent n <template>
    <h1>{{ counter }}</h1>
  	<Counter @counter-event="(n) => incrCounter(n)"/>
</template>

<script>
import Counter from './Counter.vue'
export default {
    // Add our components
  	components: {
      Counter
    },
    // Store our data
    data() {
        return {
            counter: 0
        }
    },
    methods: {
        incrCounter: function(value) {
            this.counter += value;
        }
    }
}
</script> Now, our counter will increase by the value put in the child component, allowing us to pass data to our parent component as well. As you'd expect, this is not limited to just numbers but can include any data structure - including objects and strings. Using with the Options API $emit We've shown a quite simple example, but we could also have written our child component using a function instead. Here is an example with the , using : Counter.vue Options API this.$emit <template>
    <button @click="emitFunction">Click Me</button>
</template>

<script>
export default {
  emits: [ 'counterEvent' ],
 	methods: {
		emitFunction: function() {
		    this.$emit('counterEvent', 2)
        }
    }
}
</script> This may prove to be a slightly cleaner way to use , especially if you want to do other things along with using whenever a button is clicked. $emit $emit Adding your emit events to your prototype You may note that we also defined our emit event in on the prototype. This is good practice for two reasons: emits by showing which emit events are possible in this component. It lets you self-document the code , since Vue will throw an error if an emit event is used but not found in the array. It helps you keep track of deprecated emits emits Using with the Composition API $emit We can use with the Composition API - the only difference is we have to use instead. $emit defineEmits <template>
    <button @click="emitFunction">Click Me</button>
</template>

<script setup>
import { defineEmits } from 'vue'

const emit = defineEmits(['counterEvent']);
const emitFunction = function() {
    emit('counterEvent', 2)
}
</script> is used to define a full list of all permissible events. Here, we only have one, . If you had more than one, you could define them as so: defineEmits emit counterEvent const emit = defineEmits(['counterEvent', 'anotherEvent', 'finalEvent']); If you use an emit event not listed in , Vue will throw a warning, similar to using on the Options API. Otherwise, you can then use the function to emit as usual, without needing to use the at all. defineEmits emits emit() Options API Final Thoughts and Best Practices Emit is a powerful tool for sending data back up to the parent when we have to. This means datastreams can be two-way in Vue. When defining code, the two main best practices are: emit Always define your emit events in either or , which will help you keep your code clean and well documented. emits defineEmits Normal convention in Vue 3 is to use kebab case ( ) for HTML, and camel case ( ) in script. As such, it is best to follow this convention here too. this-is-kebab-case thisIsCamelCase I hope you've enjoyed this guide on how works. Stay tuned for more . $emit Vue Content Also Published Here

Learn How to Emit Custom Events in Vue With $emit

Over the past decade, 3D sensors have emerged to become one of the most versatile and ubiquitous types of sensor used in robotics. In many robotic applications, 3D sensing has become the de facto choice for tasks such as near-field object detection and collision avoidance, surface and object inspection, and map creation. There are a number of 3D sensing modalities available. In this roundup, and we’ll be covering the three most common that we work with at Tangram Vision: Stereo CMOS (both active and passive), structured light, and time of flight. We won’t be covering LiDAR, despite the fact that LiDAR data is also three dimensional. LiDAR users, fret not…we’ll be covering LiDAR as its own separate class of sensors in a separate post in the future! 1. Stereo 3D Sensors These ubiquitous sensors bring a combination of good overall performance coupled with low cost. Stereo 3D sensors come in two basic types: passive and active. Structure Core Active Stereo 3D Sensor. Image attribution: , Inc. Occipital Passive stereo 3D sensors are the least expensive of all 3D sensors, as they use off-the-shelf components that are inexpensive to source and manufacture. In addition, they are available in a wide range of baselines, giving users a choice of sensing ranges that are best adapted to their use case. However, most rely on visible light to operate effectively, which means they struggle to perform in low-light or unlit conditions. Active stereo 3D sensors add an infrared (IR) pattern projector which both increases the fidelity of 3D data captured, as well as adding more reliable operation in lower light conditions. However, the IR projectors on these sensors are limited in range, which relegates these sensors to near- and mid-range applications, regardless of baseline. The addition of an IR pattern projector effectively turns these sensors into hybrid stereo/structured light sensors (more on structured light 3D sensors in the next paragraph!). Examples of stereo 3D sensors: — Passive stereo 3D sensors available in large and small form factors, with wide and narrow baselines. The Zed Mini and Zed 2 feature integrated IMUs for kinetic data streams. Stereolabs Zed - Industrial-grade stereo 3D sensors with ruggedized construction and Gigabit Ethernet connections. Ensenso — Active stereo 3D sensor with bespoke IR projector and built-in IMU for kinetic data. Available as an embeddable module or as an encased standalone sensor. Occipital Structure Core — The D series (D415, D435, D435i, D455) are a diverse range of low cost active stereo 3D sensors. The D435i adds a built-in IMU for kinetic data, while the newest D455 adds extended range that is suitable for mid-range robotic navigation. Intel RealSense 2. Structured Light Sensors Structured light is the most ubiquitous 3D sensing modality used in robotics, thanks to the popularity of the original PrimeSense 3D sensors, which included the first generation Microsoft Kinect. Microsoft Kinect for Xbox 360. By Evan Amos — Own work, Public Domain, https://commons.wikimedia.org/w/index.php?curid=16059165 Structured light 3D sensors combine low cost with high-fidelity 3D data capture, as well as performance in a wide set of lighting conditions, with one notable exception: direct sunlight, or bright indirect sunlight. This is because the infrared light source used by structured light sensors gets overpowered by infrared light in the same bandwidth that is naturally present in sunlight. Like stereo 3D sensors, structured light sensors are available in an array of small to large baselines that are suited to tasks that occur at different ranges from the sensor. Examples: — A low-cost structured light 3D sensor that is often used as a drop-in substitute for the original Microsoft Kinect. Orbbec Astra (No longer available new) — The first standalone 3D sensor designed specifically to be used with mobile devices like Apple iPad and iPhone models. Compatible with for depth-powered application development. Now replaced by the active stereo sensor. Occipital Structure Sensor OpenNI Occipital Structure Sensor Mark II (No longer available new) — The most ubiquitous consumer-grade 3D sensor ever produced. Originally launched to support Xbox gaming, the first Kinect became popular with roboticists and hackers for adding robust 3D sensing at low cost to robots, drones and other platforms. Compatible with for depth-powered application development. Microsoft Kinect V1 OpenNI (No longer available new) — The Carmine series is near-identical in specification to the original Microsoft Kinect, and was available in short- and mid-range models. Unlike the original Microsoft Kinect, the Carmine does not include a microphone array. Compatible with for depth-powered application development. PrimeSense Carmine OpenNI — Ruggedized 3D sensors for industrial applications. The Photoneo phoXi line is available in multiple baselines. Photoneo — A range of structured light 3D sensors available in multiple baselines, with USB3 connection for the Zivid One series, and CAT 6A Ethernet connection for the Zivid Two. Zivid 3. Time of Flight Sensors Time of flight (ToF) sensors are effectively LiDAR sensors — aka, light radar. They send out packets of infrared light, and record the time it takes for the infrared light sent out to return. Despite using infrared light sources like structured light sensors, the manner in which ToF sensors transmit and receive light make them less susceptible to interference from bright and indirect sunlight. PMD Time of Flight (ToF) 3D sensor. ToFExpert, , via Wikimedia Commons CC BY-SA 3.0 ToF sensors often have wider range capabilities than structured light sensors, and can operate accurately for longer distance use cases. However, they aren’t able to return 3D images with the fidelity offered by structured light or stereo 3D sensors. Therefore, they are better suited for tasks like navigation assistance versus surface inspections. Examples: — The newest member of the Microsoft Kinect family of 3D sensors is a compact ToF module that adds a 12MP RGB video camera that is registered to the depth stream for high quality color imaging and textures. The Azure Kinect includes a built-in IMU for kinetic data. Microsoft Azure Kinect — Simple, compact ToF 3D sensors that can be purchased as embeddable modules, or as enclosed standalone sensors. PMD Pico — Increasingly hard-to-find. However, like the original Microsoft Kinect, PrimeSense Carmine and original Occipital Structure Sensor 3D sensors, the Xtion 2 is compatible with for depth-powered application development. ASUS Xtion 2 OpenNI Comparing Stereo Sensors to Structured Light to ToF Perception Sensors So why choose one modality over another? Well, we provided hints in the above descriptions for each modality, but let’s get more concrete on why you might choose one over another. At the highest level, the primary factors that determine 3D sensing selection are: Sensor Performance Impact on Host Environmental Limitations Let’s explore each of these factors in detail, and then show how each sensing modality compares. Sensor Performance The main parameters we’ll be exploring here are range, resolution and reliability. Range is perhaps the most important parameter, because different robot tasks are defined by the range in which they occur. For example, mapping environments at high speed for autonomous vehicles is typically left to rotating LiDAR units, as they have ranges that can exceed 100 meters, which allows them to see distant obstacles well before the autonomous vehicle is upon those obstacles. Those same LiDAR units, however, are poorly suited for close visual inspections, as these tasks benefit from sensors with sub-meter ranges that can support enhanced detail capture and accuracy. For stereo and structured light 3D sensors, range is a function of the baseline between the optical elements on the sensor itself. The rule of thumb is that the wider the baseline, the longer the range. Similarly, a narrower baseline creates a shorter range. So why not just spec a 3D sensor with the widest baseline possible? Well, for one, form factor limitations can preclude this approach. But, more importantly, wider baseline sensors’ sensing ranges start further away from the sensor. That means that a wide baseline sensor might ignore objects that are less than two meters from the sensor. A shorter baseline sensor might not have as long of a range, but it might be able to start sensing within .25 meters of the sensor. Therefore, baseline and the resultant range becomes a tradeoff of how near you need to sense versus how distant you need to sense. You get one or the other, but you can’t get both. Most 3D sensors have ranges that start at around 0.5–1m from the sensor, and extend to 3–6m at best. Keep in mind that at the furthest reaches of a sensor’s range, resolution and accuracy can start to diminish rapidly. ToF 3D sensors typically have much greater range than structured light or stereo 3D sensors, and can often receive reliable signals from objects that are up to 25m away. However, ToF sensors often suffer from excessive noise, and resolution can be much lower than similarly priced 3D modalities. Speaking of resolution, this is both a function of range, as well as the nature of the optics used by the sensor. Structured light sensors’ resolution is limited by the density of the pattern that they project (which gets less dense as distance increases), and the resolution of the IR camera that reads the pattern as it falls on objects and environments to be sensed. Stereo sensors’ resolution is limited by the native resolution of the cameras used for the sensor. For tasks such as navigation, high resolution isn’t strictly necessary, unless objects that might cause an obstruction are very fine (an extension cable, for instance, may be on the edge of detectability for some low-resolution 3D sensors). For tasks such as surface inspections and 3D modeling, high resolution is much more desirable. Because these tasks often occur at close range, many structured light and stereo sensors are well suited to this kind of task. ToF sensors are best suited for mid-distance navigation tasks, or skeletal tracking. These tasks don’t require high-resolution sensing, but benefit from the range and operational speed of ToF sensors. Lastly, let’s talk about reliability. For sensors, this comes in two forms: data acquisition reliability, and mechanical reliability. Data acquisition reliability comes down to the inherent limitations of each sensor modality. Stereo sensors, for instance, need sufficient ambient light in the visible spectrum to work properly. Conversely, infrared-based sensors like structured light and ToF suffer when exposed to too much ambient light, causing sensor washout and data integrity issues. These sensors also struggle with black surfaces (they absorb IR wavelengths) and shiny surfaces (they scatter projected IR light). Active stereo sensors with infrared pattern projectors offer a “best of both worlds” approach that broaden sensor reliability over a number of environments and lighting conditions. Mechanical reliability is often the result of compromises inherent to a specific modality, or result from manufacturer cost cutting. Many consumer-grade sensors use USB for data and power transfer; this is fine in a lab or test environment. However, in real-world use, USB can be flaky and error-prone, resulting in intermittent sensor failures and compromised data. Ethernet is much more stable, but is rarely found on consumer-grade sensors. It is much more likely to be found on much more expensive industrial-grade sensors, where it is a prerequisite for a purchase decision. Impact on Host The main parameters we’ll be exploring here are power consumption, compute use, heat dissipation, data transmission and form factor. All robotic hosts have finite amounts of compute and power available. Robots, drones and other vision-enabled devices are often resource balancing acts where multiple sensors, motors, actuators, and other components fight for scarce system resources. Different sensors have different impacts. Sensors with active IR components are much more power hungry than passive stereo 3d sensors, for instance. But passive stereo sensors with high dynamic range (HDR) cameras may use much more compute than a structured light sensor, as the images they capture are much more data rich. Thus, sensor decisions must be made holistically in context of the other components that require system resources (and that subject is complex enough to deserve its own blog post). Similarly, data transmission and storage can often be limited on robots and drones. Again, the example of using an HDR-equipped 3D sensor can strain even robust WiFi networks given the sheer size of the datastreams being captured and transmitted. Lastly, there are industrial design considerations that can limit what sensor options can be specified. These considerations include the size and weight of a sensor (particularly important for lightweight devices like drones), as well as a sensor’s ability to dissipate heat generated. The latter is of particular importance to IR-equipped sensors. The laser-based IR components in these sensors rely on operating within a specific temperature range to maintain the bandwidth at which they operate. Deviating by even a few degrees Fahrenheit outside of the operating temperature range can cause significant variations in sensor readings, corrupting the data generated enough to make it useless. Therefore, considerations must be made to ventilate or insulate these sensors, depending on the environments in which they will be used. Environmental Limitations The main parameters we’ll be exploring here are ambient lighting, and connectivity. We mentioned above that many of these sensors are sensitive to ambient lighting, but it bears repeating. Lighting challenges are one of the most common causes of 3D sensor failures for robots. A robot that worked exceptionally well in a laboratory environment with consistent lighting conditions will suddenly experience drastic and intermittent failures when placed in a real world context with changing, inconsistent lighting conditions. Therefore, it is important to know all the lighting permutations that may exist upon deployment, and account for those during sensor selection. Connectivity is also important. If sensor data is to be processed and used locally, then connectivity isn’t an issue. However, for robots that will be deployed in an environment with low bandwidth, then it is either necessary to compress or filter sensor data on the host before transmitting, or it is necessary to choose a low bandwidth sensor type (for instance, ToF). Again, it is important to test for packet size thresholds under real world bandwidth constraints, as laboratory environments may not sufficiently reveal deficiencies. Synthesizing the Data and Deciding We’ve put together a handy chart showing different , and the 3D sensors that work best for those scenarios. We’ve noted what key decision factors should be considered for each application, and added notes to further clarify why specific choices should be made. robot tasks to be done We hope that you’ve found this guide to 3D sensors useful! Our next sensor roundup will cover LiDAR sensors, so stay tuned. If you’re currently working on a vision-enabled product like a robot or a drone, please take a look at what we do at . We can help you get your product to market faster, and keep it functioning more reliably upon deployment, too. Tangram Vision Also published at https://medium.com/tangram-visions/sensors-101-3d-sensing-66e528f276c4

3 Common Types of 3D Sensors: Stereo, Structured Light, and ToF

The OWASP Top 10 is a standard awareness document for developers and web application security. It represents a broad consensus about the most critical security risks to web applications. In this article, I want to cover the second part of the TOP 10 vulnerabilities and how to protect against them using .NET. 6. Security Misconfiguration Security misconfiguration can happen at any level of an application stack, including the network services, platform, web server, application server, database, frameworks, custom code, and pre-installed virtual machines, containers, or storage. Such flaws frequently give attackers unauthorized access to some system data or functionality. What to do? (When using TLS) Redirect a request made over Http to https: app.UseHttpsRedirection(); //Startup.cs Send the anti-forgery token with every POST/PUT request: To automatically validate all requests other than GET, HEAD, OPTIONS and TRACE you need to add a global action filter with the attribute inside your Startup.cs AutoValidateAntiforgeryToken services.AddMvc(options =>
{
    options.Filters.Add( AutoValidateAntiforgeryTokenAttribute());
}); new If you need to also validate the token on GET, HEAD, OPTIONS or TRACE - requests you can add the attribute to the controller method (for MVC controllers) or parent class (for Razor pages): ValidateAntiforgeryToken [ ]
[ ] HttpGet ValidateAntiforgeryToken IActionResult ( ) public DoSomethingDangerous [ ]
[ ] : HttpGet ValidateAntiforgeryToken public class SafeModel PageModel In case you can't use a global action filter, add the attribute to your controller classes or razor page models: AutoValidateAntiforgeryToken [ ] AutoValidateAntiforgeryToken public class UserController [ ] : AutoValidateAntiforgeryToken public class SafeModel PageModel 7. Cross-Site Scripting (XSS) XSS is the second most prevalent issue in the OWASP Top 10, and is found in around two-thirds of all applications. The impact of XSS is moderate for reflected and DOM XSS, and severe for stored XSS, with remote code execution on the victim’s browser, such as stealing credentials, sessions, or delivering malware to the victim. What to do? DO NOT: Trust any data the user sends you, prefer white lists (always safe) over black lists DO NOT: Use the [AllowHTML] attribute or helper class @Html.Raw unless you really know that the content you are writing to the browser is safe and has been escaped properly. 8. Insecure Deserialization The impact of deserialization flaws cannot be overstated. These flaws can lead to remote code execution attacks, one of the most serious attacks possible. What to do? Validate User Input Malicious users are able to use objects like cookies to insert malicious information to change user roles. In some cases, hackers are able to elevate their privileges to administrator rights by using a pre-existing or cached password hash from a previous session. Prevent Deserialization of Domain Objects Run the Deserialization Code with Limited Access Permissions If a deserialized hostile object tries to initiate a system processes or access a resource within the server or the host's OS, it will be denied access and a permission flag will be raised so that a system administrator is made aware of any anomalous activity on the server. Don't allow the datastream to define the type of object that the stream will be deserialized to. You can prevent this by for example using the DataContractSerializer or XmlSerializer if at all possible. Where JSON.Net is being used make sure the TypeNameHandling is only set to None. TypeNameHandling = TypeNameHandling.None If JavaScriptSerializer is to be used then do not use it with a JavaScriptTypeResolver. If you must deserialise data streams that define their own type, then restrict the types that are allowed to be deserialized. One should be aware that this is still risky as many native .Net types potentially dangerous in themselves. 9. Using Components with Known Vulnerabilities Prevalence of this issue is very widespread. Component-heavy development patterns can lead to development teams not even understanding which components they use in their application or API, much less keeping them up to date. While some known vulnerabilities lead to only minor impacts, some of the largest breaches to date have relied on exploiting known vulnerabilities in components. What to do? Keep the .Net framework updated with the latest patches Keep your packages up to date, many will contain their own vulnerabilities. NuGet Run the against your application as part of your build process and act on any high level vulnerabilities. OWASP Dependency Checker 10. Insufficient Logging & Monitoring Attackers rely on the lack of monitoring and timely response to achieve their goals without being detected. Most successful attacks start with vulnerability probing. Allowing such probes to continue can raise the likelihood of successful exploit to nearly 100%. What to do? Ensure all login, access control failures and server-side input validation failures can be logged with sufficient user context to identify suspicious or malicious accounts. Establish effective monitoring and alerting so suspicious activities are detected and responded to in a timely fashion. Log all errors from the Startup.cs, so that anytime an error is thrown it will be logged. { (env.IsDevelopment())
    {
        _isDevelopment = ;
        app.UseDeveloperExceptionPage();
    } app.UseExceptionHandler(errorApp =>
    {
        errorApp.Run( context =>
        { errorFeature = context.Features.Get<IExceptionHandlerFeature>(); exception = errorFeature.Error;

            Log.Error(String.Format( , exception.StackTrace.ToString()));
        });
    });

    app.UseAuthentication();
    app.UseMvc();
} ( ) public void Configure IApplicationBuilder app, IHostingEnvironment env if true //Log all errors in the application async var var "Stacktrace of error: {0}" Monitor key performance indicators. In .NET a great option to add monitoring capabilities is . Application Insights Thanks for reading! - Part one https://hackernoon.com/owasp-top-10-net-protection-a-guide-part-1-w92r3wis P - Read more about OWAS https://owasp.org/www-project-top-ten/

DeltaStream

deltastream.io

-- employees

Since n.d.

COMPANY RANKING

DELTASTREAM

EVERGREEN INDEX #3422

DeltaStream WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

DeltaStream

deltastream.io

-- employees

Since n.d.

COMPANY RANKING

DELTASTREAM

EVERGREEN INDEX #3422

RELATED COMPANIESRANK

STORIES ABOUT DELTASTREAM

DeltaStream WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!