Amazon Aurora DSQL is a distributed SQL database launched by Amazon Web Services in December 2024. It is designed for building applications with unlimited scalability, high availability, and zero infrastructure management. Aurora DSQL offers high availability, serverless architecture, strong compatibility, fault tolerance, and advanced security. Because Aurora DSQL’s authentication mechanism integrates with IAM, accessing the database requires generating tokens based on IAM identities. Tokens are valid for only 15 minutes by default, which means many mainstream data synchronization tools currently do not support migrating data from other databases to Aurora DSQL. In response, the author developed a dedicated sink Connector for Aurora DSQL based on the Apache SeaTunnel data integration tool, enabling seamless migration of data from other databases to Aurora DSQL. developed a dedicated sink Connector for Aurora DSQL based on the Apache SeaTunnel data integration tool Introduction to SeaTunnel SeaTunnel is an easy-to-use, multi-modal, high-performance distributed data integration platform, focusing on data integration and synchronization, and aims to solve common problems in the data integration field. Key Features of SeaTunnel Rich and Extensible Connectors: SeaTunnel currently supports over 190 connectors and continues to grow. Mainstream databases like MySQL, Oracle, SQLServer, and PostgreSQL are already supported. Its plugin-based design allows users to develop and integrate custom connectors easily.Batch and Stream Integration: Connectors developed with the SeaTunnel Connector API are fully compatible with offline synchronization, real-time synchronization, full-load, and incremental sync scenarios, greatly simplifying task management.Distributed Snapshots: Supports distributed snapshot algorithms to ensure data consistency.Multi-Engine Support: SeaTunnel uses the SeaTunnel engine (Zeta) by default for data synchronization. It also supports Flink or Spark as execution engines, adapting to existing enterprise technology stacks, with support for multiple versions.JDBC Reuse and Multi-Table Log Parsing: SeaTunnel supports multi-table or full-database synchronization, reducing excessive JDBC connections. It also supports log parsing for multiple tables to handle CDC scenarios efficiently.High Throughput, Low Latency: Parallel read/write operations provide stable, high-throughput, and low-latency data synchronization.Comprehensive Real-Time Monitoring: Monitors every step of data synchronization, showing data volume, size, and QPS. Rich and Extensible Connectors: SeaTunnel currently supports over 190 connectors and continues to grow. Mainstream databases like MySQL, Oracle, SQLServer, and PostgreSQL are already supported. Its plugin-based design allows users to develop and integrate custom connectors easily. Rich and Extensible Connectors: Batch and Stream Integration: Connectors developed with the SeaTunnel Connector API are fully compatible with offline synchronization, real-time synchronization, full-load, and incremental sync scenarios, greatly simplifying task management. Batch and Stream Integration: Distributed Snapshots: Supports distributed snapshot algorithms to ensure data consistency. Distributed Snapshots: Multi-Engine Support: SeaTunnel uses the SeaTunnel engine (Zeta) by default for data synchronization. It also supports Flink or Spark as execution engines, adapting to existing enterprise technology stacks, with support for multiple versions. Multi-Engine Support: JDBC Reuse and Multi-Table Log Parsing: SeaTunnel supports multi-table or full-database synchronization, reducing excessive JDBC connections. It also supports log parsing for multiple tables to handle CDC scenarios efficiently. JDBC Reuse and Multi-Table Log Parsing: High Throughput, Low Latency: Parallel read/write operations provide stable, high-throughput, and low-latency data synchronization. High Throughput, Low Latency: Comprehensive Real-Time Monitoring: Monitors every step of data synchronization, showing data volume, size, and QPS. Comprehensive Real-Time Monitoring: SeaTunnel Workflow The workflow shows that users configure job information and select an execution engine. Source Connectors read source data in parallel and send it downstream to Transform or directly to Sink, which writes data to the destination. Building SeaTunnel from Source git clone https://github.com/apache/seatunnel.git
cd seatunnel
sh ./mvnw clean install -DskipTests -Dskip.spotless=true
cp seatunnel-dist/target/apache-seatunnel-${version}-bin.tar.gz /The-Path-You-Want-To-Copy
cd /The-Path-You-Want-To-Copy
tar -xzvf "apache-seatunnel-${version}-bin.tar.gz" git clone https://github.com/apache/seatunnel.git
cd seatunnel
sh ./mvnw clean install -DskipTests -Dskip.spotless=true
cp seatunnel-dist/target/apache-seatunnel-${version}-bin.tar.gz /The-Path-You-Want-To-Copy
cd /The-Path-You-Want-To-Copy
tar -xzvf "apache-seatunnel-${version}-bin.tar.gz" After building from source, all connector plugins and necessary dependencies (e.g., MySQL driver) are included in the binary package. Users can use connectors directly without separate installation. Example: Syncing MySQL Data to Aurora DSQL with SeaTunnel env {
  parallelism = 1
  job.mode = "STREAMING"
  checkpoint.interval = 6000
  checkpoint.timeout = 1200000
}
source {
  MySQL-CDC {
    username = "user name"
    password = "password"
    table-names = ["db.table1"]
    url = "jdbc:mysql://dbhost:3306/db?useSSL=false&allowPublicKeyRetrieval=true&serverTimezone=UTC&connectTimeout=120000&socketTimeout=120000&autoReconnect=true&failOverReadOnly=false&maxReconnects=10"
    table-names-config = [
      {
        table = "db.table1"
        primaryKeys = ["id"]
      }
    ]
  }
}
transform {

}
sink {
    Jdbc {
        url="jdbc:postgresql://<dsql_endpoint>:5432/postgres"
        dialect="dsql"
        driver = "org.postgresql.Driver"
        username = "admin"
        access_key_id = "ACCESSKEYIDEXAMPLE"
        secret_access_key = "SECRETACCESSKEYEXAMPLE"
        region = "us-east-1"
        database = "postgres"
        generate_sink_sql = true
        primary_keys = ["id"]
        max_retries="3"
        batch_size =1000
    }
} env {
  parallelism = 1
  job.mode = "STREAMING"
  checkpoint.interval = 6000
  checkpoint.timeout = 1200000
}
source {
  MySQL-CDC {
    username = "user name"
    password = "password"
    table-names = ["db.table1"]
    url = "jdbc:mysql://dbhost:3306/db?useSSL=false&allowPublicKeyRetrieval=true&serverTimezone=UTC&connectTimeout=120000&socketTimeout=120000&autoReconnect=true&failOverReadOnly=false&maxReconnects=10"
    table-names-config = [
      {
        table = "db.table1"
        primaryKeys = ["id"]
      }
    ]
  }
}
transform {

}
sink {
    Jdbc {
        url="jdbc:postgresql://<dsql_endpoint>:5432/postgres"
        dialect="dsql"
        driver = "org.postgresql.Driver"
        username = "admin"
        access_key_id = "ACCESSKEYIDEXAMPLE"
        secret_access_key = "SECRETACCESSKEYEXAMPLE"
        region = "us-east-1"
        database = "postgres"
        generate_sink_sql = true
        primary_keys = ["id"]
        max_retries="3"
        batch_size =1000
    }
} Running the Data Sync Job Save the above configuration as mysql-to-dsql.conf(replace example values with real parameters) in the config directory of apache-seatunnel-${version}, then run: mysql-to-dsql.conf config apache-seatunnel-${version} cd "apache-seatunnel-${version}"
./bin/seatunnel.sh --config ./config/mysql-to-dsql.conf -m local cd "apache-seatunnel-${version}"
./bin/seatunnel.sh --config ./config/mysql-to-dsql.conf -m local After execution, check the logs for task progress. Errors like connection timeouts or missing tables can be identified. Normally, data will be successfully written to Aurora DSQL. Summary Aurora DSQL is a secure, scalable, serverless distributed database. Its IAM-based authentication makes data synchronization challenging, especially for real-time scenarios. SeaTunnel is an excellent data integration and sync tool that supports multiple data sources and flexible custom sync needs (full-load or incremental). This article introduced a dedicated Sink Connector for Aurora DSQL to meet these requirements. References SeaTunnel Deployment: https://seatunnel.apache.org/zh-CN/docs/start-v2/locally/deploymentDeveloping New SeaTunnel Connectors: https://github.com/apache/seatunnel/blob/dev/seatunnel-connectors-v2/README.zh.mdGenerating Auth Tokens in Aurora DSQL: https://docs.aws.amazon.com/aurora-dsql/latest/userguide/SECTION_authentication-token.html SeaTunnel Deployment: https://seatunnel.apache.org/zh-CN/docs/start-v2/locally/deployment https://seatunnel.apache.org/zh-CN/docs/start-v2/locally/deployment Developing New SeaTunnel Connectors: https://github.com/apache/seatunnel/blob/dev/seatunnel-connectors-v2/README.zh.md https://github.com/apache/seatunnel/blob/dev/seatunnel-connectors-v2/README.zh.md Generating Auth Tokens in Aurora DSQL: https://docs.aws.amazon.com/aurora-dsql/latest/userguide/SECTION_authentication-token.html https://docs.aws.amazon.com/aurora-dsql/latest/userguide/SECTION_authentication-token.html AWS generative AI services are available in overseas regions; in China, AWS services are operated by Xiyun Data and Guanghuan Xinnet. Refer to regional websites for details. AWS generative AI services are available in overseas regions; in China, AWS services are operated by Xiyun Data and Guanghuan Xinnet. Refer to regional websites for details.

The code in this story is for educational purposes. The readers are solely responsible for whatever they build with it.

Solving Aurora DSQL’s IAM Token Problem: A New SeaTunnel Sink Connector for Seamless Data Migration

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

A 10-Minute Deep Dive Into the Core Architecture of Apache SeaTunnel and DataX

A 10-Minute Deep Dive Into the Core Architecture of Apache SeaTunnel and DataX

This One Data Company Abandoned DataX, Upgraded to SeaTunnel, and Turbocharged Its Workflows

101 Stories To Learn About Cloud Infrastructure

10 Things in Engineering We Don't Spend Enough Time On

10 Things I Did To Increase CloudTrail Logs Security

A 10-Minute Deep Dive Into the Core Architecture of Apache SeaTunnel and DataX

A 10-Minute Deep Dive Into the Core Architecture of Apache SeaTunnel and DataX

This One Data Company Abandoned DataX, Upgraded to SeaTunnel, and Turbocharged Its Workflows

101 Stories To Learn About Cloud Infrastructure

10 Things in Engineering We Don't Spend Enough Time On

10 Things I Did To Increase CloudTrail Logs Security

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps