Following the article “SeaTunnel CDC Under the Hood: Snapshots, Backfills, and Why Your Checkpoints Time Out”, which detailed the implementation mechanisms and principles of the Apache SeaTunnel CDC Source, this article will continue to explore the underlying technical logic of Apache SeaTunnel CDC by explaining the relationship between Debezium and Apache SeaTunnel. SeaTunnel CDC Under the Hood: Snapshots, Backfills, and Why Your Checkpoints Time Out To summarize their relationship in one sentence: Debezium is the core underlying engine of SeaTunnel CDC, while SeaTunnel CDC encapsulates, enhances, and extends Debezium’s functionalities. Debezium is the core underlying engine of SeaTunnel CDC, while SeaTunnel CDC encapsulates, enhances, and extends Debezium’s functionalities. Below is a detailed explanation of their relationship: 1. Foundation and Core: The Role of Debezium “Debezium can be regarded as the pioneer of CDC.” Within the SeaTunnel CDC ecosystem, Debezium plays an irreplaceable “foundation” role. Provider of Core Capabilities: Debezium provides the most essential CDC functionality, namely monitoring row-level changes in source databases (such as MySQL Binlog, PostgreSQL WAL, etc.) and standardizing these changes into event streams.
Mature Connector Library: SeaTunnel leverages Debezium’s long-established, mature connector libraries to ensure stable support for various mainstream databases.
Standardized Data Format: Debezium defines a clear data structure (SourceRecord), containing the before and after states, operation type (Envelope Operation: CREATE/READ/UPDATE/DELETE), and other information, providing a standardized input for upper-layer processing. Provider of Core Capabilities: Debezium provides the most essential CDC functionality, namely monitoring row-level changes in source databases (such as MySQL Binlog, PostgreSQL WAL, etc.) and standardizing these changes into event streams. Provider of Core Capabilities Mature Connector Library: SeaTunnel leverages Debezium’s long-established, mature connector libraries to ensure stable support for various mainstream databases. Mature Connector Library Standardized Data Format: Debezium defines a clear data structure (SourceRecord), containing the before and after states, operation type (Envelope Operation: CREATE/READ/UPDATE/DELETE), and other information, providing a standardized input for upper-layer processing. Standardized Data Format SourceRecord 2. Key Turning Point: Dropping Kafka Connect in Favor of an Embedded Engine This is the most critical point for understanding their relationship. Traditional Debezium: Usually relies on Apache Kafka Connect for deployment, meaning data must flow through a Kafka cluster. While highly reliable, this approach introduces heavy infrastructure dependencies.
SeaTunnel’s Choice: To achieve a more lightweight and flexible integration, SeaTunnel does not use Debezium’s Kafka Connect mode. Instead, it utilizes Debezium’s embedded engine (debezium-embedded).
Nature of the Integration: SeaTunnel introduces Maven dependencies (debezium-api and debezium-embedded) to run the Debezium engine as a library directly within SeaTunnel’s process. This completely removes the mandatory dependency on a Kafka cluster. Traditional Debezium: Usually relies on Apache Kafka Connect for deployment, meaning data must flow through a Kafka cluster. While highly reliable, this approach introduces heavy infrastructure dependencies. Traditional Debezium SeaTunnel’s Choice: To achieve a more lightweight and flexible integration, SeaTunnel does not use Debezium’s Kafka Connect mode. Instead, it utilizes Debezium’s embedded engine (debezium-embedded). SeaTunnel’s Choice embedded engine ( debezium-embedded Nature of the Integration: SeaTunnel introduces Maven dependencies (debezium-api and debezium-embedded) to run the Debezium engine as a library directly within SeaTunnel’s process. This completely removes the mandatory dependency on a Kafka cluster. Nature of the Integration debezium-api debezium-embedded This completely removes the mandatory dependency on a Kafka cluster. 3. Orchestration and Encapsulation: The Architecture of SeaTunnel CDC SeaTunnel builds a sophisticated “orchestration layer” on top of the Debezium engine to manage and schedule Debezium’s operations. SeaTunnel sits at the top layer, handling read logic, deserialization, streaming fetch, and connection management; Debezium sits at the bottom layer, driving the database’s CDC mechanism and generating standardized data records. SeaTunnel’s utilization of Debezium’s core functionalities is summarized in the table below: Function

Provided by Debezium (Core Capability)

Used by SeaTunnel (Encapsulation/Invocation)



Full Snapshot Read

Snapshot reading

SnapshotChangeEventSourceexecutes SELECT reads



Incremental Read

Incremental reading

StreamingChangeEventSourcereads Binlog/WAL, etc.



Data Structure

Data record (SourceRecord)

Extracts raw before/after information



Operation Type

Envelope.Operation

Identifies CREATE/UPDATE/DELETE operations



State Management

Offset & Schema management

Tracks read positions and DDL changes Function

Provided by Debezium (Core Capability)

Used by SeaTunnel (Encapsulation/Invocation)



Full Snapshot Read

Snapshot reading

SnapshotChangeEventSourceexecutes SELECT reads



Incremental Read

Incremental reading

StreamingChangeEventSourcereads Binlog/WAL, etc.



Data Structure

Data record (SourceRecord)

Extracts raw before/after information



Operation Type

Envelope.Operation

Identifies CREATE/UPDATE/DELETE operations



State Management

Offset & Schema management

Tracks read positions and DDL changes Function

Provided by Debezium (Core Capability)

Used by SeaTunnel (Encapsulation/Invocation) Function Function Function Provided by Debezium (Core Capability) Provided by Debezium (Core Capability) Provided by Debezium (Core Capability) Used by SeaTunnel (Encapsulation/Invocation) Used by SeaTunnel (Encapsulation/Invocation) Used by SeaTunnel (Encapsulation/Invocation) Full Snapshot Read

Snapshot reading

SnapshotChangeEventSourceexecutes SELECT reads Full Snapshot Read Full Snapshot Read Full Snapshot Read Snapshot reading Snapshot reading SnapshotChangeEventSourceexecutes SELECT reads SnapshotChangeEventSourceexecutes SELECT reads SnapshotChangeEventSource Incremental Read

Incremental reading

StreamingChangeEventSourcereads Binlog/WAL, etc. Incremental Read Incremental Read Incremental Read Incremental reading Incremental reading StreamingChangeEventSourcereads Binlog/WAL, etc. StreamingChangeEventSourcereads Binlog/WAL, etc. StreamingChangeEventSource Data Structure

Data record (SourceRecord)

Extracts raw before/after information Data Structure Data Structure Data Structure Data record (SourceRecord) Data record (SourceRecord) SourceRecord Extracts raw before/after information Extracts raw before/after information Operation Type

Envelope.Operation

Identifies CREATE/UPDATE/DELETE operations Operation Type Operation Type Operation Type Envelope.Operation Envelope.Operation Identifies CREATE/UPDATE/DELETE operations Identifies CREATE/UPDATE/DELETE operations State Management

Offset & Schema management

Tracks read positions and DDL changes State Management State Management State Management Offset & Schema management Offset & Schema management Tracks read positions and DDL changes Tracks read positions and DDL changes 4. Data Flow and Translation The two are connected in the data processing pipeline. Debezium produces the “raw material,” and SeaTunnel “processes” it into a standardized internal format. Debezium Output: Produces SourceRecordcontaining raw change information.
SeaTunnel Translation: Uses DebeziumDeserializeSchema to deserialize SourceRecord, extract key information, and convert it into SeaTunnel’s internal row format SeaTunnelRow, while tagging the row type (RowKind, e.g., INSERT/UPDATE_AFTER). Debezium Output: Produces SourceRecordcontaining raw change information. Debezium Output SourceRecord SeaTunnel Translation: Uses DebeziumDeserializeSchema to deserialize SourceRecord, extract key information, and convert it into SeaTunnel’s internal row format SeaTunnelRow, while tagging the row type (RowKind, e.g., INSERT/UPDATE_AFTER). SeaTunnel Translation DebeziumDeserializeSchema SourceRecord SeaTunnelRow 5. Enhancement and Extension: The Value of SeaTunnel By embedding and encapsulating Debezium, SeaTunnel CDC achieves significant enhancements compared to the native Debezium solution, as illustrated below: Key Enhancements Provided by SeaTunnel: Key Enhancements Provided by SeaTunnel: Kafka Decoupling: This is the biggest difference. SeaTunnel CDC can write data directly to any supported Sink (e.g., data lake or warehouse) without passing through Kafka.
Parallel Reading Capability: SeaTunnel introduces parallel slicing to concurrently read full historical data, greatly improving efficiency.
Native Engine Integration: Deep integration with SeaTunnel (and Flink/Spark) checkpoint mechanism, ensuring exactly-once semantics.
Schema Evolution Support: Better handling of source-side DDL changes to adapt to table structure evolution. Kafka Decoupling: This is the biggest difference. SeaTunnel CDC can write data directly to any supported Sink (e.g., data lake or warehouse) without passing through Kafka. Kafka Decoupling Parallel Reading Capability: SeaTunnel introduces parallel slicing to concurrently read full historical data, greatly improving efficiency. Parallel Reading Capability Native Engine Integration: Deep integration with SeaTunnel (and Flink/Spark) checkpoint mechanism, ensuring exactly-once semantics. Native Engine Integration Schema Evolution Support: Better handling of source-side DDL changes to adapt to table structure evolution. Schema Evolution Support

Inside SeaTunnel CDC’s Debezium Integration: Embedded Engine, Offsets, and Checkpoints

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

A 10-Minute Deep Dive Into the Core Architecture of Apache SeaTunnel and DataX

What the heck is Apache SeaTunnel?

How to Sync Data From MySQL to Doris Using SeaTunnel

How to Simplify Data Flow By Multi-Table Synchronization With Apache SeaTunnel

Source Code Analysis of Apache SeaTunnel Zeta Engine (Part 1): Server Initialization

The Evolution of Apache SeaTunnel’s Technical Architecture and Its Applications in the AI Field

A 10-Minute Deep Dive Into the Core Architecture of Apache SeaTunnel and DataX

What the heck is Apache SeaTunnel?

How to Sync Data From MySQL to Doris Using SeaTunnel

How to Simplify Data Flow By Multi-Table Synchronization With Apache SeaTunnel

Source Code Analysis of Apache SeaTunnel Zeta Engine (Part 1): Server Initialization

The Evolution of Apache SeaTunnel’s Technical Architecture and Its Applications in the AI Field

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps