Over the past two weeks, we interviewed several outstanding developers from the OSPP program to learn about their project experiences. Today, we’re sharing a detailed technical report from one of these projects — supporting Schema Evolution on the Flink engine — to help the community better understand the latest development progress in Apache SeaTunnel. supporting Schema Evolution on the Flink engine I. Completed Work According to the original plan (link) and schedule, the support for Schema Evolution on the Flink engine has been completed. link On the Sink side, the following components now support dynamic schema changes during streaming: Sink JdbcSinkWriterJdbcExactlyOnceSinkWriterConsoleSinkWriter JdbcSinkWriter JdbcSinkWriter JdbcExactlyOnceSinkWriter JdbcExactlyOnceSinkWriter ConsoleSinkWriter ConsoleSinkWriter All have passed testing with no significant bugs found during the schema evolution process. ✅ Dynamically inserted operators between source and transform. When a class implementing SupportSchemaEvolution is detected and the configuration is enabled, a SchemaOperator is inserted. source transform SupportSchemaEvolution SchemaOperator ✅ Implemented a SchemaCoordinator to receive flush reports from sink writers and schema change requests from SchemaOperator. SchemaCoordinator SchemaOperator ✅ Extended the SchemaChangeEvent subclass to support FlushEvent propagation. SchemaChangeEvent FlushEvent ✅ Enhanced SupportSchemaEvolutionSinkWriter to report flush success events and handle FlushEvent. SupportSchemaEvolutionSinkWriter FlushEvent ✅ Implemented SchemaOperator to detect and process schema change events emitted by the source and forward them downstream. SchemaOperator ✅ Overrode SupportSchemaEvolutionSinkWriterschema evolution methods, currently supporting JdbcSinkWriter, JdbcExactlyOnceSinkWriter, and ConsoleSinkWriter. All tested and working as expected. SupportSchemaEvolutionSinkWriter JdbcSinkWriter JdbcExactlyOnceSinkWriter ConsoleSinkWriter ✅ Extended FlinkRowCollector.collect() to support change event collection. FlinkRowCollector.collect() ✅ Extended FlinkSinkWriter to detect and process schema change events. FlinkSinkWriter ✅ Added detailed error codes and exception handling for schema evolution errors. ✅ Implemented automatic exception throwing and retry mechanism when schema change errors occur. II. Challenges and Solutions 1. Event Propagation Issue When inserting an operator between the source and transform, it must distinguish between normal data and event messages. If it encounters an event, it should block, wait for the flush and schema update to complete, then continue propagation. source transform Two approaches were considered: Option 1: Follow the Zeta engine’s approach by creating a new StreamElement class similar to Record. However, this would require modifying the entire data flow from source to sink, introducing high risk and invasiveness.Option 2: Add a special marker within SeaTunnelRow, such as an option in options, to indicate events. This approach minimizes intrusion but slightly violates the single-responsibility principle, as SeaTunnelRow ideally should not handle event logic. For now, the focus is on functionality and stability, with future refactoring planned for cleaner design. Option 1: Follow the Zeta engine’s approach by creating a new StreamElement class similar to Record. However, this would require modifying the entire data flow from source to sink, introducing high risk and invasiveness. Option 1: StreamElement Record source sink Option 2: Add a special marker within SeaTunnelRow, such as an option in options, to indicate events. This approach minimizes intrusion but slightly violates the single-responsibility principle, as SeaTunnelRow ideally should not handle event logic. For now, the focus is on functionality and stability, with future refactoring planned for cleaner design. Option 2: SeaTunnelRow options SeaTunnelRow This marker allows the SchemaOperator to detect schema change events. However, it also introduced an issue — creating an empty row caused sink write errors. This was resolved by adding detection logic on the sink side. SchemaOperator Through this mechanism, event propagation issues were successfully addressed. 2. Multi-Parallelism Issue In Flink CDC, during the incremental snapshot phase, multiple tasks run in parallel based on user-defined parallelism to read snapshot data. However, in the incremental phase, only one task continues reading to preserve event order. Therefore, the coordinator doesn’t need a complex design — it only needs to handle sink responses without worrying about duplicated or out-of-order reports. Further investigation showed that although the CDC source uses a single task to read binlogs, data is distributed downstream via KeyGroupStreamPartitioner by primary key hash, sending different keys to different sink subtasks. KeyGroupStreamPartitioner When sink parallelism differs from input parallelism, Flink performs primary key shuffle to realign partitions. primary key shuffle However, the following challenges were identified: Implementing shuffle logic would be complex.If multiple parallel sinks receive schema change commands simultaneously, non-idempotent databases (e.g., StarRocks) could be affected. A workaround could involve coordinating through acknowledgments, but this would add unnecessary complexity. Implementing shuffle logic would be complex. If multiple parallel sinks receive schema change commands simultaneously, non-idempotent databases (e.g., StarRocks) could be affected. A workaround could involve coordinating through acknowledgments, but this would add unnecessary complexity. For now, schema change detection forces sink parallelism to 1 for simplicity and correctness. Future iterations may reintroduce multi-parallelism support. 1 3. Flush and Request Execution Order Originally, the SchemaOperator handled schema changes by flushing first, then requesting a schema change from the coordinator. SchemaOperator flushing first requesting a schema change This caused issues because the coordinator’s SchemaChangeState wasn’t initialized yet, resulting in missed FlushEvent notifications and task timeout failures. SchemaChangeState FlushEvent Log analysis showed: 12:33:36,597 — FlushEvent processed; sink reported flush success.12:33:36,597 — Coordinator warning: “No schema change state found.”12:33:36,598 — Coordinator created schema change state. 12:33:36,597 — FlushEvent processed; sink reported flush success. FlushEvent 12:33:36,597 — Coordinator warning: “No schema change state found.” 12:33:36,598 — Coordinator created schema change state. The issue was resolved by reversing the execution order — requesting schema change first, then flushing — ensuring correct synchronization and preventing task blockage. requesting schema change first, then flushing 4. FlushData and Schema Change Separation Initially, FlushEvent contained SchemaChangeEvent, combining data flushing and schema change in one step. FlushEvent SchemaChangeEvent This mixed responsibilities and caused confusion. The design was refactored so that data flushing only handles data, while schema change is handled separately by the coordinator afterward. This separation clarified logic and improved maintainability. 5. Default Implementation and Interface Responsibilities To maintain backward compatibility, new methods in SupportSchemaEvolutionSinkWriter are temporarily marked as default. SupportSchemaEvolutionSinkWriter default Once all implementations are complete and stable, the default keyword will be removed in subsequent iterations. default 6. Handling Schema Change Failures: Mark Failure vs. Rollback If a job fails (due to network or other reasons), should it mark the job as failed and let Flink recover from checkpoints, or perform rollback? mark the job as failed perform rollback Flink CDC handles this by marking the job as failed and recovering from the last checkpoint. Following the same design, this project marks failure and lets the built-in retry mechanism handle recovery, leveraging Flink’s global retry strategies (no, fixed-delay, failure-rate) and SeaTunnel’s own retry logic. no fixed-delay failure-rate To improve debuggability, custom exception classes and error codes were introduced instead of using generic RuntimeException, providing clearer diagnostic information for both developers and users. RuntimeException III. Test Cases and Results Test Scenario: MySQL CDC → MySQL Test Scenario: ✅ Add Column Test Report✅ Drop Column Test Report✅ Modify Column Test Report✅ Change Column Test Report ✅ Add Column Test Report Add Column Test Report ✅ Drop Column Test Report Drop Column Test Report ✅ Modify Column Test Report Modify Column Test Report ✅ Change Column Test Report Change Column Test Report All tests passed successfully, confirming stable schema evolution behavior across multiple DDL event types. IV. Future Work Plan Implement the remaining sink types that support SupportSchemaEvolutionSinkWriter.Test data flow consistency across different data sources and fix minor bugs if found.Conduct stress testing with large datasets to check for potential blocking issues.Verify data consistency under high concurrency scenarios. Implement the remaining sink types that support SupportSchemaEvolutionSinkWriter. SupportSchemaEvolutionSinkWriter Test data flow consistency across different data sources and fix minor bugs if found. Conduct stress testing with large datasets to check for potential blocking issues. Verify data consistency under high concurrency scenarios. In Summary This project marks an important milestone in the evolution of Apache SeaTunnel’s Flink engine, enabling robust Schema Evolution support for streaming pipelines — a foundational step toward more flexible, self-adaptive data integration. Schema Evolution support

This story contains new, firsthand information uncovered by the writer.

Final Project Report 1: Schema Evolution Support on Apache SeaTunnel Flink Engine

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

A 10-Minute Deep Dive Into the Core Architecture of Apache SeaTunnel and DataX

A Brief History of Flink: Tracing the Big Data Engine’s Open-source Development

Better to Give and to Receive: Alibaba’s Open-source Contributions to Flink

Data Drama: Navigating the Spark-Flink Dilemma

Flink or Flunk? Why Ele.me Is Developing a Taste for Apache Flink

How to Master Stream Processing - Hopping and Tumbling Windows

A 10-Minute Deep Dive Into the Core Architecture of Apache SeaTunnel and DataX

A Brief History of Flink: Tracing the Big Data Engine’s Open-source Development

Better to Give and to Receive: Alibaba’s Open-source Contributions to Flink

Data Drama: Navigating the Spark-Flink Dilemma

Flink or Flunk? Why Ele.me Is Developing a Taste for Apache Flink

How to Master Stream Processing - Hopping and Tumbling Windows

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps