As this year’s Summer of Open Source (OSPP) program comes to an end, developers from the Apache DolphinScheduler community have completed their long development journey and achieved impressive results — including a brand-new gRPC task plugin for DolphinScheduler. Summer of Open Source (OSPP) Apache DolphinScheduler gRPC task plugin This contribution comes from Haibo Zhao, an outstanding student majoring in Electrical Engineering at Beijing Jiaotong University. Let’s take a look at how he completed this development task during the program! Haibo Zhao Electrical Engineering at Beijing Jiaotong University Personal Introduction Personal Introduction Name: Haibo Zhao Name: University: Beijing Jiaotong University, Electrical Engineering University: Mentor: Tan Hengliang Mentor: GitHub ID: npofsi GitHub ID: Research Interests: Time-series data analysis, model optimization, and control Research Interests: Hobbies: Digital painting, gourmet food Hobbies: Project Title Project Title Add a gRPC Task Plugin for Apache DolphinScheduler Add a gRPC Task Plugin for Apache DolphinScheduler Add a gRPC Task Plugin for Apache DolphinScheduler Project Overview Project Overview In OSPP 2025, I developed a new gRPC task plugin for Apache DolphinScheduler, enabling workflow tasks to interact directly with gRPC services. OSPP 2025 gRPC task plugin for Apache DolphinScheduler Similar to DolphinScheduler’s HTTP task, this plugin allows workflows to access external gRPC service interfaces. It was designed using the same configuration logic as the HTTP plugin and implemented based on DolphinScheduler’s SPI plugin mechanism. SPI plugin mechanism As a distributed workflow scheduler listed in the CNCF Landscape, DolphinScheduler has seen broad adoption in cloud-native environments. However, it previously lacked support for the gRPC protocol, which is widely used in such scenarios — thus motivating this project. distributed workflow scheduler CNCF Landscape gRPC protocol Development Path Development Path Based on this requirement, I analyzed how users typically invoke gRPC services within workflows. With guidance from my mentor, I designed an interaction model similar to API debugging tools, allowing separate configuration of schema and request messages, which are then merged during execution. This led to a plugin architecture that preprocesses protobuf and data on the frontend and merges them on the backend. When configuring the gRPC task, the frontend accepts parameters such as the gRPC service endpoint, protobuf service definition, request message, and SSL configuration. gRPC service endpoint protobuf service definition request message SSL configuration Upon saving, protobuf.js compiles the protobuf definition into a specific descriptor and passes it to the backend. During execution, the backend reconstructs the protobuf descriptor (which defines the message’s binary format), merges it with the user-provided JSON message, and sends a gRPC request to the configured service endpoint. protobuf.js protobuf descriptor Project Outcomes Project Outcomes Completed the design and implementation of the DolphinScheduler gRPC task plugin, which enables dynamic gRPC calls at Java runtime.Introduced a configuration model similar to gRPC debugging tools, allowing reuse of existing configuration files and test messages.Finished comprehensive unit and integration tests, covering major features and edge cases.Provided detailed documentation, parameter descriptions, and usage examples. Completed the design and implementation of the DolphinScheduler gRPC task plugin, which enables dynamic gRPC calls at Java runtime. gRPC task plugin Introduced a configuration model similar to gRPC debugging tools, allowing reuse of existing configuration files and test messages. Finished comprehensive unit and integration tests, covering major features and edge cases. unit and integration tests Provided detailed documentation, parameter descriptions, and usage examples. documentation Key Technical Highlights Key Technical Highlights The most challenging technical aspect was implementing dynamic invocation in gRPC. dynamic invocation in gRPC Unlike RESTful APIs that use a schema-on-read JSON format, gRPC uses schema-on-write through predefined protobuf definitions. These must be compiled ahead of time and are not transmitted with data. This design reduces payload size, improves efficiency and consistency, and allows more compile-time checks—but it also limits runtime flexibility. schema-on-read schema-on-write protobuf definitions To address this, I used protobuf.js as the compiler to pre-compile protobuf files on the frontend. Task data includes both the source protobuf and its intermediate descriptor, enabling the backend to build binary formats directly without depending on protoc. This means users can make gRPC requests without installing protoc. protobuf.js protoc without installing protoc After completing the development, the DolphinScheduler community conducted a short interview to learn more about participants’ experiences and reflections. Below is the interview transcript: interview transcript Q1: Among all projects, why did you choose Apache DolphinScheduler? Q1: Among all projects, why did you choose Apache DolphinScheduler? A: As a distributed workflow scheduling system, DolphinScheduler’s architecture itself is fascinating. It covers key topics such as task scheduling, distributed systems, and cloud-native design — areas I’ve always wanted to explore. A: gRPC, as the preferred RPC framework for modern microservices, has also been a technology I wanted to study deeply. Additionally, the DolphinScheduler community is very mature. During the application phase, I communicated with mentors who clearly explained the project goals and expectations. The documentation was extensive, and the community environment was friendly — all of which gave me confidence to complete the project successfully. Q2: Does this project relate to your academic studies? Q2: Does this project relate to your academic studies? A: My academic background is in electrical engineering, focusing on control and optimization of renewable energy systems. The power grid itself is a complex distributed system requiring high coordination — concepts closely related to cloud-native technologies. A: For example, China’s State Grid uses Kubernetes to manage a large number of monitoring nodes. Therefore, I wanted to learn more about technologies that could eventually be applied in my field. China’s State Grid The open-source community provides a platform for deep technical discussions and collaboration. I really enjoy exchanging ideas and working with people from different backgrounds to build better solutions. It’s a great way to meet like-minded peers and discover new opportunities. Q3: How has participating in this project influenced your studies and future plans? Q3: How has participating in this project influenced your studies and future plans? A: I learned a lot about gRPC and plan to use it in other systems in the future. I also gained a deeper understanding of workflow systems like DolphinScheduler, as well as valuable testing experience. A: Most importantly, I felt a strong sense of participation. With my mentor’s encouragement, I was able to complete the entire process — from submitting the RFC and issue to having my PR merged. That sense of accomplishment was deeply rewarding. Q4: What was the biggest challenge during development, and how did you overcome it? Q4: What was the biggest challenge during development, and how did you overcome it? A: The biggest challenge wasn’t a specific technical problem, but time management. A: time management Most of my time is devoted to my lab research, so I had to contribute to the open-source project in the evenings and on weekends. However, the community’s constant positive feedback kept me motivated. The sense of achievement, belonging, and recognition from contributing value was my biggest driving force. Q5: How long have you been involved in open source? Do you enjoy it? Q5: How long have you been involved in open source? Do you enjoy it? A: My first exposure to open source came from a book about JavaScript optimization, which introduced me to open-source projects. Although I didn’t participate directly at the time, it sparked my curiosity. A: Later, after using open-source software and learning its philosophy, I decided to join in. The experience has been very enjoyable. Unlike working in isolation, open source amplifies individual contributions through sharing and feedback, helping me grow faster. I’ve also learned to pay more attention to upstream projects and gained a lot of practical experience. Q6: What was your first impression of the DolphinScheduler community? Any feedback or suggestions? Q6: What was your first impression of the DolphinScheduler community? Any feedback or suggestions? A: My first impression came during the application phase. The mentors were friendly and patient, answering all my questions in detail. A: The community’s comprehensive documentation also helped me quickly set up the environment and run the software successfully — showing the community’s professionalism and maturity. Through OSPP, I learned a lot about software development and hope to further understand how open-source communities operate. One improvement I’d suggest is updating the SPI-related documentation, which is currently somewhat outdated. Enhancing this could make plugin development easier and faster, boosting DolphinScheduler’s ecosystem. SPI-related documentation Q7: Will you continue contributing to the Apache DolphinScheduler community? Q7: Will you continue contributing to the Apache DolphinScheduler community? A: Yes, definitely. Especially with ongoing maintenance of the gRPC plugin — I plan to further improve its maintainability and replace some low-maintainability components. A:

The code in this story is for educational purposes. The readers are solely responsible for whatever they build with it.

Between Two Computer Monitors: This story includes an interview between the writer and guest/interviewee. 

DolphinScheduler Adds First-Ever gRPC Task Plugin, Thanks to OSPP 2025 Contributor Haibo Zhao

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

3 Ways to Seamlessly Integrate Databend with SeaTunnel for Streaming ETL

How to Seamlessly Transfer From Airflow to Apache DolphinScheduler With Air2phin

A Comprehensive Guide to Building DolphinScheduler 3.2.0 Production-Grade Cluster Deployment

How to Regularly Delete Log Instances in DolphinScheduler

How to Enable Auto-Start for Apache DolphinScheduler

Solutions for Upgrading Apache DolphinScheduler from Version 1.3.4 to 3.1.2

3 Ways to Seamlessly Integrate Databend with SeaTunnel for Streaming ETL

How to Seamlessly Transfer From Airflow to Apache DolphinScheduler With Air2phin

A Comprehensive Guide to Building DolphinScheduler 3.2.0 Production-Grade Cluster Deployment

How to Regularly Delete Log Instances in DolphinScheduler

How to Enable Auto-Start for Apache DolphinScheduler

Solutions for Upgrading Apache DolphinScheduler from Version 1.3.4 to 3.1.2

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps