Over the past two weeks, we conducted a brief interview with the outstanding developers participating in this year's Summer of Open Source program, gaining an initial understanding of their development process and takeaways. Today, through their full final reports, we will take a deeper look into the technical details of each project, with the hope of helping everyone better understand the latest progress in the Apache DolphinScheduler project. brief interview Below is the complete report for the project “Adding a gRPC Plugin to Apache DolphinScheduler”: “Adding a gRPC Plugin to Apache DolphinScheduler” 1. Project Objective 1. Project Objective DolphinScheduler is an open-source distributed workflow scheduling system. To meet users’ needs for invoking gRPC services within workflows, a gRPC plugin for DolphinScheduler was designed and implemented. This plugin aims to: Support flexible configuration of gRPC service address, port, method name, request parameters, and other information in task settings.Provide both a visual interface and JSON-based configuration for specifying gRPC request parameters.Ensure that task execution results can be correctly returned and bound to the DolphinScheduler workflow context.Support exception handling and retry mechanisms to improve task stability.Implement SSL/TLS-encrypted connections to ensure data security. Support flexible configuration of gRPC service address, port, method name, request parameters, and other information in task settings. Provide both a visual interface and JSON-based configuration for specifying gRPC request parameters. Ensure that task execution results can be correctly returned and bound to the DolphinScheduler workflow context. Support exception handling and retry mechanisms to improve task stability. Implement SSL/TLS-encrypted connections to ensure data security. 2. Task Implementation 2. Task Implementation According to the RFC document, the plugin fulfills the following requirements: Configuration is similar to the HTTP Task, reducing the learning curve for users. Functional Requirements Functional Requirements 2.1 Frontend 2.1 Frontend Added gRPC task–related files and icons.Implemented the parameter form, including configuration items for service address, port, gRPC service definition, request method, request content, parameter list, and more.Used protobuf.js to implement JSON-to-Protobuf type mapping. Added gRPC task–related files and icons. Implemented the parameter form, including configuration items for service address, port, gRPC service definition, request method, request content, parameter list, and more. Used protobuf.js to implement JSON-to-Protobuf type mapping. protobuf.js 2.2 Backend 2.2 Backend Created foundational code for gRPC Task and implemented timeout and retry features.Parsed parameters received from the frontend and supported dynamic gRPC invocation generation.Implemented error handling and workflow context binding, with validation.Developed request sending, response processing, exception capturing, and retry logic.Security: Supported establishing connections using client-side SSL/TLS. Created foundational code for gRPC Task and implemented timeout and retry features. Parsed parameters received from the frontend and supported dynamic gRPC invocation generation. Implemented error handling and workflow context binding, with validation. Developed request sending, response processing, exception capturing, and retry logic. Security: Supported establishing connections using client-side SSL/TLS. Security: 2.3 Test Coverage 2.3 Test Coverage Comprehensive unit tests for overall plugin functionality:https://github.com/apache/dolphinscheduler/blob/dev/dolphinscheduler-task-plugin/dolphinscheduler-task-grpc/src/test/java/org/apache/dolphinscheduler/plugin/task/grpc/GrpcTaskTest.javaBasic gRPC functionality testsWorkflow and context binding testsException handling testsUnit tests for Protobuf parser syntax support:https://github.com/apache/dolphinscheduler/blob/dev/dolphinscheduler-task-plugin/dolphinscheduler-task-grpc/src/test/java/org/apache/dolphinscheduler/plugin/task/grpc/GrpcParserTest.javaBasic type testsTests for nested types such as map and repeated fieldsTests for optional and oneof fieldsAPI tests based on task parameters:https://github.com/apache/dolphinscheduler/blob/dev/dolphinscheduler-api-test/dolphinscheduler-api-test-case/src/test/java/org/apache/dolphinscheduler/api/test/cases/tasks/GrpcTaskAPITest.java Comprehensive unit tests for overall plugin functionality: https://github.com/apache/dolphinscheduler/blob/dev/dolphinscheduler-task-plugin/dolphinscheduler-task-grpc/src/test/java/org/apache/dolphinscheduler/plugin/task/grpc/GrpcTaskTest.java https://github.com/apache/dolphinscheduler/blob/dev/dolphinscheduler-task-plugin/dolphinscheduler-task-grpc/src/test/java/org/apache/dolphinscheduler/plugin/task/grpc/GrpcTaskTest.java Basic gRPC functionality tests Workflow and context binding tests Exception handling tests Unit tests for Protobuf parser syntax support: https://github.com/apache/dolphinscheduler/blob/dev/dolphinscheduler-task-plugin/dolphinscheduler-task-grpc/src/test/java/org/apache/dolphinscheduler/plugin/task/grpc/GrpcParserTest.java https://github.com/apache/dolphinscheduler/blob/dev/dolphinscheduler-task-plugin/dolphinscheduler-task-grpc/src/test/java/org/apache/dolphinscheduler/plugin/task/grpc/GrpcParserTest.java Basic type tests Tests for nested types such as map and repeated fields Tests for optional and oneof fields API tests based on task parameters: https://github.com/apache/dolphinscheduler/blob/dev/dolphinscheduler-api-test/dolphinscheduler-api-test-case/src/test/java/org/apache/dolphinscheduler/api/test/cases/tasks/GrpcTaskAPITest.java https://github.com/apache/dolphinscheduler/blob/dev/dolphinscheduler-api-test/dolphinscheduler-api-test-case/src/test/java/org/apache/dolphinscheduler/api/test/cases/tasks/GrpcTaskAPITest.java 2.4 Documentation & Examples 2.4 Documentation & Examples gRPC Task Plugin—Chinese documentation and examples:https://github.com/apache/dolphinscheduler/blob/dev/docs/docs/zh/guide/task/grpc.mdgRPC Task Plugin—English documentation and examples:https://github.com/apache/dolphinscheduler/blob/dev/docs/docs/en/guide/task/grpc.md gRPC Task Plugin—Chinese documentation and examples: https://github.com/apache/dolphinscheduler/blob/dev/docs/docs/zh/guide/task/grpc.md https://github.com/apache/dolphinscheduler/blob/dev/docs/docs/zh/guide/task/grpc.md gRPC Task Plugin—English documentation and examples: https://github.com/apache/dolphinscheduler/blob/dev/docs/docs/en/guide/task/grpc.md https://github.com/apache/dolphinscheduler/blob/dev/docs/docs/en/guide/task/grpc.md 3. Summary of Results 3. Summary of Results Completed the design and development of the DolphinScheduler gRPC plugin.Completed unit and integration testing, covering major functionalities and edge cases.Provided detailed plugin usage documentation, parameter lists, and examples. Completed the design and development of the DolphinScheduler gRPC plugin. Completed unit and integration testing, covering major functionalities and edge cases. Provided detailed plugin usage documentation, parameter lists, and examples. Project GitHub link: https://github.com/apache/dolphinscheduler/pull/17402 https://github.com/apache/dolphinscheduler/pull/17402 We express our gratitude to OSPP and all DolphinScheduler community mentors for their support and guidance.

Unit

Final Report | Adding a gRPC Plugin to Apache DolphinScheduler

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

A 10-Minute Deep Dive Into the Core Architecture of Apache SeaTunnel and DataX

How to Seamlessly Transfer From Airflow to Apache DolphinScheduler With Air2phin

A Comprehensive Guide to Building DolphinScheduler 3.2.0 Production-Grade Cluster Deployment

How to Regularly Delete Log Instances in DolphinScheduler

How to Enable Auto-Start for Apache DolphinScheduler

Solutions for Upgrading Apache DolphinScheduler from Version 1.3.4 to 3.1.2

A 10-Minute Deep Dive Into the Core Architecture of Apache SeaTunnel and DataX

How to Seamlessly Transfer From Airflow to Apache DolphinScheduler With Air2phin

A Comprehensive Guide to Building DolphinScheduler 3.2.0 Production-Grade Cluster Deployment

How to Regularly Delete Log Instances in DolphinScheduler

How to Enable Auto-Start for Apache DolphinScheduler

Solutions for Upgrading Apache DolphinScheduler from Version 1.3.4 to 3.1.2

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps