In today’s global wave of digital transformation, intelligent manufacturing has become the core engine driving high-quality development in the manufacturing industry. However, on the path toward intelligence, enterprises face numerous challenges: data silos across multiple systems, complex scheduling dependencies, delayed monitoring and alerting, and many other issues that emerge one after another. Today, we'll share a practical experience on how Apache DolphinScheduler is applied in real-world intelligent manufacturing scenarios within a smart factory in Shenzhen. The Era of Intelligent Manufacturing With the deep advancement of Industry 4.0, intelligent manufacturing has become a focal point of global manufacturing competition. Intelligent manufacturing maturity models are typically divided into multiple levels, from low to high. Enterprises must gradually enhance automation, digitalization, and networking capabilities, ultimately achieving intelligent production. Throughout this process, data becomes a core production factor. How to efficiently, reliably, and stably collect, process, and schedule data has become a major challenge faced by every manufacturing enterprise. Modern manufacturing enterprises operate in increasingly complex data environments. On one hand, they rely on numerous business systems, including MES (Manufacturing Execution Systems), ERP (Enterprise Resource Planning), WMS (Warehouse Management Systems), WCS (Warehouse Control Systems), CRM (Customer Relationship Management), QMS (Quality Management Systems), PLM (Product Lifecycle Management), SCM (Supply Chain Management), APS (Advanced Planning and Scheduling), and more. Data interactions among these systems are often implemented through hard-coded integrations, resulting in intricate inter-system relationships, high maintenance costs, poor scalability, and difficulties in troubleshooting. On the other hand, enterprises also face complex network environments, such as corporate production networks, internal factory networks, and international or domestic leased-line networks. Data collection, transmission, and scheduling requirements vary significantly across these environments, making unified management and task isolation a major challenge. The Dilemma of Traditional Data Processing Approaches As intelligent manufacturing advances data-driven initiatives, enterprises encounter multidimensional pain points. First, there are foundational barriers caused by data diversity: device protocols are highly heterogeneous, ranging from proprietary protocols such as PLM and S7 to general protocols like MQTT. Data formats vary as well, including binary and semi-structured data. Combined with differences across vendors and production lines, this makes standardization extremely difficult. foundational barriers caused by data diversity On top of that, cross-system and cross-factory data collaboration becomes particularly challenging. Data pipelines span devices, multiple systems, and geographically distributed factories. Network environments mix intranets, dedicated lines, and public networks. Meanwhile, business scenarios such as production scheduling and capacity calculation impose strict real-time data requirements, further increasing complexity. cross-system and cross-factory data collaboration becomes particularly challenging At the same time, data visualization and traceability capabilities are insufficient. Traditional systems cannot intuitively display data flow nodes. Logs are stored in a scattered manner, making exception diagnosis inefficient. Building a complete traceability system requires significant additional manpower. data visualization and traceability capabilities are insufficient Finally, data collection quality lacks effective guarantees. Various exceptions related to networks and devices occur frequently, while anomaly detection is delayed and manual recovery is inefficient. When failures occur during multi-system interactions, root cause analysis often relies on deep familiarity with the entire data chain, further undermining data reliability. data collection quality lacks effective guarantees Apache DolphinScheduler as the Solution To address these challenges, Apache DolphinScheduler provides a comprehensive solution. As a distributed, highly scalable, and visual workflow scheduling platform, it has demonstrated strong capabilities in manufacturing scenarios. Worker Node Grouping: Adapting to Multi-Network Environments To cope with complex manufacturing network environments, Apache DolphinScheduler introduces flexible isolation strategies through Worker node grouping. Worker nodes are grouped by network environment—such as corporate production network Workers, factory internal network Workers, and international/domestic leased-line Workers—and further divided by business type, including PLC device data collection, production data processing, and quality data analysis. flexible isolation strategies This design enables task isolation across different network environments and business scenarios, ensuring data collection security and reliability. It effectively supports critical use cases such as production data lake ingestion, customer data feedback, and cross-network data synchronization. Data Collection In terms of data collection, Apache DolphinScheduler builds a complete data processing pipeline. The data source layer includes IoT devices (such as sensors, heartbeat data, status monitoring, and operational data), business systems (MES, WMS, APS, SAP, and other databases), agent probes, and user-uploaded data. For processing, DataX is used for offline data synchronization, Flink for real-time stream processing, and Kafka as a message queue buffer. All data is eventually ingested into a unified data lake to support BI analysis and AI applications. Through Apache DolphinScheduler’s unified orchestration, enterprises can manage the entire lifecycle from data collection and processing to downstream applications. Data Interaction In traditional architectures, systems interact with each other in a point-to-point manner, resulting in highly complex relationships. After introducing Apache DolphinScheduler, all data interactions are centralized through a scheduling hub. This enables centralized management of all data interaction tasks, visual monitoring of execution status, unified exception handling and alerting mechanisms, while also reducing coupling between systems and significantly improving reliability. centralized management of all data interaction tasks, visual monitoring of execution status, unified exception handling and alerting mechanisms Template-Based Data Collection and Distribution Across Multiple Factories For manufacturing enterprises operating multiple factories, Apache DolphinScheduler offers a template-based solution. How can homogeneous systems—such as unified MES/WMS deployments or identical PLC device types—be rolled out rapidly? By solidifying core workflows (task list reading, parameter injection, collection/distribution execution, completion or exception marking) into reusable templates, and combining them with task configuration tables (including data source settings, SQL statements, source/target system IDs, dedicated parameters, and checkpoint settings), enterprises achieve a flexible model of “common templates with customized parameters.” “common templates with customized parameters.” This approach delivers significant advantages: Parameterized configuration allows core workflows to be fixed as templates, while factory-specific parameters (IP addresses, accounts, paths) are configured independently.Batch deployment capabilities enable dozens of factories to be deployed within a single day, dramatically improving efficiency.A unified iteration mechanism ensures that template updates are automatically synchronized to all factories without manual adjustments.Flexible extensibility supports template version management, allowing customized derivatives for specific factories (e.g., additional data fields).Cross-scenario support accommodates both “multi-factory data collection to headquarters” and “headquarters data distribution to multiple factories” (such as unified production plan delivery). Parameterized configuration allows core workflows to be fixed as templates, while factory-specific parameters (IP addresses, accounts, paths) are configured independently. Batch deployment capabilities enable dozens of factories to be deployed within a single day, dramatically improving efficiency. A unified iteration mechanism ensures that template updates are automatically synchronized to all factories without manual adjustments. Flexible extensibility supports template version management, allowing customized derivatives for specific factories (e.g., additional data fields). Cross-scenario support accommodates both “multi-factory data collection to headquarters” and “headquarters data distribution to multiple factories” (such as unified production plan delivery). A Qualitative Leap: From Manual Workshops to Industrial Assembly Lines After introducing Apache DolphinScheduler, the enterprise achieved a qualitative transformation in data processing. DimensionTraditional CodingApache DolphinSchedulerDevelopment EfficiencyRequires database connection, exception handling, repeated module coding; high labor inputDrag-and-drop configuration, built-in 10+ components, complete development in minutesDependency ManagementDifficult to manage complex dependencies (e.g., blood relationship, priority); error-proneVisual DAG operation interfaceMonitoring & AlertingManual monitoring/log review; delayed fault discoveryAutomatic monitoring, real-time task status tracking, real-time alerts (DingTalk, etc. for enterprises)Fault RecoveryManual code modification, tedious rollback/recoveryOne-click restart/stop, built-in automatic retry functionResource SchedulingLack of management; easy to cause single-machine CPU/memory overload; high distributed resource wasteDistributed & centralized resource management; fast dynamic scheduling via visual means DimensionTraditional CodingApache DolphinSchedulerDevelopment EfficiencyRequires database connection, exception handling, repeated module coding; high labor inputDrag-and-drop configuration, built-in 10+ components, complete development in minutesDependency ManagementDifficult to manage complex dependencies (e.g., blood relationship, priority); error-proneVisual DAG operation interfaceMonitoring & AlertingManual monitoring/log review; delayed fault discoveryAutomatic monitoring, real-time task status tracking, real-time alerts (DingTalk, etc. for enterprises)Fault RecoveryManual code modification, tedious rollback/recoveryOne-click restart/stop, built-in automatic retry functionResource SchedulingLack of management; easy to cause single-machine CPU/memory overload; high distributed resource wasteDistributed & centralized resource management; fast dynamic scheduling via visual means DimensionTraditional CodingApache DolphinScheduler DimensionTraditional CodingApache DolphinScheduler Dimension Traditional Coding Apache DolphinScheduler Development EfficiencyRequires database connection, exception handling, repeated module coding; high labor inputDrag-and-drop configuration, built-in 10+ components, complete development in minutesDependency ManagementDifficult to manage complex dependencies (e.g., blood relationship, priority); error-proneVisual DAG operation interfaceMonitoring & AlertingManual monitoring/log review; delayed fault discoveryAutomatic monitoring, real-time task status tracking, real-time alerts (DingTalk, etc. for enterprises)Fault RecoveryManual code modification, tedious rollback/recoveryOne-click restart/stop, built-in automatic retry functionResource SchedulingLack of management; easy to cause single-machine CPU/memory overload; high distributed resource wasteDistributed & centralized resource management; fast dynamic scheduling via visual means Development EfficiencyRequires database connection, exception handling, repeated module coding; high labor inputDrag-and-drop configuration, built-in 10+ components, complete development in minutes Development Efficiency Requires database connection, exception handling, repeated module coding; high labor input Drag-and-drop configuration, built-in 10+ components, complete development in minutes Dependency ManagementDifficult to manage complex dependencies (e.g., blood relationship, priority); error-proneVisual DAG operation interface Dependency Management Difficult to manage complex dependencies (e.g., blood relationship, priority); error-prone Visual DAG operation interface Monitoring & AlertingManual monitoring/log review; delayed fault discoveryAutomatic monitoring, real-time task status tracking, real-time alerts (DingTalk, etc. for enterprises) Monitoring & Alerting Manual monitoring/log review; delayed fault discovery Automatic monitoring, real-time task status tracking, real-time alerts (DingTalk, etc. for enterprises) Fault RecoveryManual code modification, tedious rollback/recoveryOne-click restart/stop, built-in automatic retry function Fault Recovery Manual code modification, tedious rollback/recovery One-click restart/stop, built-in automatic retry function Resource SchedulingLack of management; easy to cause single-machine CPU/memory overload; high distributed resource wasteDistributed & centralized resource management; fast dynamic scheduling via visual means Resource Scheduling Lack of management; easy to cause single-machine CPU/memory overload; high distributed resource waste Distributed & centralized resource management; fast dynamic scheduling via visual means Traditional coding approaches require extensive development of data connections, exception handling, and retry logic, resulting in heavy manpower investment. In contrast, Apache DolphinScheduler offers drag-and-drop configuration and dozens of built-in plugins, enabling development to be completed in minutes. Dependency management improvements are equally significant. Traditional methods struggle with complex cross-system scheduling and require careful handling of idempotency and consistency, making them error-prone. Apache DolphinScheduler’s visual DAG makes dependency orchestration intuitive and efficient. Monitoring and alerting capabilities see the most dramatic improvement. Traditional approaches rely on custom scripts or manual log inspection, leading to delayed issue detection. Apache DolphinScheduler provides built-in monitoring with real-time task status and logs, and integrates with enterprise messaging tools such as WeChat, DingTalk, and email for alerts. Fault tolerance and recovery are also enhanced. Traditional methods require manual code or script modifications and complex backfill logic, while Apache DolphinScheduler offers one-click reruns, stop controls, and built-in automatic retries. Resource scheduling capabilities are likewise strengthened. Traditional setups lack unified management, often leading to CPU or memory exhaustion and downtime. Apache DolphinScheduler adopts a distributed, decentralized cluster architecture, enabling rapid dynamic scaling through monitoring and fine-grained resource management. These improvements generate tangible value at multiple levels. At the development level, drag-and-drop workflows lower technical barriers, parameter automation boosts efficiency, second-level log location accelerates troubleshooting, and overall operational costs are significantly reduced. development level At the business level, visual monitoring provides full transparency, multi-channel alerts ensure timely response, flexible backfill strategies handle diverse exceptions, and unified cross-system coordination reduces excessive reliance on developers. business level At the decision-making level, depersonalization transforms knowledge into organizational assets that are resilient to staff turnover. Comprehensive audit trails meet compliance requirements. Centralized database configuration reduces security risks. Transparent workflows enable optimization, while quantified resource usage supports data-driven decision-making. Together, these benefits form a solid foundation for digital transformation. decision-making level Practical Outcomes and Future Outlook Through Apache DolphinScheduler, this intelligent manufacturing enterprise achieved significant improvements across multiple dimensions: higher development efficiency, shorter deployment cycles, sharply reduced operations costs and manpower investment, and a substantial increase in task success rates. The platform supports rapid expansion, enabling new factories to be deployed within one day, while realizing standardized processes, transparent management, and data-driven decision-making. Looking ahead, as intelligent manufacturing continues to evolve, data scheduling will play an increasingly critical role. As an open-source project, Apache DolphinScheduler will continue to evolve in several directions: AI enablement for intelligent scheduling and predictive maintenance, deeper cloud-native integration for elastic scalability, and ecosystem expansion through richer plugin coverage for more business scenarios. Conclusion On the journey toward intelligent manufacturing, data scheduling is not the destination—it is the starting point. Apache DolphinScheduler helps enterprises solve the “last mile” of data processing, enabling them to focus more on business innovation and value creation. Digital transformation is a long and challenging road, but steady progress leads to success. May more manufacturing enterprises harness the power of open source and achieve a remarkable transformation from “manufacturing” to “intelligent manufacturing.”

This story contains AI-generated text. The author has used AI either for research, to generate outlines, or write the text itself. 

Apache

Assembly

How a Shenzhen Smart Factory Uses Apache DolphinScheduler to Orchestrate Industrial Data

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

A 10-Minute Deep Dive Into the Core Architecture of Apache SeaTunnel and DataX

The Noonification: How Often Do NFTs Pass The Howey Test? (1/13/2023)

Darwin's Hybrid Intelligence to Align AI & Human Goals for Startups & VCs

The Noonification: White Man (11/26/2022)

The Noonification: The Metaverse is a Sh*tshow (11/2/2022)

100 Days of AI Day 1: From Newsletter to Podcast, Leveraging AI for Audio Transformation

A 10-Minute Deep Dive Into the Core Architecture of Apache SeaTunnel and DataX

The Noonification: How Often Do NFTs Pass The Howey Test? (1/13/2023)

Darwin's Hybrid Intelligence to Align AI & Human Goals for Startups & VCs

The Noonification: White Man (11/26/2022)

The Noonification: The Metaverse is a Sh*tshow (11/2/2022)

100 Days of AI Day 1: From Newsletter to Podcast, Leveraging AI for Audio Transformation

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps