Background Recently, while exploring the big data scheduling platform Apache DolphinScheduler, I noticed that it is distributed yet decentralized, which is quite different from the traditional Master–Slave and High Availability (HA) architectures I was familiar with. This raised a natural question: what does decentralization actually mean? What is special about this architectural approach, and what advantages does it bring? This article provides a detailed explanation. Apache DolphinScheduler distributed yet decentralized Master–Slave High Availability (HA) decentralization Common Architecture Patterns in the Big Data Domain To understand the relationships and differences among decentralized design, Master–Slave architecture, and HA, we must first clarify their core definitions. Then, by analyzing them from three dimensions—architectural goals, node relationships, and availability mechanisms—we can systematically explain their connections and highlight their differences through comparison. decentralized design Master–Slave architecture HA architectural goals node relationships availability mechanisms I. Core Concept Definitions Clarifying the essence of each concept is the foundation for understanding their relationships. Table 1. Core Architecture Concepts Table 1. Core Architecture Concepts ConceptDefinitionKey CharacteristicsMaster–Slave ArchitectureA centralized architecture in which a Master node is responsible for coordination and control, while Slave nodes handle execution tasksClear role separation, centralized control, potential single point of failureDecentralized ArchitectureAn architecture with no fixed central node; all nodes are logically equal and collaborate through coordination mechanismsNo single control node, peer-to-peer collaboration, strong fault toleranceHigh Availability (HA)A system design objective aimed at ensuring continuous service despite node or component failuresRedundancy, fault tolerance, failover mechanisms ConceptDefinitionKey CharacteristicsMaster–Slave ArchitectureA centralized architecture in which a Master node is responsible for coordination and control, while Slave nodes handle execution tasksClear role separation, centralized control, potential single point of failureDecentralized ArchitectureAn architecture with no fixed central node; all nodes are logically equal and collaborate through coordination mechanismsNo single control node, peer-to-peer collaboration, strong fault toleranceHigh Availability (HA)A system design objective aimed at ensuring continuous service despite node or component failuresRedundancy, fault tolerance, failover mechanisms ConceptDefinitionKey Characteristics ConceptDefinitionKey Characteristics Concept Definition Key Characteristics Master–Slave ArchitectureA centralized architecture in which a Master node is responsible for coordination and control, while Slave nodes handle execution tasksClear role separation, centralized control, potential single point of failureDecentralized ArchitectureAn architecture with no fixed central node; all nodes are logically equal and collaborate through coordination mechanismsNo single control node, peer-to-peer collaboration, strong fault toleranceHigh Availability (HA)A system design objective aimed at ensuring continuous service despite node or component failuresRedundancy, fault tolerance, failover mechanisms Master–Slave ArchitectureA centralized architecture in which a Master node is responsible for coordination and control, while Slave nodes handle execution tasksClear role separation, centralized control, potential single point of failure Master–Slave Architecture A centralized architecture in which a Master node is responsible for coordination and control, while Slave nodes handle execution tasks Clear role separation, centralized control, potential single point of failure Decentralized ArchitectureAn architecture with no fixed central node; all nodes are logically equal and collaborate through coordination mechanismsNo single control node, peer-to-peer collaboration, strong fault tolerance Decentralized Architecture An architecture with no fixed central node; all nodes are logically equal and collaborate through coordination mechanisms No single control node, peer-to-peer collaboration, strong fault tolerance High Availability (HA)A system design objective aimed at ensuring continuous service despite node or component failuresRedundancy, fault tolerance, failover mechanisms High Availability (HA) A system design objective aimed at ensuring continuous service despite node or component failures Redundancy, fault tolerance, failover mechanisms II. Core Relationships Among the Three These three concepts are not mutually exclusive. In practice, they are often combined to achieve architectural goals. Their core relationship can be summarized as follows: HA is the shared objective, while Master–Slave and decentralized architectures are two different paths to achieving HA. not mutually exclusive HA is the shared objective, while Master–Slave and decentralized architectures are two different paths to achieving HA. 1. Master–Slave and HA: HA in a Centralized Architecture A pure Master–Slave architecture does not inherently provide HA. If the Master fails, the entire system may become unavailable. To compensate for this weakness, additional HA mechanisms are introduced, forming a Master–Slave + failover model. not inherently provide HA Master–Slave + failover Principle: Failure detection mechanisms (such as Keepalived or ZooKeeper) continuously monitor the Master’s status.When the Master fails, the system automatically elects a new Master from the Slave nodes—typically after heartbeat detection confirms the failure and triggers failover—so that service continuity is maintained.Typical scenarios:MySQL primary–replica replication (writes on the primary, reads on replicas) combined with MySQL High Availability (MHA) tools.Kubernetes control-plane high availability, where multiple Master nodes participate in leader election via etcd, which is essentially an HA enhancement through Master clustering. Principle: Failure detection mechanisms (such as Keepalived or ZooKeeper) continuously monitor the Master’s status. Principle When the Master fails, the system automatically elects a new Master from the Slave nodes—typically after heartbeat detection confirms the failure and triggers failover—so that service continuity is maintained. Typical scenarios: Typical scenarios MySQL primary–replica replication (writes on the primary, reads on replicas) combined with MySQL High Availability (MHA) tools. Kubernetes control-plane high availability, where multiple Master nodes participate in leader election via etcd, which is essentially an HA enhancement through Master clustering. 2. Decentralized Design and HA: Architectures Born for HA The core characteristics of decentralized architecture—node equality and the absence of a fixed central dependency—naturally align with HA goals, without requiring “patch-style” remedies. node equality and the absence of a fixed central dependency Data layer: Each node stores full or replicated data (for example, the Bitcoin blockchain), so the failure of a single node does not compromise data integrity.Service layer: Requests can be routed to any healthy node (as in P2P file-sharing networks), avoiding system-wide outages caused by a central node failure.Coordination layer: Consensus mechanisms such as Raft ensure consistency among nodes and prevent split-brain scenarios, as seen in decentralized systems like the distributed database TiDB. Data layer: Each node stores full or replicated data (for example, the Bitcoin blockchain), so the failure of a single node does not compromise data integrity. Data layer Service layer: Requests can be routed to any healthy node (as in P2P file-sharing networks), avoiding system-wide outages caused by a central node failure. Service layer Coordination layer: Consensus mechanisms such as Raft ensure consistency among nodes and prevent split-brain scenarios, as seen in decentralized systems like the distributed database TiDB. Coordination layer 3. The Underlying Link: Availability Goals Drive Architectural Choices Whether Master–Slave or decentralized, both ultimately serve the availability objective. availability objective HA defines the “what”—the goal of uninterrupted service.Master–Slave and decentralization define the “how”—two architectural approaches to achieving that goal. HA defines the “what”—the goal of uninterrupted service. HA defines the “what” Master–Slave and decentralization define the “how”—two architectural approaches to achieving that goal. Master–Slave and decentralization define the “how” If a system prioritizes simplicity and low latency (such as read–write separation), a Master–Slave architecture with HA enhancements may be appropriate. If a system prioritizes extreme fault tolerance, resilience, and scalability (such as in financial systems or distributed schedulers), a decentralized design is often the better choice. simplicity and low latency Master–Slave architecture with HA enhancements extreme fault tolerance, resilience, and scalability decentralized design III. Key Differences Among the Three Comparing the three across multiple dimensions makes their differences clear. Table 2. Architectural Differences Table 2. Architectural Differences DimensionMaster–Slave ArchitectureDecentralized ArchitectureHigh Availability (HA)Node RelationshipMaster controls SlavesAll nodes are peersNot an architecture, but a system objectiveSingle Point of FailureExists (the Master)None by designEliminated through redundancyFailure ImpactMaster failure may disrupt serviceIndividual node failures have limited impactService continues despite failuresScalabilityLimited by Master capacityHorizontally scalableDepends on the underlying architectureTypical Use CasesDatabases, simple schedulersDistributed databases, schedulersAny system requiring continuous service DimensionMaster–Slave ArchitectureDecentralized ArchitectureHigh Availability (HA)Node RelationshipMaster controls SlavesAll nodes are peersNot an architecture, but a system objectiveSingle Point of FailureExists (the Master)None by designEliminated through redundancyFailure ImpactMaster failure may disrupt serviceIndividual node failures have limited impactService continues despite failuresScalabilityLimited by Master capacityHorizontally scalableDepends on the underlying architectureTypical Use CasesDatabases, simple schedulersDistributed databases, schedulersAny system requiring continuous service DimensionMaster–Slave ArchitectureDecentralized ArchitectureHigh Availability (HA) DimensionMaster–Slave ArchitectureDecentralized ArchitectureHigh Availability (HA) Dimension Master–Slave Architecture Decentralized Architecture High Availability (HA) Node RelationshipMaster controls SlavesAll nodes are peersNot an architecture, but a system objectiveSingle Point of FailureExists (the Master)None by designEliminated through redundancyFailure ImpactMaster failure may disrupt serviceIndividual node failures have limited impactService continues despite failuresScalabilityLimited by Master capacityHorizontally scalableDepends on the underlying architectureTypical Use CasesDatabases, simple schedulersDistributed databases, schedulersAny system requiring continuous service Node RelationshipMaster controls SlavesAll nodes are peersNot an architecture, but a system objective Node Relationship Master controls Slaves All nodes are peers Not an architecture, but a system objective Single Point of FailureExists (the Master)None by designEliminated through redundancy Single Point of Failure Exists (the Master) None by design Eliminated through redundancy Failure ImpactMaster failure may disrupt serviceIndividual node failures have limited impactService continues despite failures Failure Impact Master failure may disrupt service Individual node failures have limited impact Service continues despite failures ScalabilityLimited by Master capacityHorizontally scalableDepends on the underlying architecture Scalability Limited by Master capacity Horizontally scalable Depends on the underlying architecture Typical Use CasesDatabases, simple schedulersDistributed databases, schedulersAny system requiring continuous service Typical Use Cases Databases, simple schedulers Distributed databases, schedulers Any system requiring continuous service Summary: Clarifying the Relationship in One Sentence Master–Slave is a centralized division-of-labor architecture that requires HA mechanisms to avoid single points of failure.Decentralized architecture is a centerless, peer-based architecture that inherently provides high availability.HA is the goal of uninterrupted service, which can be achieved through either Master–Slave or decentralized architectures and represents a shared pursuit of both. Master–Slave is a centralized division-of-labor architecture that requires HA mechanisms to avoid single points of failure. Master–Slave Decentralized architecture is a centerless, peer-based architecture that inherently provides high availability. Decentralized architecture HA is the goal of uninterrupted service, which can be achieved through either Master–Slave or decentralized architectures and represents a shared pursuit of both. HA The Decentralized Architecture of Apache DolphinScheduler Core Design Principles Apache DolphinScheduler’s decentralized design is reflected in several key aspects: No central node: Both MasterServer and WorkerServer adopt a distributed, centerless design, with no traditional master–slave role distinction.ZooKeeper-based registration: All nodes register ephemeral nodes in ZooKeeper and handle fault tolerance by listening for ZooKeeper node changes.Dynamic election mechanisms: ZooKeeper distributed locks are used to dynamically elect coordinators when management responsibilities are required. No central node: Both MasterServer and WorkerServer adopt a distributed, centerless design, with no traditional master–slave role distinction. No central node ZooKeeper-based registration: All nodes register ephemeral nodes in ZooKeeper and handle fault tolerance by listening for ZooKeeper node changes. ZooKeeper-based registration Dynamic election mechanisms: ZooKeeper distributed locks are used to dynamically elect coordinators when management responsibilities are required. Dynamic election mechanisms Task Reception and Allocation In Apache DolphinScheduler’s decentralized architecture, there is no single control center. A cluster of Master nodes jointly handles task reception and allocation. When an external task submission request arrives, any Master node can receive it. no single control center any Master node Each Master node evaluates tasks based on locally maintained resource information—such as CPU usage, memory availability, and network load—together with predefined scheduling algorithms to determine task placement. Master nodes synchronize their states through heartbeat mechanisms. If one Master becomes overloaded, it can transfer tasks to other Masters with lower load. This dynamic allocation ensures balanced resource utilization across the cluster and avoids single-node bottlenecks. Task Execution and Coordination Worker nodes are responsible for actual task execution. After startup, each Worker registers its resource information and execution capabilities with the Master cluster. Master nodes consider these attributes when assigning tasks. Once a Worker receives a task from a Master, it executes the task independently. During execution, the Worker continuously reports task status—such as start, in progress, completion, or failure—to the Masters. If exceptions occur, Workers follow predefined handling strategies, such as task retries or alert notifications. Based on feedback from Workers, Master nodes coordinate and optimize the overall scheduling process to ensure efficient and reliable execution. Compared with traditional Master–Slave and HA-enhanced architectures, Apache DolphinScheduler’s decentralized design offers clear advantages. Master–Slave systems inherently risk failure at the Master node, while HA solutions rely on standby nodes that may cause brief service interruptions during failover and often remain underutilized. In contrast, DolphinScheduler treats all nodes as equal participants. When some nodes fail, others seamlessly take over their workload, ensuring continuous operation. Moreover, decentralized architecture enables straightforward horizontal scaling. As workloads grow, system capacity can be increased simply by adding nodes, without complex reconfiguration. This significantly improves flexibility, resilience, and operational efficiency while reducing long-term maintenance costs.

Apache DolphinScheduler’s “Decentralized” Design Rethinks the Master–Slave Playbook

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

A 10-Minute Deep Dive Into the Core Architecture of Apache SeaTunnel and DataX

How to Seamlessly Transfer From Airflow to Apache DolphinScheduler With Air2phin

A Comprehensive Guide to Building DolphinScheduler 3.2.0 Production-Grade Cluster Deployment

How to Regularly Delete Log Instances in DolphinScheduler

How to Enable Auto-Start for Apache DolphinScheduler

Solutions for Upgrading Apache DolphinScheduler from Version 1.3.4 to 3.1.2

A 10-Minute Deep Dive Into the Core Architecture of Apache SeaTunnel and DataX

How to Seamlessly Transfer From Airflow to Apache DolphinScheduler With Air2phin

A Comprehensive Guide to Building DolphinScheduler 3.2.0 Production-Grade Cluster Deployment

How to Regularly Delete Log Instances in DolphinScheduler

How to Enable Auto-Start for Apache DolphinScheduler

Solutions for Upgrading Apache DolphinScheduler from Version 1.3.4 to 3.1.2

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps