Cluster Configuration Issues Since April, there have been three occurrences of cluster split-brain phenomena, all involving a specific node experiencing split-brain or automatically shutting down. Master Node The following core logs were observed: Slow Operation logs printed by the Hazelcast monitoring thread. Hazelcast heartbeat timeout of 60s showed node 198 leaving the cluster. Node 198 (Worker) No heartbeat detected from Hazelcast cluster nodes, with a timeout exceeding 60000ms. Attempt to reconnect to the cluster Attempt to reconnect to the cluster Status queries and job submission requests to the node were unresponsive, leading to a cluster-wide deadlock. Our node health check interfaces were also unresponsive. This resulted in service unavailability during peak hours, prompting us to quickly restart the cluster after detecting split-brain issues. After parameter adjustments, nodes sometimes automatically shut down. Problem Analysis The potential causes for Hazelcast cluster split-brain issues are as follows: Inconsistent NTP across ECS systems in the cluster.
Network jitter in the ECS where the cluster is located, leading to inaccessible services.
Full GC in SeaTunnel causing JVM stalling, leading to cluster formation failure. Inconsistent NTP across ECS systems in the cluster. Network jitter in the ECS where the cluster is located, leading to inaccessible services. Full GC in SeaTunnel causing JVM stalling, leading to cluster formation failure. We verified with the operations team that there were no network issues, and the Alibaba Cloud NTP service confirmed that all three servers were synchronized. The third issue was investigated by checking the last Hazelcast health check logs on node 198 before the anomaly. The cluster time was -100 milliseconds, which seemed negligible. We then added JVM GC log parameters to monitor full GC times, which reached up to 27s, easily leading to Hazelcast’s 60s heartbeat timeout. Additionally, a specific CK table with 1.4 billion records frequently triggered FullGC after running for a certain period. Solutions Increase ST Cluster Heartbeat Interval Hazelcast’s cluster failure detector identifies members that are unreachable or crashed. However, according to the famous FLP result, it’s impossible to distinguish between a crashed member and a slow member in asynchronous systems. The solution is to use an unreliable failure detector. Hazelcast has built-in detectors: Deadline Failure Detector and Phi Accrual Failure Detector. FLP result Deadline Failure Detector Phi Accrual Failure Detector By default, Hazelcast uses the Deadline Failure Detector. Deadline Failure Detector Deadline Failure Detector Uses absolute timeout for missed heartbeats. Members are considered unreachable after the timeout. Configuration: hazelcast:
  properties:
    hazelcast.heartbeat.failuredetector.type: deadline
    hazelcast.heartbeat.interval.seconds: 5
    hazelcast.max.no.heartbeat.seconds: 120 hazelcast:
  properties:
    hazelcast.heartbeat.failuredetector.type: deadline
    hazelcast.heartbeat.interval.seconds: 5
    hazelcast.max.no.heartbeat.seconds: 120 Phi-accrual Failure Detector Phi-accrual Failure Detector Tracks the intervals between heartbeats in a sliding time window, calculates suspicion level (Phi) based on these samples. Configuration: hazelcast:
  properties:
    hazelcast.heartbeat.failuredetector.type: phi-accrual
    hazelcast.heartbeat.interval.seconds: 1
    hazelcast.max.no.heartbeat.seconds: 60
    hazelcast.heartbeat.phiaccrual.failuredetector.threshold: 10
    hazelcast.heartbeat.phiaccrual.failuredetector.sample.size: 200
    hazelcast.heartbeat.phiaccrual.failuredetector.min.std.dev.millis: 100 hazelcast:
  properties:
    hazelcast.heartbeat.failuredetector.type: phi-accrual
    hazelcast.heartbeat.interval.seconds: 1
    hazelcast.max.no.heartbeat.seconds: 60
    hazelcast.heartbeat.phiaccrual.failuredetector.threshold: 10
    hazelcast.heartbeat.phiaccrual.failuredetector.sample.size: 200
    hazelcast.heartbeat.phiaccrual.failuredetector.min.std.dev.millis: 100 References: Hazelcast System Properties
Hazelcast Failure Detector Configuration
Hazelcast Phi Accrual Detector Hazelcast System Properties Hazelcast System Properties Hazelcast System Properties Hazelcast Failure Detector Configuration Hazelcast Failure Detector Configuration Hazelcast Failure Detector Configuration Hazelcast Phi Accrual Detector Hazelcast Phi Accrual Detector Hazelcast Phi Accrual Detector To ensure accuracy, we used the community-recommended phi-accrual detector with a 180s timeout in hazelcast.yml: phi-accrual hazelcast.yml hazelcast:
  properties:
    hazelcast.heartbeat.failuredetector.type: phi-accrual
    hazelcast.heartbeat.interval.seconds: 1
    hazelcast.max.no.heartbeat.seconds: 180
    hazelcast.heartbeat.phiaccrual.failuredetector.threshold: 10
    hazelcast.heartbeat.phiaccrual.failuredetector.sample.size: 200
    hazelcast.heartbeat.phiaccrual.failuredetector.min.std.dev.millis: 100 hazelcast:
  properties:
    hazelcast.heartbeat.failuredetector.type: phi-accrual
    hazelcast.heartbeat.interval.seconds: 1
    hazelcast.max.no.heartbeat.seconds: 180
    hazelcast.heartbeat.phiaccrual.failuredetector.threshold: 10
    hazelcast.heartbeat.phiaccrual.failuredetector.sample.size: 200
    hazelcast.heartbeat.phiaccrual.failuredetector.min.std.dev.millis: 100 Optimize GC Settings SeaTunnel uses the G1 garbage collector by default. Larger memory settings can cause frequent FullGC if YoungGC/MixedGC does not reclaim enough resources. Our goal is to maximize the memory reclaimed by YoungGC/MixedGC threads. Initial settings: -Xms32g
-Xmx32g
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/tmp/seatunnel/dump/zeta-server
-XX:MaxMetaspaceSize=8g
-XX:+UseG1GC
-XX:+PrintGCDetails
-Xloggc:/alidata1/za-seatunnel/logs/gc.log
-XX:+PrintGCDateStamps -Xms32g
-Xmx32g
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/tmp/seatunnel/dump/zeta-server
-XX:MaxMetaspaceSize=8g
-XX:+UseG1GC
-XX:+PrintGCDetails
-Xloggc:/alidata1/za-seatunnel/logs/gc.log
-XX:+PrintGCDateStamps We increased the GC pause time: -XX:MaxGCPauseMillis=5000 -XX:MaxGCPauseMillis=5000 First optimization attempt: -Xms32g
-Xmx32g
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/tmp/seatunnel/dump/zeta-server
-XX:MaxMetaspaceSize=8g
-XX:+UseG1GC
-XX:+PrintGCDetails
-Xloggc:/alidata1/za-seatunnel/logs/gc.log
-XX:+PrintGCDateStamps
-XX:MaxGCPauseMillis=5000 -Xms32g
-Xmx32g
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/tmp/seatunnel/dump/zeta-server
-XX:MaxMetaspaceSize=8g
-XX:+UseG1GC
-XX:+PrintGCDetails
-Xloggc:/alidata1/za-seatunnel/logs/gc.log
-XX:+PrintGCDateStamps
-XX:MaxGCPauseMillis=5000 MixedGC logs showed that the pause times were within expectations but did not eliminate FullGC, which lasted around 20s. To further optimize, we increased the old generation memory and adjusted G1 garbage collector parameters: -Xms42g
-Xmx42g
-XX:GCTimeRatio=4
-XX:G1ReservePercent=15
-XX:G1HeapRegionSize=32M -Xms42g
-Xmx42g
-XX:GCTimeRatio=4
-XX:G1ReservePercent=15
-XX:G1HeapRegionSize=32M Heap memory was increased from 32G to 42G, effectively increasing the old generation limit, which should reduce FullGC frequency. Second optimization attempt -Xms42g
-Xmx42g
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/tmp/seatunnel/dump/zeta-server
-XX:MaxMetaspaceSize=8g
-XX:+UseG1GC
-XX:+PrintGCDetails
-Xloggc:/alidata1/za-seatunnel/logs/gc.log
-XX:+PrintGCDateStamps
-XX:MaxGCPauseMillis=5000
-XX:GCTimeRatio=4
-XX:G1ReservePercent=15
-XX:G1HeapRegionSize=32M -Xms42g
-Xmx42g
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/tmp/seatunnel/dump/zeta-server
-XX:MaxMetaspaceSize=8g
-XX:+UseG1GC
-XX:+PrintGCDetails
-Xloggc:/alidata1/za-seatunnel/logs/gc.log
-XX:+PrintGCDateStamps
-XX:MaxGCPauseMillis=5000
-XX:GCTimeRatio=4
-XX:G1ReservePercent=15
-XX:G1HeapRegionSize=32M While the number of FullGCs decreased, they did not disappear entirely. We noticed that old generation data was not being properly cleaned during MixedGC. Final optimization attempt: -Xms42g
-Xmx42g
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/tmp/seatunnel/dump/zeta-server
-XX:MaxMetaspaceSize=8g
-XX:+UseG1GC
-XX:+PrintGCDetails
-Xloggc:/alidata1/za-seatunnel/logs/gc.log
-XX:+PrintGCDateStamps
-XX:MaxGCPauseMillis=5000
-XX:InitiatingHeapOccupancyPercent=50
-XX:+UseStringDeduplication
-XX:GCTimeRatio=4
-XX:G1ReservePercent=15
-XX:ConcGCThreads=12
-XX:G1HeapRegionSize=32M -Xms42g
-Xmx42g
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/tmp/seatunnel/dump/zeta-server
-XX:MaxMetaspaceSize=8g
-XX:+UseG1GC
-XX:+PrintGCDetails
-Xloggc:/alidata1/za-seatunnel/logs/gc.log
-XX:+PrintGCDateStamps
-XX:MaxGCPauseMillis=5000
-XX:InitiatingHeapOccupancyPercent=50
-XX:+UseStringDeduplication
-XX:GCTimeRatio=4
-XX:G1ReservePercent=15
-XX:ConcGCThreads=12
-XX:G1HeapRegionSize=32M References: G1 GC Tuning
Pause Time Goal
G1 GC Tuning Guide
JVM GC Tuning G1 GC Tuning G1 GC Tuning G1 GC Tuning Pause Time Goal Pause Time Goal Pause Time Goal G1 GC Tuning Guide G1 GC Tuning Guide G1 GC Tuning Guide JVM GC Tuning JVM GC Tuning JVM GC Tuning Optimization Results Optimization Results Since the configuration changes on April 26, there have been no further split-brain issues, and service availability has been restored. Since JVM parameter adjustments on April 30, we achieved zero FullGCs during the May Day holiday, with no health check interface issues. While this optimization may have slightly impacted application thread throughput, it ensured cluster stability, enabling widespread internal deployment of Zeta.

This One Data Company Abandoned DataX, Upgraded to SeaTunnel, and Turbocharged Its Workflows

Handling JSON, Schema Issues, and Doris Quirks in an Apache SeaTunnel Pipeline

Read My Stories

Too Long; Didn't Read

Make resilience your competitive advantage

Fixing SeaTunnel Outages With JVM Voodoo and Hazelcast Tweaks

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

SeaTunnel + Bedrock + OpenSearch = AI That Gets What You’re Saying

8 Essential Maven Plugins Beyond The Core

A Comprehensive Guide to Apache Cassandra Architecture

A Detailed Guide To Using Apache Storm

ACID Transactions Are Coming To Apache Cassandra: Here's Why We're Excited

SeaTunnel + Bedrock + OpenSearch = AI That Gets What You’re Saying

8 Essential Maven Plugins Beyond The Core

A Comprehensive Guide to Apache Cassandra Architecture

A Detailed Guide To Using Apache Storm

ACID Transactions Are Coming To Apache Cassandra: Here's Why We're Excited

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps