SeaTunnel provides a method to run the Zeta engine in cluster mode, allowing Kubernetes to locally deploy the Zeta engine for more efficient application deployment and management. In this guide, we will explore more about running the Zeta engine in cluster mode with SeaTunnel on Kubernetes and understand how to leverage the advantages of the Zeta engine better. Upload SeaTunnel to the server.Previously, I have already decompressed and executed the install-plugin.sh script. Here, for convenience, I directly demonstrate using the SeaTunnel after executing the install-plugin.sh script. Upload SeaTunnel to the server.Previously, I have already decompressed and executed the install-plugin.sh script. Here, for convenience, I directly demonstrate using the SeaTunnel after executing the install-plugin.sh script. Upload SeaTunnel to the server.Previously, Upload SeaTunnel to the server.Previously I have already decompressed and executed the install-plugin.sh script. Here, for convenience, I directly demonstrate using the SeaTunnel after executing the install-plugin.sh script. install-plugin.sh install-plugin.sh After executing the install-plugin, the lib directory contains the following: tar -zxvf apache-seatunnel-2.3.3-bin.tar.gz sh apache-seatunnel-2.3.3/bin/install-plugin.sh tar -czvf apache-seatunnel-2.3.3-bin.tar.gz apache-seatunnel-2.3.3 tar -zxvf apache-seatunnel-2.3.3-bin.tar.gz sh apache-seatunnel-2.3.3/bin/install-plugin.sh tar -czvf apache-seatunnel-2.3.3-bin.tar.gz apache-seatunnel-2.3.3 2.Build the SeaTunnel image. Build the SeaTunnel image. Create a Dockerfile in the same directory where SeaTunnel is installed. Configure as follows, you can choose the version yourself: FROM openjdk:8 ENV SEATUNNEL_HOME="/opt/seatunnel" ENV SEATUNNEL_VERSION="2.3.3" COPY /apache-seatunnel-${SEATUNNEL_VERSION}-bin.tar.gz /opt/apache-seatunnel-${SEATUNNEL_VERSION}-bin.tar.gz WORKDIR /opt RUN tar -xzvf apache-seatunnel-${SEATUNNEL_VERSION}-bin.tar.gz RUN mv apache-seatunnel-${SEATUNNEL_VERSION} seatunnel RUN rm -f /opt/apache-seatunnel-${SEATUNNEL_VERSION}-bin.tar.gz WORKDIR /opt/seatunnel FROM openjdk:8 ENV SEATUNNEL_HOME="/opt/seatunnel" ENV SEATUNNEL_VERSION="2.3.3" COPY /apache-seatunnel-${SEATUNNEL_VERSION}-bin.tar.gz /opt/apache-seatunnel-${SEATUNNEL_VERSION}-bin.tar.gz WORKDIR /opt RUN tar -xzvf apache-seatunnel-${SEATUNNEL_VERSION}-bin.tar.gz RUN mv apache-seatunnel-${SEATUNNEL_VERSION} seatunnel RUN rm -f /opt/apache-seatunnel-${SEATUNNEL_VERSION}-bin.tar.gz WORKDIR /opt/seatunnel Execute the command: docker build -t seatunnel:2.3.3 -f Dockerfile. docker build -t seatunnel:2.3.3 -f Dockerfile. 3. View the image: 3. View the image: docker images docker images The image appears as follows: Load the image into Kubernetes. Here I’m using Minikube for demonstration: Load the image into Kubernetes. Here I’m using Minikube for demonstration: Load the image into Kubernetes minikube image load seatunnel:2.3.3 minikube image load seatunnel:2.3.3 Refer to the above link for details: Mastering SeaTunnel Running Zeta Engine in local mode on Kubernetes: A Step-by-Step Guide Mastering SeaTunnel Running Zeta Engine in local mode on Kubernetes: A Step-by-Step Guide 5. Create config maps as follows: 5. Create config maps kubectl create configmap hazelcast-client --from-file=config/hazelcast-client.yaml kubectl create configmap hazelcast --from-file=config/hazelcast.yaml kubectl create configmap seatunnelmap --from-file=config/seatunnel.yaml kubectl create configmap hazelcast-client --from-file=config/hazelcast-client.yaml kubectl create configmap hazelcast --from-file=config/hazelcast.yaml kubectl create configmap seatunnelmap --from-file=config/seatunnel.yaml 6. Use Reloader to restart podsafter updating configmaps automatically: 6. Use Reloader to restart pods wget https://raw.githubusercontent.com/stakater/Reloader/master/deployments/kubernetes/reloader.yaml wget https://raw.githubusercontent.com/stakater/Reloader/master/deployments/kubernetes/reloader.yaml 7. Create seatunnel-cluster.ymlas follows: 7. Create seatunnel-cluster.yml apiVersion: v1 kind: Service metadata: name: seatunnel spec: selector: app: seatunnel ports: - port: 5801 name: seatunnel clusterIP: None --- apiVersion: apps/v1 kind: StatefulSet metadata: name: seatunnel annotations: configmap.reloader.stakater.com/reload: "hazelcast,hazelcast-client,seatunnelmap" spec: serviceName: "seatunnel" replicas: 3 selector: matchLabels: app: seatunnel template: metadata: labels: app: seatunnel spec: containers: - name: seatunnel image: seatunnel:2.3.3 imagePullPolicy: IfNotPresent ports: - containerPort: 5801 name: client command: ["/bin/sh","-c","/opt/seatunnel/bin/seatunnel-cluster.sh -DJvmOption=-Xms2G -Xmx2G"] resources: limits: cpu: "1" memory: 4G requests: cpu: "1" memory: 2G volumeMounts: - mountPath: "/opt/seatunnel/config/hazelcast.yaml" name: hazelcast subPath: hazelcast.yaml - mountPath: "/opt/seatunnel/config/hazelcast-client.yaml" name: hazelcast-client subPath: hazelcast-client.yaml - mountPath: "/opt/seatunnel/config/seatunnel.yaml" name: seatunnelmap subPath: seatunnel.yaml volumes: - name: hazelcast configMap: name: hazelcast - name: hazelcast-client configMap: name: hazelcast-client - name: seatunnelmap configMap: name: seatunnelmap apiVersion: v1 kind: Service metadata: name: seatunnel spec: selector: app: seatunnel ports: - port: 5801 name: seatunnel clusterIP: None --- apiVersion: apps/v1 kind: StatefulSet metadata: name: seatunnel annotations: configmap.reloader.stakater.com/reload: "hazelcast,hazelcast-client,seatunnelmap" spec: serviceName: "seatunnel" replicas: 3 selector: matchLabels: app: seatunnel template: metadata: labels: app: seatunnel spec: containers: - name: seatunnel image: seatunnel:2.3.3 imagePullPolicy: IfNotPresent ports: - containerPort: 5801 name: client command: ["/bin/sh","-c","/opt/seatunnel/bin/seatunnel-cluster.sh -DJvmOption=-Xms2G -Xmx2G"] resources: limits: cpu: "1" memory: 4G requests: cpu: "1" memory: 2G volumeMounts: - mountPath: "/opt/seatunnel/config/hazelcast.yaml" name: hazelcast subPath: hazelcast.yaml - mountPath: "/opt/seatunnel/config/hazelcast-client.yaml" name: hazelcast-client subPath: hazelcast-client.yaml - mountPath: "/opt/seatunnel/config/seatunnel.yaml" name: seatunnelmap subPath: seatunnel.yaml volumes: - name: hazelcast configMap: name: hazelcast - name: hazelcast-client configMap: name: hazelcast-client - name: seatunnelmap configMap: name: seatunnelmap 8. Execute: 8. Execute: kubectl apply -f seatunnel-cluster.yml kubectl apply -f seatunnel-cluster.yml 9. Modify the configuration in config maps: 9. Modify the configuration in config maps: kubectl edit cm hazelcast kubectl edit cm hazelcast Modify the cluster address. Here, headless service access mode is used. The general format for pod-to-pod communication is <pod-name>.<service-name>.<namespace>.svc.cluster.local Modify the cluster address. Here, headless service access mode is used. The general format for pod-to-pod communication is <pod-name>.<service-name>.<namespace>.svc.cluster.local Example: seatunnel-0.seatunnel.default.svc.cluster.local seatunnel-1.seatunnel.default.svc.cluster.local seatunnel-2.seatunnel.default.svc.cluster.local seatunnel-0.seatunnel.default.svc.cluster.local seatunnel-1.seatunnel.default.svc.cluster.local seatunnel-2.seatunnel.default.svc.cluster.local Reminder: Use spaces instead of tabs. Otherwise, it will cause errors. kubectl edit cm hazelcast-client kubectl edit cm hazelcast-client kubectl edit cm seatunnelmap kubectl edit cm seatunnelmap Modify it to your own HDFS address. 10. You can see the following result: 10. You can see the following result 11. After all nodes have finished updating and are running, you canenter the container to check if the paths have been modified: enter the container to check if the paths have been modified kubectl exec -it seatunnel-0 /bin/bash cat config/hazelcast.yaml kubectl exec -it seatunnel-0 /bin/bash cat config/hazelcast.yaml 12. Check the logs inside the container: 12. Check the logs inside the container: tail -200f logs/seatunnel-engine-server.log tail -200f logs/seatunnel-engine-server.log We can see that the cluster is running normally. 13. Run tasks: 13. Run tasks: You can open a new connection, log in to another pod node, and execute tasks to test the cluster: kubectl exec -it seatunnel-1 /bin/bash kubectl exec -it seatunnel-1 /bin/bash bin/seatunnel.sh --config config/v2.streaming.conf.template We can see that tasks have also started running in other pods. bin/seatunnel.sh --config config/v2.streaming.conf.template We can see that tasks have also started running in other pods. That’s all for running the Zeta engine in cluster mode with SeaTunnel on Kubernetes!