SeaTunnel provides a method to run the Zeta engine in cluster mode, allowing Kubernetes to locally deploy the Zeta engine for more efficient application deployment and management. In this guide, we will explore more about running the Zeta engine in cluster mode with SeaTunnel on Kubernetes and understand how to leverage the advantages of the Zeta engine better. Upload SeaTunnel to the server.Previously,
I have already decompressed and executed the install-plugin.sh script. Here, for convenience, I directly demonstrate using the SeaTunnel after executing the install-plugin.sh script. Upload SeaTunnel to the server.Previously,
I have already decompressed and executed the install-plugin.sh script. Here, for convenience, I directly demonstrate using the SeaTunnel after executing the install-plugin.sh script. Upload SeaTunnel to the server.Previously, Upload SeaTunnel to the server.Previously I have already decompressed and executed the install-plugin.sh script. Here, for convenience, I directly demonstrate using the SeaTunnel after executing the install-plugin.sh script. install-plugin.sh install-plugin.sh After executing the install-plugin, the lib directory contains the following: tar -zxvf apache-seatunnel-2.3.3-bin.tar.gz
sh apache-seatunnel-2.3.3/bin/install-plugin.sh
tar -czvf apache-seatunnel-2.3.3-bin.tar.gz apache-seatunnel-2.3.3 tar -zxvf apache-seatunnel-2.3.3-bin.tar.gz
sh apache-seatunnel-2.3.3/bin/install-plugin.sh
tar -czvf apache-seatunnel-2.3.3-bin.tar.gz apache-seatunnel-2.3.3 2.Build the SeaTunnel image. Build the SeaTunnel image. Create a Dockerfile in the same directory where SeaTunnel is installed. Configure as follows, you can choose the version yourself: FROM openjdk:8
ENV SEATUNNEL_HOME="/opt/seatunnel"
ENV SEATUNNEL_VERSION="2.3.3"
COPY /apache-seatunnel-${SEATUNNEL_VERSION}-bin.tar.gz /opt/apache-seatunnel-${SEATUNNEL_VERSION}-bin.tar.gz
WORKDIR /opt
RUN tar -xzvf apache-seatunnel-${SEATUNNEL_VERSION}-bin.tar.gz
RUN mv apache-seatunnel-${SEATUNNEL_VERSION} seatunnel
RUN rm -f /opt/apache-seatunnel-${SEATUNNEL_VERSION}-bin.tar.gz
WORKDIR /opt/seatunnel FROM openjdk:8
ENV SEATUNNEL_HOME="/opt/seatunnel"
ENV SEATUNNEL_VERSION="2.3.3"
COPY /apache-seatunnel-${SEATUNNEL_VERSION}-bin.tar.gz /opt/apache-seatunnel-${SEATUNNEL_VERSION}-bin.tar.gz
WORKDIR /opt
RUN tar -xzvf apache-seatunnel-${SEATUNNEL_VERSION}-bin.tar.gz
RUN mv apache-seatunnel-${SEATUNNEL_VERSION} seatunnel
RUN rm -f /opt/apache-seatunnel-${SEATUNNEL_VERSION}-bin.tar.gz
WORKDIR /opt/seatunnel Execute the command: docker build -t seatunnel:2.3.3 -f Dockerfile. docker build -t seatunnel:2.3.3 -f Dockerfile. 3. View the image: 3. View the image: docker images docker images The image appears as follows: Load the image into Kubernetes. Here I’m using Minikube for demonstration: Load the image into Kubernetes. Here I’m using Minikube for demonstration: Load the image into Kubernetes minikube image load seatunnel:2.3.3 minikube image load seatunnel:2.3.3 Refer to the above link for details: Mastering SeaTunnel Running Zeta Engine in local mode on Kubernetes: A Step-by-Step Guide Mastering SeaTunnel Running Zeta Engine in local mode on Kubernetes: A Step-by-Step Guide 5. Create config maps as follows: 5. Create config maps kubectl create configmap hazelcast-client --from-file=config/hazelcast-client.yaml
kubectl create configmap hazelcast --from-file=config/hazelcast.yaml
kubectl create configmap seatunnelmap --from-file=config/seatunnel.yaml kubectl create configmap hazelcast-client --from-file=config/hazelcast-client.yaml
kubectl create configmap hazelcast --from-file=config/hazelcast.yaml
kubectl create configmap seatunnelmap --from-file=config/seatunnel.yaml 6. Use Reloader to restart podsafter updating configmaps automatically: 6. Use Reloader to restart pods wget https://raw.githubusercontent.com/stakater/Reloader/master/deployments/kubernetes/reloader.yaml wget https://raw.githubusercontent.com/stakater/Reloader/master/deployments/kubernetes/reloader.yaml 7. Create seatunnel-cluster.ymlas follows: 7. Create seatunnel-cluster.yml apiVersion: v1
kind: Service
metadata:
  name: seatunnel
spec:
  selector:
    app: seatunnel
  ports:
  - port: 5801
    name: seatunnel
  clusterIP: None
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: seatunnel
  annotations:
    configmap.reloader.stakater.com/reload: "hazelcast,hazelcast-client,seatunnelmap"
spec:
  serviceName: "seatunnel"
  replicas: 3
  selector:
    matchLabels:
      app: seatunnel
  template:
    metadata:
      labels:
        app: seatunnel
    spec:
      containers:
        - name: seatunnel
          image: seatunnel:2.3.3
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 5801
              name: client
          command: ["/bin/sh","-c","/opt/seatunnel/bin/seatunnel-cluster.sh -DJvmOption=-Xms2G -Xmx2G"]
          resources:
            limits:
              cpu: "1"
              memory: 4G
            requests:
              cpu: "1"
              memory: 2G
          volumeMounts:
            - mountPath: "/opt/seatunnel/config/hazelcast.yaml"
              name: hazelcast
              subPath: hazelcast.yaml
            - mountPath: "/opt/seatunnel/config/hazelcast-client.yaml"
              name: hazelcast-client
              subPath: hazelcast-client.yaml
            - mountPath: "/opt/seatunnel/config/seatunnel.yaml"
              name: seatunnelmap
              subPath: seatunnel.yaml
      volumes:
        - name: hazelcast
          configMap:
            name: hazelcast
        - name: hazelcast-client
          configMap:
            name: hazelcast-client
        - name: seatunnelmap
          configMap:
            name: seatunnelmap apiVersion: v1
kind: Service
metadata:
  name: seatunnel
spec:
  selector:
    app: seatunnel
  ports:
  - port: 5801
    name: seatunnel
  clusterIP: None
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: seatunnel
  annotations:
    configmap.reloader.stakater.com/reload: "hazelcast,hazelcast-client,seatunnelmap"
spec:
  serviceName: "seatunnel"
  replicas: 3
  selector:
    matchLabels:
      app: seatunnel
  template:
    metadata:
      labels:
        app: seatunnel
    spec:
      containers:
        - name: seatunnel
          image: seatunnel:2.3.3
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 5801
              name: client
          command: ["/bin/sh","-c","/opt/seatunnel/bin/seatunnel-cluster.sh -DJvmOption=-Xms2G -Xmx2G"]
          resources:
            limits:
              cpu: "1"
              memory: 4G
            requests:
              cpu: "1"
              memory: 2G
          volumeMounts:
            - mountPath: "/opt/seatunnel/config/hazelcast.yaml"
              name: hazelcast
              subPath: hazelcast.yaml
            - mountPath: "/opt/seatunnel/config/hazelcast-client.yaml"
              name: hazelcast-client
              subPath: hazelcast-client.yaml
            - mountPath: "/opt/seatunnel/config/seatunnel.yaml"
              name: seatunnelmap
              subPath: seatunnel.yaml
      volumes:
        - name: hazelcast
          configMap:
            name: hazelcast
        - name: hazelcast-client
          configMap:
            name: hazelcast-client
        - name: seatunnelmap
          configMap:
            name: seatunnelmap 8. Execute: 8. Execute: kubectl apply -f seatunnel-cluster.yml kubectl apply -f seatunnel-cluster.yml 9. Modify the configuration in config maps: 9. Modify the configuration in config maps: kubectl edit cm hazelcast kubectl edit cm hazelcast Modify the cluster address.
Here, headless service access mode is used.
The general format for pod-to-pod communication is <pod-name>.<service-name>.<namespace>.svc.cluster.local Modify the cluster address.
Here, headless service access mode is used.
The general format for pod-to-pod communication is <pod-name>.<service-name>.<namespace>.svc.cluster.local Example: seatunnel-0.seatunnel.default.svc.cluster.local
seatunnel-1.seatunnel.default.svc.cluster.local
seatunnel-2.seatunnel.default.svc.cluster.local seatunnel-0.seatunnel.default.svc.cluster.local seatunnel-1.seatunnel.default.svc.cluster.local seatunnel-2.seatunnel.default.svc.cluster.local Reminder: Use spaces instead of tabs. Otherwise, it will cause errors. kubectl edit cm hazelcast-client kubectl edit cm hazelcast-client kubectl edit cm seatunnelmap kubectl edit cm seatunnelmap Modify it to your own HDFS address. 10. You can see the following result: 10. You can see the following result 11. After all nodes have finished updating and are running, you canenter the container to check if the paths have been modified: enter the container to check if the paths have been modified kubectl exec -it seatunnel-0 /bin/bash
cat config/hazelcast.yaml kubectl exec -it seatunnel-0 /bin/bash
cat config/hazelcast.yaml 12. Check the logs inside the container: 12. Check the logs inside the container: tail -200f logs/seatunnel-engine-server.log tail -200f logs/seatunnel-engine-server.log We can see that the cluster is running normally. 13. Run tasks: 13. Run tasks: You can open a new connection, log in to another pod node, and execute tasks to test the cluster: kubectl exec -it seatunnel-1 /bin/bash kubectl exec -it seatunnel-1 /bin/bash bin/seatunnel.sh --config config/v2.streaming.conf.template

We can see that tasks have also started running in other pods. bin/seatunnel.sh --config config/v2.streaming.conf.template

We can see that tasks have also started running in other pods. That’s all for running the Zeta engine in cluster mode with SeaTunnel on Kubernetes!

The code in this story is for educational purposes. The readers are solely responsible for whatever they build with it.

Ever Wanted to Run a Data Engine on Kubernetes? SeaTunnel Just Made It Weirdly Simple

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

AI’s Data Bottleneck Is Getting Worse—Here’s a New Way to Fix It

102 Stories To Learn About Open Source Software

10 Top Open Source AI Technologies For Startups

10 Reasons You Should Quit Your HTTP Client

About Ten Open Source Technologies That Are Changing the World

10 Open Source AI Project Ideas For Startups

AI’s Data Bottleneck Is Getting Worse—Here’s a New Way to Fix It

102 Stories To Learn About Open Source Software

10 Top Open Source AI Technologies For Startups

10 Reasons You Should Quit Your HTTP Client

About Ten Open Source Technologies That Are Changing the World

10 Open Source AI Project Ideas For Startups

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps