Cliently

cliently

SeaTunnel provides a method to run the Zeta engine in cluster mode, allowing Kubernetes to locally deploy the Zeta engine for more efficient application deployment and management. In this guide, we will explore more about running the Zeta engine in cluster mode with SeaTunnel on Kubernetes and understand how to leverage the advantages of the Zeta engine better. Upload SeaTunnel to the server.Previously,
I have already decompressed and executed the install-plugin.sh script. Here, for convenience, I directly demonstrate using the SeaTunnel after executing the install-plugin.sh script. Upload SeaTunnel to the server.Previously,
I have already decompressed and executed the install-plugin.sh script. Here, for convenience, I directly demonstrate using the SeaTunnel after executing the install-plugin.sh script. Upload SeaTunnel to the server.Previously, Upload SeaTunnel to the server.Previously I have already decompressed and executed the install-plugin.sh script. Here, for convenience, I directly demonstrate using the SeaTunnel after executing the install-plugin.sh script. install-plugin.sh install-plugin.sh After executing the install-plugin, the lib directory contains the following: tar -zxvf apache-seatunnel-2.3.3-bin.tar.gz
sh apache-seatunnel-2.3.3/bin/install-plugin.sh
tar -czvf apache-seatunnel-2.3.3-bin.tar.gz apache-seatunnel-2.3.3 tar -zxvf apache-seatunnel-2.3.3-bin.tar.gz
sh apache-seatunnel-2.3.3/bin/install-plugin.sh
tar -czvf apache-seatunnel-2.3.3-bin.tar.gz apache-seatunnel-2.3.3 2.Build the SeaTunnel image. Build the SeaTunnel image. Create a Dockerfile in the same directory where SeaTunnel is installed. Configure as follows, you can choose the version yourself: FROM openjdk:8
ENV SEATUNNEL_HOME="/opt/seatunnel"
ENV SEATUNNEL_VERSION="2.3.3"
COPY /apache-seatunnel-${SEATUNNEL_VERSION}-bin.tar.gz /opt/apache-seatunnel-${SEATUNNEL_VERSION}-bin.tar.gz
WORKDIR /opt
RUN tar -xzvf apache-seatunnel-${SEATUNNEL_VERSION}-bin.tar.gz
RUN mv apache-seatunnel-${SEATUNNEL_VERSION} seatunnel
RUN rm -f /opt/apache-seatunnel-${SEATUNNEL_VERSION}-bin.tar.gz
WORKDIR /opt/seatunnel FROM openjdk:8
ENV SEATUNNEL_HOME="/opt/seatunnel"
ENV SEATUNNEL_VERSION="2.3.3"
COPY /apache-seatunnel-${SEATUNNEL_VERSION}-bin.tar.gz /opt/apache-seatunnel-${SEATUNNEL_VERSION}-bin.tar.gz
WORKDIR /opt
RUN tar -xzvf apache-seatunnel-${SEATUNNEL_VERSION}-bin.tar.gz
RUN mv apache-seatunnel-${SEATUNNEL_VERSION} seatunnel
RUN rm -f /opt/apache-seatunnel-${SEATUNNEL_VERSION}-bin.tar.gz
WORKDIR /opt/seatunnel Execute the command: docker build -t seatunnel:2.3.3 -f Dockerfile. docker build -t seatunnel:2.3.3 -f Dockerfile. 3. View the image: 3. View the image: docker images docker images The image appears as follows: Load the image into Kubernetes. Here I’m using Minikube for demonstration: Load the image into Kubernetes. Here I’m using Minikube for demonstration: Load the image into Kubernetes minikube image load seatunnel:2.3.3 minikube image load seatunnel:2.3.3 Refer to the above link for details: Mastering SeaTunnel Running Zeta Engine in local mode on Kubernetes: A Step-by-Step Guide Mastering SeaTunnel Running Zeta Engine in local mode on Kubernetes: A Step-by-Step Guide 5. Create config maps as follows: 5. Create config maps kubectl create configmap hazelcast-client --from-file=config/hazelcast-client.yaml
kubectl create configmap hazelcast --from-file=config/hazelcast.yaml
kubectl create configmap seatunnelmap --from-file=config/seatunnel.yaml kubectl create configmap hazelcast-client --from-file=config/hazelcast-client.yaml
kubectl create configmap hazelcast --from-file=config/hazelcast.yaml
kubectl create configmap seatunnelmap --from-file=config/seatunnel.yaml 6. Use Reloader to restart podsafter updating configmaps automatically: 6. Use Reloader to restart pods wget https://raw.githubusercontent.com/stakater/Reloader/master/deployments/kubernetes/reloader.yaml wget https://raw.githubusercontent.com/stakater/Reloader/master/deployments/kubernetes/reloader.yaml 7. Create seatunnel-cluster.ymlas follows: 7. Create seatunnel-cluster.yml apiVersion: v1
kind: Service
metadata:
  name: seatunnel
spec:
  selector:
    app: seatunnel
  ports:
  - port: 5801
    name: seatunnel
  clusterIP: None
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: seatunnel
  annotations:
    configmap.reloader.stakater.com/reload: "hazelcast,hazelcast-client,seatunnelmap"
spec:
  serviceName: "seatunnel"
  replicas: 3
  selector:
    matchLabels:
      app: seatunnel
  template:
    metadata:
      labels:
        app: seatunnel
    spec:
      containers:
        - name: seatunnel
          image: seatunnel:2.3.3
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 5801
              name: client
          command: ["/bin/sh","-c","/opt/seatunnel/bin/seatunnel-cluster.sh -DJvmOption=-Xms2G -Xmx2G"]
          resources:
            limits:
              cpu: "1"
              memory: 4G
            requests:
              cpu: "1"
              memory: 2G
          volumeMounts:
            - mountPath: "/opt/seatunnel/config/hazelcast.yaml"
              name: hazelcast
              subPath: hazelcast.yaml
            - mountPath: "/opt/seatunnel/config/hazelcast-client.yaml"
              name: hazelcast-client
              subPath: hazelcast-client.yaml
            - mountPath: "/opt/seatunnel/config/seatunnel.yaml"
              name: seatunnelmap
              subPath: seatunnel.yaml
      volumes:
        - name: hazelcast
          configMap:
            name: hazelcast
        - name: hazelcast-client
          configMap:
            name: hazelcast-client
        - name: seatunnelmap
          configMap:
            name: seatunnelmap apiVersion: v1
kind: Service
metadata:
  name: seatunnel
spec:
  selector:
    app: seatunnel
  ports:
  - port: 5801
    name: seatunnel
  clusterIP: None
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: seatunnel
  annotations:
    configmap.reloader.stakater.com/reload: "hazelcast,hazelcast-client,seatunnelmap"
spec:
  serviceName: "seatunnel"
  replicas: 3
  selector:
    matchLabels:
      app: seatunnel
  template:
    metadata:
      labels:
        app: seatunnel
    spec:
      containers:
        - name: seatunnel
          image: seatunnel:2.3.3
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 5801
              name: client
          command: ["/bin/sh","-c","/opt/seatunnel/bin/seatunnel-cluster.sh -DJvmOption=-Xms2G -Xmx2G"]
          resources:
            limits:
              cpu: "1"
              memory: 4G
            requests:
              cpu: "1"
              memory: 2G
          volumeMounts:
            - mountPath: "/opt/seatunnel/config/hazelcast.yaml"
              name: hazelcast
              subPath: hazelcast.yaml
            - mountPath: "/opt/seatunnel/config/hazelcast-client.yaml"
              name: hazelcast-client
              subPath: hazelcast-client.yaml
            - mountPath: "/opt/seatunnel/config/seatunnel.yaml"
              name: seatunnelmap
              subPath: seatunnel.yaml
      volumes:
        - name: hazelcast
          configMap:
            name: hazelcast
        - name: hazelcast-client
          configMap:
            name: hazelcast-client
        - name: seatunnelmap
          configMap:
            name: seatunnelmap 8. Execute: 8. Execute: kubectl apply -f seatunnel-cluster.yml kubectl apply -f seatunnel-cluster.yml 9. Modify the configuration in config maps: 9. Modify the configuration in config maps: kubectl edit cm hazelcast kubectl edit cm hazelcast Modify the cluster address.
Here, headless service access mode is used.
The general format for pod-to-pod communication is <pod-name>.<service-name>.<namespace>.svc.cluster.local Modify the cluster address.
Here, headless service access mode is used.
The general format for pod-to-pod communication is <pod-name>.<service-name>.<namespace>.svc.cluster.local Example: seatunnel-0.seatunnel.default.svc.cluster.local
seatunnel-1.seatunnel.default.svc.cluster.local
seatunnel-2.seatunnel.default.svc.cluster.local seatunnel-0.seatunnel.default.svc.cluster.local seatunnel-1.seatunnel.default.svc.cluster.local seatunnel-2.seatunnel.default.svc.cluster.local Reminder: Use spaces instead of tabs. Otherwise, it will cause errors. kubectl edit cm hazelcast-client kubectl edit cm hazelcast-client kubectl edit cm seatunnelmap kubectl edit cm seatunnelmap Modify it to your own HDFS address. 10. You can see the following result: 10. You can see the following result 11. After all nodes have finished updating and are running, you canenter the container to check if the paths have been modified: enter the container to check if the paths have been modified kubectl exec -it seatunnel-0 /bin/bash
cat config/hazelcast.yaml kubectl exec -it seatunnel-0 /bin/bash
cat config/hazelcast.yaml 12. Check the logs inside the container: 12. Check the logs inside the container: tail -200f logs/seatunnel-engine-server.log tail -200f logs/seatunnel-engine-server.log We can see that the cluster is running normally. 13. Run tasks: 13. Run tasks: You can open a new connection, log in to another pod node, and execute tasks to test the cluster: kubectl exec -it seatunnel-1 /bin/bash kubectl exec -it seatunnel-1 /bin/bash bin/seatunnel.sh --config config/v2.streaming.conf.template

We can see that tasks have also started running in other pods. bin/seatunnel.sh --config config/v2.streaming.conf.template

We can see that tasks have also started running in other pods. That’s all for running the Zeta engine in cluster mode with SeaTunnel on Kubernetes!

Ever Wanted to Run a Data Engine on Kubernetes? SeaTunnel Just Made It Weirdly Simple

Model Context Protocol (MCP): The USB-C of AI Data Connectivity

How to Simplify Multi-LLM Integration With KubeMQ: The Path to Scalable AI Solutions

How to Build a Scalable Data Mesh in AWS with Lake Formation

A DevOps Approach to AEM Packages: Automating Creation, Configuration, and More

'n DevOps-benadering tot AEM-pakkette: outomatisering van skepping, konfigurasie en meer

En DevOps-tilgang til AEM-pakker: Automatisering af oprettelse, konfiguration og mere

Ein DevOps-Ansatz für AEM-Pakete: Automatisieren der Erstellung, Konfiguration und mehr

Μια προσέγγιση DevOps στα πακέτα AEM: Αυτοματοποίηση δημιουργίας, διαμόρφωσης και άλλα

Un enfoque DevOps para los paquetes AEM: automatización de la creación, configuración y más

AEM パッケージへの DevOps アプローチ: 作成、構成などの自動化

AEM 패키지에 대한 DevOps 접근 방식: 생성, 구성 등 자동화

„DevOps“ požiūris į AEM paketus: kūrimo, konfigūravimo ir kt. automatizavimas

AEM багцуудад зориулсан DevOps арга: Үүсгэх, тохируулах болон бусад зүйлсийг автоматжуулах

ДевОпс приступ АЕМ пакетима: аутоматизација креирања, конфигурисања и још много тога

Isang DevOps Approach sa AEM Packages: Automating Creation, Configuration, at Higit Pa

Indlela ye-DevOps kumaPhakheji e-AEM: Ukuzidalela Okuzenzakalelayo, Ukumisa, Nokunye

Why Mamba Could Be the Future of Big Data Processing

The One Tool You Absolutely Need to Efficiently Scale Retrieval-Augmented Generation

Choosing the Right Caching Strategy

gRPC-Secret: Mastering Deadlines, Timeouts, and Custom Contexts

gRPC-সিক্রেট: সময়সীমা, সময়সীমা, এবং কাস্টম প্রসঙ্গ আয়ত্ত করা

gRPC-Geheimnis: Deadlines, Timeouts und benutzerdefinierte Kontexte meistern

gRPC-Secret: Dominar los plazos, los tiempos de espera y los contextos personalizados

gRPC-Secret : maîtriser les délais, les délais d'attente et les contextes personnalisés

gRPC-सीक्रेट: डेडलाइन, टाइमआउट और कस्टम संदर्भों पर महारत हासिल करना

gRPC-Secret: 期限、タイムアウト、カスタムコンテキストをマスターする

gRPC 비밀: 기한, 시간 초과, 사용자 정의 컨텍스트 마스터하기

gRPC-Secret: dominando prazos, tempos limite e contextos personalizados

gRPC-Secret: освоение сроков, тайм-аутов и пользовательских контекстов

gRPC-Secret: Son Teslim Tarihleri, Zaman Aşımları ve Özel Bağlamlarda Uzmanlaşma

gRPC-Secret: Nắm vững thời hạn, thời gian chờ và bối cảnh tùy chỉnh

gRPC 秘密：掌握截止期限、超时和自定义上下文

Effortlessly Launch LangChain APIs with LangServe and MinIO Integration

অনায়াসে LangServe এবং MinIO ইন্টিগ্রেশন সহ LangChain API চালু করুন

Müheloses Starten von LangChain-APIs mit LangServe- und MinIO-Integration

Lance las API de LangChain sin esfuerzo con la integración de LangServe y MinIO

Lancez sans effort les API LangChain avec l'intégration LangServe et MinIO

LangServe और MinIO एकीकरण के साथ LangChain API को आसानी से लॉन्च करें

LangServe と MinIO の統合により LangChain API を簡単に起動

LangServe 및 MinIO 통합을 통해 LangChain API를 손쉽게 실행

Inicie APIs LangChain sem esforço com integração LangServe e MinIO

Легко запускайте API-интерфейсы LangChain с интеграцией LangServe и MinIO.

LangServe ve MinIO Entegrasyonu ile LangChain API'lerini Zahmetsizce Başlatın

Dễ dàng khởi chạy API LangChain với tích hợp LangServe và MinIO

通过 LangServe 和 MinIO 集成轻松启动 LangChain API

Stories, News & Mentions about CLIENTLY

Categories

Trending Topics

Stories, News & Mentions about CLIENTLY

Cliently’s Stories on HackerNoon

Categories

Trending Topics