SeaTunnel + Bedrock + OpenSearch = AI That Gets What You’re Saying

Business Background As enterprises embrace digital transformation, information retrieval is evolving from simple keyword matching to more advanced semantic understanding. Traditional search engines that rely on inverted indexes struggle to grasp the real meaning behind user queries. keyword matching semantic understanding This creates a bottleneck in user experience across search, recommendations, customer support, and Q&A systems. For instance, in an e-commerce platform, when a user searches for “white dress suitable for summer,” a keyword-based system might match only product titles or categories. However, it would miss the multidimensional semantics in “suitable for summer,” such as fabric, style, and breathability. This problem also manifests in financial document search, smart customer support, and knowledge graph querying. To address this, we aim to build a semantic search system based on vector retrieval. The core idea is to transform text fields in business data into semantic vectors in real time, store them in a vector-supporting database, and enable retrieval based on semantic similarity. semantic search system based on vector retrieval Key challenges include: High-performance access and synchronization of heterogeneous data sources Online generation of text embeddings via AI models Structured storage of embeddings and vector index construction A highly available and low-latency storage system for vector search End-to-end observability and scalability High-performance access and synchronization of heterogeneous data sources Online generation of text embeddings via AI models Structured storage of embeddings and vector index construction A highly available and low-latency storage system for vector search End-to-end observability and scalability Technology Stack & Core Architecture To meet the system requirements, we chose a modern data engineering tech stack and implemented an end-to-end solution. Apache SeaTunnel: Centralized Data Integration and Sync Engine Apache SeaTunnel is an open-source, high-performance distributed data integration platform designed for both real-time and batch use cases. Key features include: Apache SeaTunnel Extensive connector plugin ecosystem: Supports 100+ data sources such as databases, message queues, file systems, object storage, and NoSQL systems Unified batch and stream processing: Suitable for full ingestion, incremental sync, and real-time CDC scenarios. Multi-engine support: Compatible with the native SeaTunnel Zeta engine, Flink, and Spark for flexible resource allocation. Pluggable transform layer: Custom Transform plugins can handle intermediate logic such as text preprocessing and external API calls for embeddings. Robust monitoring & ops: SeaTunnel Web offers graphical job orchestration and real-time monitoring. Extensive connector plugin ecosystem: Supports 100+ data sources such as databases, message queues, file systems, object storage, and NoSQL systems Extensive connector plugin ecosystem Unified batch and stream processing: Suitable for full ingestion, incremental sync, and real-time CDC scenarios. Unified batch and stream processing Multi-engine support: Compatible with the native SeaTunnel Zeta engine, Flink, and Spark for flexible resource allocation. Multi-engine support Pluggable transform layer: Custom Transform plugins can handle intermediate logic such as text preprocessing and external API calls for embeddings. Pluggable transform layer Robust monitoring & ops: SeaTunnel Web offers graphical job orchestration and real-time monitoring. Robust monitoring & ops In this solution, SeaTunnel serves as the data backbone. We use the Source module to extract raw data from databases or object storage, the Transform module to call the Amazon Bedrock API for text embeddings, and the Sink module to write the results into Amazon OpenSearch for vector-based semantic indexing. Source module Transform module Amazon Bedrock API for text embeddings Sink module Amazon Bedrock: Enterprise-Grade Embedding Service Amazon Bedrock is a fully managed service from AWS that provides API access to popular foundation models (FMs) such as Claude (Anthropic), Cohere, Stability AI, Mistral, and Amazon’s own Titan models — without needing to build or manage any infrastructure. Amazon Bedrock For the text embedding task, we evaluated two models: Cohere Embed v3 Offers multimodal embedding for both text and image, supports 100+ languages, and excels in cross-lingual and semantic-rich retrieval systems. It’s especially effective for complex multi-domain matching tasks. Amazon Titan Embeddings v2 A native AWS offering, it provides compact, high-quality embeddings with 256/512/1024 dimensions. Titan strikes a good balance between compression and retrieval accuracy, ideal for low-latency, high-concurrency, and storage-sensitive use cases. Cohere Embed v3 Offers multimodal embedding for both text and image, supports 100+ languages, and excels in cross-lingual and semantic-rich retrieval systems. It’s especially effective for complex multi-domain matching tasks. Cohere Embed v3 Amazon Titan Embeddings v2 A native AWS offering, it provides compact, high-quality embeddings with 256/512/1024 dimensions. Titan strikes a good balance between compression and retrieval accuracy, ideal for low-latency, high-concurrency, and storage-sensitive use cases. Amazon Titan Embeddings v2 Using Bedrock’s API, we invoke the embedding models during the Transform stage in SeaTunnel, converting raw text fields into high-dimensional dense vectors while retaining metadata (ID, tags, etc.) for downstream processing. Transform stage Amazon OpenSearch: Cloud-Native Vector Search Storage Amazon OpenSearch supports native knn_vector fields for indexing and retrieving vectorized data. It offers integration with popular ANN libraries like Faiss and NMSLIB and provides: Amazon OpenSearch knn_vector High concurrency for vector insertions and searches Hybrid search: combine structured queries with semantic vector retrieval Customizable vector fields (dimension, distance metric, indexing parameters) KNN plugin support for HNSW and other ANN algorithms Seamless integration with OpenSearch query syntax for compound searches like “price range + similar description” High concurrency for vector insertions and searches Hybrid search: combine structured queries with semantic vector retrieval Customizable vector fields (dimension, distance metric, indexing parameters) KNN plugin support for HNSW and other ANN algorithms Seamless integration with OpenSearch query syntax for compound searches like “price range + similar description” Using SeaTunnel’s OpenSearch Sink plugin, we can write both vectors and related structured fields (ID, title, tags) into OpenSearch — enabling low-code construction of a hybrid semantic+structured search engine. System Architecture & Implementation Steps Architecture Overview Data Ingestion Example Let’s walk through a real-world example using customer review data from Amazon’s e-commerce platform. The data is in JSON format: { "Item": { "review_id": {"S": "AEEZL8Z5691IJ"}, "date": {"S": "1215475200"}, "customer": {"S": "Amazon Customer"}, "asin": {"S": "B000Q6R4MK"}, "review": {"S": "I can hear the caller just great -- but I get frequent \\"what?\\" \\"I can't hear you\\" etc."}, "rating": {"S": "4.0"} } } { "Item": { "review_id": {"S": "AEEZL8Z5691IJ"}, "date": {"S": "1215475200"}, "customer": {"S": "Amazon Customer"}, "asin": {"S": "B000Q6R4MK"}, "review": {"S": "I can hear the caller just great -- but I get frequent \\"what?\\" \\"I can't hear you\\" etc."}, "rating": {"S": "4.0"} } } In many e-commerce scenarios, we need to search based on the review field. Here, we aim to vectorize this field and write the results to OpenSearch. review Below is the SeaTunnel config file to process the JSON: env { \# Set the execution engine to SeaTunnel Zeta Engine execution.engine = "seatunnel" \# Set job mode to BATCH for processing the JSON file job.mode = "BATCH" } source { S3File { path = "/data/3vk7gdzq6myxhn2kwexoiywjh4.json" bucket = "s3a://opensearch" file\_format\_type = "json" \# AWS region configuration fs.s3a.endpoint = "s3.us-east-1.amazonaws.com" \# Use SimpleAWSCredentialsProvider instead of InstanceProfileCredentialsProvider fs.s3a.aws.credentials.provider = "org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider" \# Provide explicit AWS credentials - access_key = "" secret_key = "" \# Additional S3A configuration options hadoop\_s3\_properties { "fs.s3a.impl" = "org.apache.hadoop.fs.s3a.S3AFileSystem" "fs.s3a.connection.ssl.enabled" = "true" } \# Required schema for JSON file format schema = { fields { Item = { review_id = { S = string } date = { S = string } customer = { S = string } asin = { S = string } review = { S = string } rating = { S = string } } } } \# Register output table plugin_output = "s3_data" } } transform { \# First transform to extract the actual review text from the nested S field Sql { plugin_input = "s3_data" plugin_output = "extracted_data" query = "SELECT Item.review\_id.S as review\_id, Item.date.S as date, Item.customer.S as customer, Item.asin.S as asin, Item.review.S as review, Item.rating.S as rating FROM s3_data" } \# Use Amazon Bedrock to generate embeddings for the review field Embedding { plugin_input = "extracted_data" plugin_output = "embedded_data" \# Specify the model provider as AMAZON for Bedrock model_provider = "AMAZON" \# Specify the model ID for Amazon Titan Embeddings model = "amazon.titan-embed-text-v2:0" \# AWS region for Bedrock service region = "us-east-1" \# AWS credentials for Bedrock service - api_key = "" secret_key = "" \# Define which fields to vectorize and their target fields vectorization_fields { review_embedding = review } \# Batch size for processing single\_vectorized\_input_number = 10 dimension = 1024 } } sink { Console { plugin_input = "embedded_data"\# 使用 Embedding 转换后的数据 limit = 10 } Elasticsearch { plugin_input = "embedded_data" \# OpenSearch endpoint hosts = \["https://xxxxxx.us-east-1.es.amazonaws.com"\] tls\_verify\_certificate = false \# Index configuration index = "reviews" username = "" password = "" vectorization_fields = \["review_embedding"\] vector_dimensions = 1024 } } env { \# Set the execution engine to SeaTunnel Zeta Engine execution.engine = "seatunnel" \# Set job mode to BATCH for processing the JSON file job.mode = "BATCH" } source { S3File { path = "/data/3vk7gdzq6myxhn2kwexoiywjh4.json" bucket = "s3a://opensearch" file\_format\_type = "json" \# AWS region configuration fs.s3a.endpoint = "s3.us-east-1.amazonaws.com" \# Use SimpleAWSCredentialsProvider instead of InstanceProfileCredentialsProvider fs.s3a.aws.credentials.provider = "org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider" \# Provide explicit AWS credentials - access_key = "" secret_key = "" \# Additional S3A configuration options hadoop\_s3\_properties { "fs.s3a.impl" = "org.apache.hadoop.fs.s3a.S3AFileSystem" "fs.s3a.connection.ssl.enabled" = "true" } \# Required schema for JSON file format schema = { fields { Item = { review_id = { S = string } date = { S = string } customer = { S = string } asin = { S = string } review = { S = string } rating = { S = string } } } } \# Register output table plugin_output = "s3_data" } } transform { \# First transform to extract the actual review text from the nested S field Sql { plugin_input = "s3_data" plugin_output = "extracted_data" query = "SELECT Item.review\_id.S as review\_id, Item.date.S as date, Item.customer.S as customer, Item.asin.S as asin, Item.review.S as review, Item.rating.S as rating FROM s3_data" } \# Use Amazon Bedrock to generate embeddings for the review field Embedding { plugin_input = "extracted_data" plugin_output = "embedded_data" \# Specify the model provider as AMAZON for Bedrock model_provider = "AMAZON" \# Specify the model ID for Amazon Titan Embeddings model = "amazon.titan-embed-text-v2:0" \# AWS region for Bedrock service region = "us-east-1" \# AWS credentials for Bedrock service - api_key = "" secret_key = "" \# Define which fields to vectorize and their target fields vectorization_fields { review_embedding = review } \# Batch size for processing single\_vectorized\_input_number = 10 dimension = 1024 } } sink { Console { plugin_input = "embedded_data"\# 使用 Embedding 转换后的数据 limit = 10 } Elasticsearch { plugin_input = "embedded_data" \# OpenSearch endpoint hosts = \["https://xxxxxx.us-east-1.es.amazonaws.com"\] tls\_verify\_certificate = false \# Index configuration index = "reviews" username = "" password = "" vectorization_fields = \["review_embedding"\] vector_dimensions = 1024 } } In this stage, we run the job in SeaTunnel. For verification purposes, we direct the output to the console using a logging sink. As shown in the log, we can see that a new field review_embedding has been added to each entry: review_embedding Until the task is completed, see the statistics Writing Embeddings to Amazon OpenSearch Once the embedding is generated, we need to persist the results into a searchable system. In this case, we use the OpenSearch sink provided by SeaTunnel to write both the original fields and the generated semantic vector review_embedding into an OpenSearch index. OpenSearch review_embedding The corresponding configuration in SeaTunnel is as follows: GET reviews/_search { "size": 5, "query": { "neural": { "review_embedding": { "query_text": "Installed and connected pretty well. Works good and keeps my eyes on the road. Took a few weeks to see the on and off button, but overall I like this and would suggest it.", "model_id": "8xwrFJYB5648rVcWvwIU", "k": 10 } } } } GET reviews/_search { "size": 5, "query": { "neural": { "review_embedding": { "query_text": "Installed and connected pretty well. Works good and keeps my eyes on the road. Took a few weeks to see the on and off button, but overall I like this and would suggest it.", "model_id": "8xwrFJYB5648rVcWvwIU", "k": 10 } } } } Summary and Outlook Building a Scalable and Loosely Coupled Semantic Search Pipeline with Apache SeaTunnel, Amazon Bedrock, and Amazon OpenSearch Building a Scalable and Loosely Coupled Semantic Search Pipeline with Apache SeaTunnel, Amazon Bedrock, and Amazon OpenSearch This article demonstrated how to build a scalable, loosely coupled semantic search pipeline by integrating Apache SeaTunnel, Amazon Bedrock, and Amazon OpenSearch. This solution integrates full-process from structured/unstructured text data ingestion to semantic vector-based retrieval. Apache SeaTunnel Amazon Bedrock Amazon OpenSearch Key Advantages of the Architecture Loosely Coupled Design: With SeaTunnel’s plugin-based architecture and modular support for Transform and SinkThe logic for embedding model invocation and vector persistence is decoupled. This allows for seamless model replacement or downstream database switching in future iterations. Cloud-Native AI Integration: By leveraging Amazon Bedrock’s API gateway and IAM-based access control, embedding capabilities can be integrated without the need to self-host model inference services — drastically reducing the operational barrier to adopting large-scale AI. Hybrid Search with Semantic and Structured Filtering: OpenSearch supports hybrid query execution, enabling both structured filtering (e.g., price ranges, categories) and semantic similarity ranking. This empowers a wide range of use cases — from e-commerce product search to enterprise knowledge retrieval. Loosely Coupled Design: With SeaTunnel’s plugin-based architecture and modular support for Transform and SinkThe logic for embedding model invocation and vector persistence is decoupled. This allows for seamless model replacement or downstream database switching in future iterations. Loosely Coupled Design Transform Sink Cloud-Native AI Integration: By leveraging Amazon Bedrock’s API gateway and IAM-based access control, embedding capabilities can be integrated without the need to self-host model inference services — drastically reducing the operational barrier to adopting large-scale AI. Cloud-Native AI Integration Hybrid Search with Semantic and Structured Filtering: OpenSearch supports hybrid query execution, enabling both structured filtering (e.g., price ranges, categories) and semantic similarity ranking. This empowers a wide range of use cases — from e-commerce product search to enterprise knowledge retrieval. Hybrid Search with Semantic and Structured Filtering Practical Tips for Enterprise-Scale Deployment To successfully adopt this architecture in a production-grade enterprise environment, we recommend focusing on the following optimization areas: 1. Embedding Caching and Batch Inference Optimization To reduce redundant model calls and lower embedding costs, implement deduplication and caching mechanisms (e.g., MD5 hashing or LRU cache) during the text embedding phase. Additionally, enable batch inference to process multiple texts in a single request — improving throughput and lowering API invocation costs for Bedrock. batch inference 2. Dimension Planning and Embedding Compression Choose embedding dimensions that balance retrieval precision, query latency, and storage cost for each business scenario. For example, Titan Embedding 512 is suitable for mid-sized use cases. For large-scale applications, consider applying dimensionality reduction techniques like PCA to reduce vector size and storage footprint. Titan Embedding 512 PCA 3. Vector Index Management and Lifecycle Control Optimize your OpenSearch index by tuning parameters such as refresh_interval, segment merge policy, and HNSW-specific settings like M and EF. Establish a regular index rebuild strategy to maintain retrieval accuracy without sacrificing write performance. refresh_interval segment merge policy M EF 4. Retrieval Quality Evaluation and Continuous Optimization Construct offline evaluation datasets with labeled queries and ground truth results. Use standard metrics like Recall@K, MRR, and nDCG to measure embedding effectiveness. Combine these metrics with A/B testing to continuously iterate on model versions and index configurations. Recall@K MRR nDCG A/B testing Final Thoughts: From Data Pipelines to Semantic Understanding Platforms This article outlines a practical path for transforming traditional data integration pipelines into intelligent, semantics-aware platforms. By integrating vector-based AI capabilities, organizations can unlock new potential in: Semantic search engines Personalized recommendation systems Knowledge retrieval platforms Enterprise-level Q&A assistants Semantic search engines Personalized recommendation systems Knowledge retrieval platforms Enterprise-level Q&A assistants Looking ahead, the fusion of multimodal embedding models and Retrieval-Augmented Generation (RAG) architectures will drive even deeper innovation. The synergy between Apache SeaTunnel and Amazon Bedrock positions them as a powerful pair for the future of AI-native data engineering. multimodal embedding models Retrieval-Augmented Generation (RAG) Apache SeaTunnel Amazon Bedrock AI-native data engineering