got an update, and you can now you set up with Amazon OpenSearch Service. But what does this even mean? Many people don't even know what OpenSearch is! Wasn't there an Elasticsearch service before? So, let's have some explanation before we check out Dashbirds OpenSearch support. Dashbird monitor the OpenSearch clusters Elastic and AWS Disagreed on Fair Use of Elasticsearch After a heated argument between Elastic, the main contributor to the Elasticsearch (ES) project, and AWS. Elastic then changed the ES license to prohibit AWS from offering their Amazon Elasticsearch Service. Since the ES versions previous to that change were still under the old Apache 2.0 license, AWS created a fork from these and called it OpenSearch (OS). Two companies disagreed on some software they implemented, and each one went on with their vision for that project. So, OS is merely a fork of ES, and these two open-source projects will probably diverge in the future in terms of compatibility and features. To put it simply: In terms of managed search services, AWS deprecated Amazon Elasticsearch Service, and Amazon OpenSearch Service (AOS) is the way to go. What is OpenSearch? OS is a search engine, just like Google or Bing, but for your own data. It will index your data and make it searchable via an HTTP interface. You can download OS and set it up on whatever computer you like. Install it on your local machine, set up an EC2 instance on AWS and install it there, or let AWS help you run it with the AOS. Get the full size infographic from . here OpenSearch vs. Elasticsearch They are pretty similar, but each one is on a different trajectory. Some developers aren't happy with AWS treating Elastic the way they did. Others say switching to a less permissive license isn't the open-source way, and while they don't like how AWS might have treated Elastic, they think the Apache 2.0 license is the way to go if things should stay open. If you want to use a managed, ES-like system on AWS, you either have to use older versions of ES or OS. What is Amazon OpenSearch Service? First, while AOS is a managed service, it's not serverless; you have to pay for idle servers. But what is AOS actually? AOS is the service on AWS that manages OS deployments for you. Like RDS is for relational databases, EKS is for Kubernetes clusters, and Elasticsearch Service is for ES, so is AOS for OS. While AOS is an excellent help to get OS running in the cloud, ES and, in turn, OS is notoriously hard to set up and scale. So hard that no serverless versions of them exist. While AOS isn't that easy to use, it's still simpler than managing your OS instances manually. AOS helps with scaling clusters, deploying VMs, and managing the underlying network and storage for you. It will also install bug fixes. If you need a cloud-based search engine for your project, you might not get around messing with OS and AOS. Amazon OpenSearch Service vs. Cloudwatch Log Insights When you deploy your infrastructure to the cloud, chances are most of the data you generate is operational logs. If you want to ease the access of these logs for the ops team, you might be inclined to pump them into AOS. But this can get costly. As mentioned above, AOS isn't a serverless system, so you pay for every hour it's deployed, even if you don't use it. This unused capacity could lead to a massive bill for nothing regarding operational monitoring data. Cloudwatch Log Insights is a managed service on top of Cloudwatch logs and, in turn, optimized for this exact use-case. It scales automatically, and you only have to pay for the data you analyze, so it can be considered a serverless service. AOS is a more low-level service you use if you need maximum flexibility. It costs more and isn't easy to scale, but it gives you all the freedom you need for a customized search engine. Amazon OpenSearch Service in Serverless Architectures As explained before, AOS isn't a serverless service. It can help with scaling, but it doesn't go down to zero, so you will pay for the underlying EC2 instances idling. This makes integrating AOS into your architecture a hard choice. But a hard choice between which options? If money is tight, or you believe that a system should scale to zero for whatever reason, you have to ask yourself what your search requirements are. AOS is the nuclear option here; it's a Lucene-based distributed search engine that finds everything you indexed with Google-like features like fuzzy matching and such. But if your search requirements aren't that strong, . Your results may vary and won't be on OpenSearch levels, but it might be just enough for a simple search feature. But most importantly, you don't pay for idle EC2 instances. you might get away with DynamoDB's query features Setting Up OpenSearch on AWS As said before, there are different methods of setting up OS on AWS, but we will go with the easiest here. Using the CDK, we will set up an AOS domain in just a few lines of code. Keep in mind this example taken from the CDK docs will deploy a production-ready OS cluster containing instances, which can get expensive quickly. 25 m3.medium const aosDomain = new opensearch.Domain(this, 'AosDomain', {

version: opensearch.EngineVersion.OPENSEARCH_1_0,

capacity: {masterNodes: 5, dataNodes: 20},

ebs: {volumeSize: 20},

zoneAwareness: {availabilityZoneCount: 3},

logging: {

slowSearchLogEnabled: true,

appLogEnabled: true,

slowIndexLogEnabled: true,

},

}); Coming from ES, you might have heard about a cluster and how it's made up of one or more nodes. An AOS domain is such a cluster, plus all the things around running such a cluster that AOS takes care of for you. Like networking, updates, resource allocation, etc. We must use the correct engine version to get an OS cluster because AOS also supports legacy ES clusters. The nodes are EC2 instances of the default instance type . m3.medium.search We use five master nodes, which means there will be one master and five master-eligible nodes which can take the place of the master if something goes wrong. We also use twenty data nodes, which will be used to store all the index shards (i.e., the actual data) of our cluster. Each of these data nodes will get a 20 GiB elastic block storage. The zone awareness makes sure AOS will deploy our 25 EC2 instances across three availability zones. This way, the cluster won't go down if one zone becomes unavailable. The logging config makes sure that queries and indices that perform poorly are logged so that when the cluster runs for some time, you get a feeling for the bottlenecks. OpenSearch Monitoring with Dashbird With the newest update to , all AOS domains will be automatically picked up in your Dashbird inventory. Dashbird This update , with all the related to your domains. Including, but not limited to, the utilization of the different node types and the query latency. gives you a dashboard out of the box health metrics While this is an excellent start to assess the status of your existing clusters in the context of their history, more interesting are the . well-architected Dashbird insights The insights are periodical checks based on the AWS Well-Architected Framework. This means they test your AOS domains for best practices and notify you if you're not following them. The insights also include some recurring checks for the availability and usage of your AOS domains. After all, you have to pay for them if they're deployed, and if they are never used, you might as well decommission them to save a few dollars. Summary OS is the search engine software that AWS will use from now on. If you want to stay with ES, you either have to use an older version or deploy your cluster manually with EC2, ECS, or EKS. But rest assured that Dashbird got your back with that new ES fork. So, you will stay on top of things with monitoring metrics and well-architected insights for your AOS domains.

Amazon

Apache

Google

Zones

6 Common DynamoDB Issues in 2022

A Serverless Perspective on HTTP API Gateway 

Serverless af

Nominated for 2022 - HackerNoon Contributor of the Year - Cloud Computing

Nominated for 2022 - HackerNoon Contributor of the Year - Devops

Nominated for 2022 - HackerNoon Contributor of the Year - Serverless

Nominated for 2022 - HackerNoon Contributor of the Year - Aws

Too Long; Didn't Read

OpenSearch From a Serverless Perspective

OpenSearch From a Serverless Perspective

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

10 Things in Engineering We Don't Spend Enough Time On

The Noonification: How Amazon Treats Warehouse Workers Who Contracted COVID (11/30/2022)

10 Free Ways to Promote Your Amazon Products

10 Failed Startup Product Examples by Google, Microsoft and Amazon

10 Best Infographics Of 2018

The Noonification: The Destroyer (12/29/2022)

10 Things in Engineering We Don't Spend Enough Time On

The Noonification: How Amazon Treats Warehouse Workers Who Contracted COVID (11/30/2022)

10 Free Ways to Promote Your Amazon Products

10 Failed Startup Product Examples by Google, Microsoft and Amazon

10 Best Infographics Of 2018

The Noonification: The Destroyer (12/29/2022)

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps