Elasticsearch, Logstash and Kibana, known as or are the tools of the trade for logs aggregation and analysis. As these devops services are amongst the most oftenly requested, we automated their deployment with our tool available on . ELK stack Elastic stack Github Approximate scheme of ELK: These manifests DO NOT include the Filebeat installation! Refer to the . official Filebeat configuration documentation Configuring a new ELK installation This installation suits deployment. Namespace `elasticsearch` is used by-default. Elasticsearch comes with 2 endpoints: and . Use both or drop the unnecessary one. Kubernetes on AWS external internal 1. Clone the repository https://github.com/ITSvitCo/aws-k8s 2. Create in order to allow Kubernetes provision AWS EBS volumes. StorageClass docker run --rm -ti -v ${HOME}/.opsbox -v ${PWD}:/opsbox itsvit/opsbox kubectl apply -f kubernetes-manifests/storageclass-gp2.yml 3. Launch . There will be 2 Elasticsearch masters, 2 Elasticsearch clients, 3 Elasticsearch data nodes. HA Elasticsearch cluster docker run --rm -ti -v ${HOME}/.opsbox -v ${PWD}:/opsbox itsvit/opsbox kubectl apply -f kubernetes-manifests/elasticsearch Customizing Logstash 1. If you need to store data in various indices, you should create a new manifest for Logstash. E.g. make a copy from existing manifest logstash-application.yaml 2. Set required index name in the output section: yamloutput {elasticsearch {hosts => "elasticsearch-logging.elasticsearch:9200"index => "new_index-%{+YYYY.MM.dd}"document_type => "%{[@metadata][type]}"}} where new_index is the required index name3. Run this command to deploy a new Logstash: docker run --rm -ti -v ${HOME}/.opsbox -v ${PWD}:/opsbox itsvit/opsbox kubectl apply -f kubernetes-manifests/elasticsearch/logstash-application.yaml Summary We successfully use this devops solution as a part of data analysis and processing system. Here is an example of a running solution: This is yet another neat module from a collection of custom-tailored IT Svit DevOps tools, which ensures quick and simple deployment of a full-cycle ELK stack to Kubernetes. This story was originally published on my company’s blog — https://itsvit.com/blog/deployment-elk-stack-kubernetes-single-command/
