Elasticsearch, Logstash and Kibana, known as [ELK stack](https://www.elastic.co/products) or [Elastic stack](https://www.elastic.co/products) are the tools of the trade for logs aggregation and analysis. As these devops services are amongst the most oftenly requested, we automated their deployment with our tool available on [Github](https://github.com/ITSvitCo/aws-k8s/tree/master/kubernetes-manifests/elasticsearch).

Approximate scheme of ELK:

![](https://hackernoon.com/hn-images/1*xpO072p2xxwL9H7LCadj_w.jpeg)

These manifests DO NOT include the Filebeat installation! Refer to the [official Filebeat configuration documentation](https://www.elastic.co/guide/en/beats/filebeat/5.5/filebeat-configuration-details.html).

### Configuring a new ELK installation

This installation suits [Kubernetes on AWS](https://github.com/ITSvitCo/aws-k8s) deployment. Namespace \`elasticsearch\` is used by-default. Elasticsearch comes with 2 endpoints: [external](https://github.com/ITSvitCo/aws-k8s/blob/master/kubernetes-manifests/elasticsearch/client-svc-external.yaml) and [internal](https://github.com/ITSvitCo/aws-k8s/blob/master/kubernetes-manifests/elasticsearch/client-svc.yaml). Use both or drop the unnecessary one.

1\. Clone the [https://github.com/ITSvitCo/aws-k8s](https://github.com/ITSvitCo/aws-k8s) repository

2\. Create [StorageClass](https://github.com/ITSvitCo/aws-k8s/blob/master/kubernetes-manifests/storageclass-gp2.yml) in order to allow Kubernetes provision AWS EBS volumes.

docker run --rm -ti -v ${HOME}/.opsbox -v ${PWD}:/opsbox itsvit/opsbox kubectl apply -f kubernetes-manifests/storageclass-gp2.yml

3\. Launch [HA Elasticsearch cluster](https://github.com/ITSvitCo/aws-k8s/tree/master/kubernetes-manifests/elasticsearch). There will be 2 Elasticsearch masters, 2 Elasticsearch clients, 3 Elasticsearch data nodes.

docker run --rm -ti -v ${HOME}/.opsbox -v ${PWD}:/opsbox itsvit/opsbox kubectl apply -f kubernetes-manifests/elasticsearch

### Customizing Logstash

1\. If you need to store data in various indices, you should create a new manifest for Logstash. E.g. make a copy from existing manifest [logstash-application.yaml](https://github.com/ITSvitCo/aws-k8s/blob/master/kubernetes-manifests/elasticsearch/logstash-application.yaml)

2\. Set required index name in the output section:

yaml  
 output {  
 elasticsearch {  
 hosts => "elasticsearch-logging.elasticsearch:9200"  
 index => "new\_index-%{+YYYY.MM.dd}"  
 document\_type => "%{\[@metadata\]\[type\]}"  
 }  
 }

where new\_index is the required index name  
3\. Run this command to deploy a new Logstash:

docker run --rm -ti -v ${HOME}/.opsbox -v ${PWD}:/opsbox itsvit/opsbox kubectl apply -f kubernetes-manifests/elasticsearch/logstash-application.yaml

### Summary

We successfully use this devops solution as a part of data analysis and processing system. Here is an example of a running solution:

![](https://hackernoon.com/hn-images/1*Wt1TYP8GtR58IU8NxBLyMQ.jpeg)

This is yet another neat module from a collection of custom-tailored IT Svit DevOps tools, which ensures quick and simple deployment of a full-cycle ELK stack to Kubernetes.

This story was originally published on my company’s blog — [https://itsvit.com/blog/deployment-elk-stack-kubernetes-single-command/](https://itsvit.com/blog/deployment-elk-stack-kubernetes-single-command/)

Too Long; Didn't Read

Cassandra Forward a free Cassandra-focused community event

Deployment of full-scale ELK stack to Kubernetes

Too Long; Didn't Read

@FedakV

RELATED STORIES