Monitor your Infrastructure with TIG Stack

In this tutorial, I will show you how to setup a monitoring stack for your infrastructure. So you can collects data from your servers, docker containers, and other kinds of network devices so you can analyze it for trends or problems.

Note: All templates are available on my Github.

1 — How it works ?

1.1 — Telegraf

Data collector written in Go for collecting, processing, and aggregating and writting metrics. Its a plugin driven tool, we will use a few plugins while implementing our use case.

1.2 — InfluxDB

Scalable time series database for metrics, events and real-time analytics.

1.3 — Grafana

Data visualization and exploration tool. It lets you create graphs and dashboards based on data from various data sources (InfluxDB, Prometheus, Elasticsearch, Cloudwatch …)

2 — Setup

Clone the repository:

|git clone https://github.com/mlabouardy/telegraf-influxdb-grafana.git

To start all of these containers I’m using docker-compose:

The docker-compose bring up 3 containers:

1 — Influxdb:

Due to the ephemeral nature of containers. We exposed the InfluxDB data folder to our host system. So our data wont disappear if the container restarts or is stopped.

The port mapping contains 3 port:

8083: this is the administration web server’s port, you can open the admin page by http://localhost:8083

8086: this is the HTTP API endpoint port, it’s used to send query to Influxdb by Telegraf

2 — Grafana

The port 3000 is the default web server port.

We used docker’s link feature to link Grafana container with our Influxdb container, so Grafana can connect to Influxdb and query data from it.

3 — Telegraf

Telegraf collect metrics from “input” plugins, parse it to the correct format then send it to “output” plugins. There is a lot of input and output plugins, you just have to activate them in the Telegraf configuration file:

Here I’m using the Docker input plugin to fetch all the stats from the docker daemon (resource usage per container) and System input plugin to pull server metrics (Disk, CPU, RAM …)

To start all of these services, we will use docker-compose:

If you type “docker ps“, you should see the TIG containers:

3 — Configure

Point your browser to http://SERVER_IP:3000, you should see Grafana Dashboard:

The default credential is admin with password admin. You will want to change this as soon as you can.

Now we need to create an Influxdb datasource pointing to the InfluxDB container.

3.1 — VM Data Source

We configure Grafana to pull data from vm_metrics database:

3.2 — Docker Data Source

Then we create another data source to fetch data from docker_metrics database.

Once that is completed, you are ready to start creating dashboards.

4 — Dashboards

On the top left menu, click on “Add a new Dashboard” then click on “Add a panel“:

4.1 — VM

4.1.1 — Memory

4.1.1 — Disk

4.1.3 — CPU

4.1.4 — Network

All graphs combined:

4.2 — Docker

4.2.1- Create Container Filter

In order to filter our data by container name, we will use a concept in Grafana called Templating which makes our Dashboard more interactive and dynamic. Therefore we won’t hard-code the name of the container in the metric query but instead we will use a variable.

So to create a variable, click on settings icon, then “Templating“:

Click on “New” and fill the fields as described below:

Once created, now the variable is shown as dropdown select boxes at the top of the dashboard. This dropdown make it easy to change the data being displayed in your dashboard.

Now our filter is created, we can jump to create our first graph:

4.2.1 — Memory

Here is a screenshot of the result: