Have you ever had issues with monitoring Nginx performance? In our recent project, my team needed to track down the time of requests in an already running Nginx web server. Since Nginx acts as a load balancer between services, you can track the request time between all of them. In this post, I want to share my expertise and show you an efficient way of collecting Nginx metrics.

\
Let's imagine that you have a small project where not everything (or nothing) is containerized. Therefore orchestration, convenient loki, and other tools for monitoring and analytics of requests are not used (but if I missed something, you can correct it in the comments).

\
## **The task**

There are several instances where the application is running and where Nginx is used as a balancer. We need to understand:

* the average request time,
* requests per second,
* response code and their intensity,
* traffic volume per unit of time,
* and so on.

\
**Initial data**

There is Prometheus + Grafana monitoring, where you will display graphs of the activity of your service.

\
**Research**

As it turned out, there are quite a few Nginx exporters. One of the most popular is [nginx-prometheus-exporter](https://github.com/nginxinc/nginx-prometheus-exporter) and grafana works well with it. But unfortunately, not many tools are able to collect the request time, and this was a very priority in our case. After a little googling, I found an [nginxlog](https://github.com/martin-helmich/prometheus-nginxlog-exporter) tool that displays metrics on a specific port by default, 4040. You can find more information about this tool [here](https://www.martin-helmich.de/). Once I’ve run some small tests, it became clear that it is suitable for our case.

\
**Configuration**

To configure the format of our logs, you need to l set such parameters in the Nginx config, which will be located in `/etc/nginx/conf.d/`

Let's set the format - just add the follow setting into the `custom-log-format.conf`:

```yaml
log_format custom '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" "$request_time"';
```

On the next step, go to /etc/nginx/sites-enabled/sitename - your web server config, and add the lines:

```yaml
access_log /var/log/nginx/sitename-access.log custom;
error_log /var/log/nginx/sitename-error.log;
```

\
After that, you need to  check  Nginx configuration:

```bash
nginx -t
```

\
 ![If you performed all the operations correctly, you’ll see the following lines](https://cdn.hackernoon.com/images/-li026zy.png)

Then, Nginx needs to re-read the configs:

```bash
service nginx reload
```

\
And now you can install  nginxlog-exporter. Based on  the documentation, it’s clear that you can supply the exporter as a deb package, which is quite suitable for your small project, both for now and for future automation using Ansible.

Now, let’s download the package:

```bash
wget https://github.com/martin-helmich/prometheus-nginxlog-exporter/releases/download/v1.8.0/prometheus-nginxlog-exporter_1.8.0_linux_amd64.deb -P /tmp
```

\
Install this package using the apt package manager:

```bash
apt install /tmp/prometheus-nginxlog-exporter_1.8.0_linux_amd64.deb
```

\
Next, you need to create a file: `/etc/prometheus-nginxlog-exporter.hcl`

\
Which is the configuration file of your exporter, and which should contain:

```javascript
listen {
  port = 4040
  address = "<server_ip>"
}

namespace "frontend" {
  source = {
    files = [
      "/var/log/nginx/sitename-access.log"
    ]
  }

  format = "$remote_addr - $remote_user [$time_local] \"$request\" $status $body_bytes_sent \"$http_referer\" \"$http_user_agent\" \"$request_time\""

  labels {
    app = "frontend"
  }
}
```

\
Nota bene! Pay special attention to the name specified in the namespace. You’ill use this name to build Prometheus requests.

\
Add your service to autostart:

```bash
systemctl enable prometheus-nginxlog-exporter
```

I took all installation steps from the [official developer repository](https://github.com/martin-helmich/prometheus-nginxlog-exporter).

\
Afterward, you need to check the status of your service:

```bash
systemctl status prometheus-nginxlog-exporter
```

\
If you implemented all commands in the right way, you’ll see the following result:

 ![output of "systemctl status prometheus-nginxlog-exporter"](https://cdn.hackernoon.com/images/-8sb26gv.png)

If you get a message that the service status is inactive, as shown in the picture below:

 ![output of "systemctl status prometheus-nginxlog-exporter" if service didn't started](https://cdn.hackernoon.com/images/LWdcfPWRMcVhK2WiyMfnsfsxN9N2-s0426vv.png)Then you need to run this command:

```bash
systemctl start prometheus-nginxlog-exporter
```

\
You  can see that the exporter returns to us by going to: `<server_ip>:4040/metric`s. If there is activity on the frontend, then you  can see a list of metrics that will be used in Prometheus and Grafana.

\
Nota bene! Don't forget to add the server ip and port to your Prometheus config file.

\
Now, your task is to collect and parse these metrics so that Grafana can draw graphs.

\
Let's take a look at Prometheus. Try to enter your frontend keyword, which you  specified in the namespace block in the configuration. You’ll see the following results:

 ![Prometheus interface](https://cdn.hackernoon.com/images/LWdcfPWRMcVhK2WiyMfnsfsxN9N2-l9526ms.png)

Now, you need to determine what metrics you want to see in your dashboard in Grafana. At the beginning of the article, we were interested in the average responce time.

\
Next, you  can move on to setting up our graphs in Grafana. First, create a new dashboard:

 ![Creating new Dashboard](https://cdn.hackernoon.com/images/LWdcfPWRMcVhK2WiyMfnsfsxN9N2-i3626ig.png)Add the panel:

 ![How new dashboard looks like](https://cdn.hackernoon.com/images/LWdcfPWRMcVhK2WiyMfnsfsxN9N2-5w726rr.png)

Then, you need to write a request:

```javascript
sum(rate(frontend_http_response_time_seconds_sum[5m])) by (app) / sum(rate(frontend_http_response_time_seconds_count[5m])) by (app)
```

\
Once you’ve added all the necessary settings to the panel, you’ll see  its final appearance:

 ![The final look of the "Average responce time" panel](https://cdn.hackernoon.com/images/-5da26rd.png)

You can also add additional metrics, for example "Status codes per second"

```javascript
sum(rate(frontend_http_response_count_total[1m])) by (status,app)
```

\
In this case, you’ll  have one more panel:

 ![The final look of the "Status codes per second" panel](https://cdn.hackernoon.com/images/LWdcfPWRMcVhK2WiyMfnsfsxN9N2-si92624.png)

Correspondingly, you can add other metrics to this panel, if you need it.

\
Once you set up all the metrics, you can configure alerts. For instance, using Alertmanager. But this is a different case.

\
A few more examples of queries that can be used by the monitoring system:

Requests per second:

```javascript
sum(rate(frontend_http_response_time_seconds_count[1m])) by (app)
```

Http traffic:

```javascript
sum(rate(frontend_http_response_size_bytes[5m])) by (app)
```

\
**Conclusion**

Now you know how to make your Grafana dashboard more informative. For a deeper analysis, other tools may be needed. But prometheus-nginxlog-exporter is quite suitable for solving such problems. Hope my guide will help you with your next project!

\
Please, leave the comments below if you have any questions or you just want to share your experience.

LOKI

2022 - HackerNoon Contributor of the Year - Linux

2022 - HackerNoon Contributor of the Year - Monitoring

Nominated for 2022 - HackerNoon Contributor of the Year - Monitoring

Nominated for 2022 - HackerNoon Contributor of the Year - Linux

Too Long; Didn't Read

Cassandra Forward a free Cassandra-focused community event

Monitor Nginx Metrics with GrafanaDR: A Step-by-Step Guide

Too Long; Didn't Read

Company Mentioned

@mykolaprokopenko

RELATED STORIES