3,402 reads

Real-time API Monitoring Using GO and Pusher

by neoMarch 8th, 2018

Too Long; Didn't Read

REST is a popular architectural style for providing standards between computer systems on the web, making it easier for systems to communicate with each other. It is mostly used by APIs to provide data to other systems requiring them.

Companies Mentioned

featured image - Real-time API Monitoring Using GO and Pusher

Disclosure: Pusher, which provide real-time APIs for developers, has previously sponsored Hacker Noon.

A basic understanding of Go and JavaScript is needed to follow this tutorial.

Sometimes, the providers of APIs would like to monitor its use. Monitoring APIs helps provide useful information, such as which endpoints are called most frequently, or what regions are the largest audience using request IP Addresses. This information can then be used to optimize the API.

In this article, we will implement realtime monitoring of a small API built with GoLang, using Pusher. Here’s a preview of what it should look like at the end:

Requirements

To follow along with this article, you will need the following:

An IDE of your choice e.g. Visual Studio Code.
Go installed on your computer.
Basic knowledge of GoLang.
Basic knowledge of JavaScript (ES6 syntax) and jQuery.
Basic knowledge of using a CLI tool or terminal.

Once you have all the above requirements, let’s proceed.

Setting up our codebase

To keep things simple, we’ll be using an already written GoLang CRUD API, which is available on GitHub. We will fork the repository and set it up following the README.md guidelines on installation.

Next, we will set up Pusher in the API project. Pusher is a service that provides a simple implementation of realtime functionality for our web and mobile applications. We will use it in this article, to provide realtime updates to our API monitor dashboard.

Let’s head over to Pusher.com, you can create a free account if you don’t already have one. On the dashboard, create a new app and copy out the app credentials (App ID, Key, Secret, and Cluster). We will use these credentials in our API.

Now that we have our Pusher app, we will install the Pusher Go library by running:

$ go get github.com/pusher/pusher-http-go

Monitoring our API

We have so far set up a functional CRUD API, and we will now implement monitoring calls to it. In this article, we will monitor:

The endpoints called with details like name, request type (GET, POST, etc) and URL.
For each call to an endpoint, we will also take note of:
The requesting IP address remove,
The response status code for the particular call.

Now that we have defined what to monitor, we will begin by creating models to keep track of the data we acquire.

Creating models for monitoring

Based on our specifications above, we will create two new model files EndPoints.go and EndPointCalls.go. As was used in the base API, we will use the GORM (the GoLang ORM) for managing data storage.

💡 Our new model files will exist in the models directory and belong to the models package.

In EndPoints.go, we will define the EndPoints object and a method to save endpoints:

package models

import ("github.com/jinzhu/gorm")

// EndPoints - endpoint modeltype EndPoints struct {gorm.ModelName, URL stringType string `gorm:"DEFAULT:'GET'"`Calls []EndPointCalls `gorm:"ForeignKey:EndPointID"`}

// SaveOrCreate - save endpoint calledfunc (ep EndPoints) SaveOrCreate() EndPoints {db.FirstOrCreate(&ep, ep)return ep}

In the code block above, our model did not re-initialize the GORM instance db, yet it was used. This is because the instance defined in the Movies.go file was global to all members of the package, and so it can be referenced and used by all members of package models.

💡 Our EndPoints model has an attribute _Calls_ which is an array of _EndPointCalls_ objects. This attribute signifies the one to many relationship between _EndPoints_ and _EndPointCalls_. For more information on model associations and relationships see the GORM documentation.

Next, we’ll fill in the model definitions and methods for our EndPointCalls model in the EndPointCalls.go file:

package models

import ("github.com/jinzhu/gorm""github.com/kataras/iris")

// EndPointCalls - Object for storing endpoints call detailstype EndPointCalls struct {gorm.ModelEndPointID uint `gorm:"index;not null"`RequestIP stringResponseCode int}

// SaveCall - Save the call details of an endpointfunc (ep EndPoints) SaveCall(context iris.Context) EndPointCalls {epCall := EndPointCalls{EndPointID: ep.ID,RequestIP: context.RemoteAddr(),ResponseCode: context.GetStatusCode(),}

db.Create(&epCall)  
  
return epCall

}

As shown above, our EndPointCalls model defines a SaveCall method, which stores the requesting IP address and the response code of an existing EndPoint object.

Finally, we will update the model migration in the index.go file to include our new models:

// index.go// ...

func main() {// ...

// Initialize ORM and auto migrate models  
db, \_ := gorm.Open("sqlite3", "./db/gorm.db")  
db.AutoMigrate(&models.Movies{}, &models.EndPoints{}, &models.EndPointCalls{})

// ...

}

Saving endpoint data for monitoring

Using our newly created models, we will edit the MoviesController.go file to save relevant data when an endpoint is called.

To do this, we will add a private helper method to MoviesController.go, which will save endpoint data with our models. See how below:

// MoviesController.go// ...

func (m MoviesController) saveEndpointCall(name string) {endpoint := models.EndPoints{Name: name,URL: m.Cntx.Path(),Type: m.Cntx.Request().Method,}

endpoint = endpoint.SaveOrCreate()  
endpointCall := endpoint.SaveCall(m.Cntx)

}

The saveEndpointCall method takes the name of the endpoint as a parameter. Using the controller’s iris.Context instance, it reads and saves the endpoint path and request method.

Now that this helper method is available, we will call it in each of the endpoint methods in the MoviesController.go file:

// MoviesController.go// ...

// Get - get a list of all available moviesfunc (m MoviesController) Get() {movie := models.Movies{}movies := movie.Get()

go m.saveEndpointCall("Movies List")  
m.Cntx.JSON(iris.Map{"status": "success", "data": movies})

}

// GetByID - Get movie by IDfunc (m MoviesController) GetByID(ID int64) {movie := models.Movies{}movie = movie.GetByID(ID)if !movie.Validate() {msg := fmt.Sprintf("Movie with ID: %v not found", ID)m.Cntx.StatusCode(iris.StatusNotFound)m.Cntx.JSON(iris.Map{"status": "error", "message": msg})} else {m.Cntx.JSON(iris.Map{"status": "success", "data": movie})}

name := fmt.Sprintf("Single Movie with ID: %v Retrieval", ID)  
go m.saveEndpointCall(name)

}

// ...

As shown in the snippet above, the saveEndpointCall helper method will be called in each CRUD method.

💡 The _saveEndpointCall_ method is called as a Goroutine. Calling it this way calls it concurrently with the execution of the endpoint’s method, and allows our monitoring code to not delay or inhibit the response of the API.

Creating the endpoint monitor dashboard

Now that we have implemented monitoring our API’s calls, we will display the data we have accrued on a dashboard.

Registering our template engine

The GoLang framework, Iris, has the ability to implement a range of template engines, which we will take advantage of.

In this section, we will implement the Handlebars template engine, and in our index.go file, we will register it to the app instance:

// index.gopackage main

import ("goggles/controllers""goggles/models""github.com/jinzhu/gorm""github.com/kataras/iris")

func main() {app := iris.New()

tmpl := iris.Handlebars("./templates", ".html")    
app.RegisterView(tmpl)

// ...

app.Run(iris.Addr("127.0.0.1:1234"))

}

💡 We have defined our template engine (Handlebars), to render _.html_ files contained in the _templates_directory.

Creating the dashboard’s route and controller

Now that we have registered our template engine to the application, we will add a route in index.go to render our API monitor dashboard:

// index.go// ...

func main() {app := iris.New()

// ...

app.Get("/admin/endpoints", func(ctx iris.Context) {  
    dashBoard := controllers.DashBoardController{Cntx: ctx}  
    dashBoard.ShowEndpoints()  
})

app.Run(iris.Addr("127.0.0.1:1234"))

}

Above, we have added definitions for the path /admin/endpoints, where we intend to render details of our API endpoints and its calls. We have also specified that the route should be handled by the ShowEndpoints method of DashBoardController.

To create DashBoardController, we will create a DashBoardController.go file in the controllers directory. And in our DashBoardController.go file, we will define the DashBoardController object and its ShowEndpoints method:

// DashBoardController.gopackage controllers

import ("goggles/models""github.com/kataras/iris""github.com/kataras/iris/mvc")

// DashBoardController - Controller object for Endpoints dashboardtype DashBoardController struct {mvc.BaseControllerCntx iris.Context}

// ShowEndpoints - show list of endpointsfunc (d DashBoardController) ShowEndpoints() {endpoints := (models.EndPoints{}).GetWithCallSummary()d.Cntx.ViewData("endpoints", endpoints)d.Cntx.View("endpoints.html")}

In ShowEndpoints(), we retrieve our endpoints and a summary of their calls for display. Then we pass this data to our view using d.Cntx.ViewData("endpoints", endpoints), and finally we render our view file templates/endpoints.html using d.Cntx.View("endpoints.html").

Retrieving endpoints and call summaries

To retrieve our list of endpoints and a summary of their calls, we will create a method in the EndPoints.go file called GetWithCallSummary.

Our GetWithCallSummary method should return the endpoints and their call summaries ready for display. For this, we will define a collection object EndPointWithCallSummary with the attributes we need for our display in the EndPoints.go file:

// EndPoints.gopackage models

import ("github.com/jinzhu/gorm")

// EndPoints - endpoint modeltype EndPoints struct {gorm.ModelName, URL stringType string `gorm:"DEFAULT:'GET'"`Calls []EndPointCalls `gorm:"ForeignKey:EndPointID"`}

// EndPointWithCallSummary - Endpoint with last call summarytype EndPointWithCallSummary struct {ID uintName, URL stringType stringLastStatus intNumRequests intLastRequester string}

And then define GetWithCallSummary method to use it as follows:

// EndPoints.go

// ...

// GetWithCallSummary - get all endpoints with call summary detailsfunc (ep EndPoints) GetWithCallSummary() []EndPointWithCallSummary {var eps []EndPointsvar epsWithDets []EndPointWithCallSummary

db.Preload("Calls").Find(&eps)

for \_, elem := range eps {  
    calls := elem.Calls  
    lastCall := calls\[len(calls)-1:\]\[0\]

    newElem := EndPointWithCallSummary{  
        elem.ID,  
        elem.Name,  
        elem.URL,  
        elem.Type,  
        lastCall.ResponseCode,  
        len(calls),  
        lastCall.RequestIP,  
    }

    epsWithDets = append(epsWithDets, newElem)  
}

return epsWithDets

}

// ...

Above, the GetWithCallSummary method leverages the Calls attribute of EndPoints, which defines its relationship with EndPointCalls. When retrieving our list of endpoints from the database, we eager load its EndPointCalls data using db.Preload("Calls").Find(&eps).

For more information on eager loading in GORM, see the documentation.

GetWithCallSummary initializes an array of EndPointWithCallSummary, and loops through the EndPointsobjects returned from our database to create EndPointWithCallSummary objects.

These EndPointWithCallSummary objects are appended to the initialized array and returned.

💡 The _EndPointWithCallSummary_ is not a model. It is a collection object and does not need to have a table in our database. This is why it does not have its own file and is not passed to _index.go_ for migration.

Implementing the dashboard and displaying data

Now that we have the dashboard’s route, controller and data for display, we will implement the dashboard view to achieve a simple list display of endpoints and their summary data.

Let’s update templates/endpoints.html to have the following code:

<!DOCTYPE html><html><head><title>Endpoints Monitor Dashboard</title><link rel="stylesheet" type="text/css" href="https://cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/4.0.0-beta.3/css/bootstrap.min.css" /></head><body><div><nav class="navbar navbar-default navbar-static-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="http://127.0.0.1:1234/">Goggles - A Real-Time API Monitor</a></div></div></nav><div class="container"><div class="row"><div class="col-xs-12 col-lg-12"><div class="endpoints list-group">{{#each endpoints}}<a id="endpoint-{{ID}}" href="#" class="list-group-itemlist-group-item-{{status_class LastStatus}}"><strong>{{name}}</strong><span class="stats">{{type}}: <strong>{{url}}</strong> |Last Status: <span class="last_status">{{LastStatus}}</span> |Times Called: <span class="times_called">{{NumRequests}}</span> |Last Request IP: <span class="request_ip">{{LastRequester}}</span></span></a>{{/each}}</div></div></div></div></div><script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/3.2.1/jquery.min.js"></script><script src="https://cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/4.0.0-beta.3/js/bootstrap.min.js"></script></body></html>

Above, we render our endpoints list using Bootstrap and our Handlebars template engine. We have also created and used a template function status_class, to colour code our list based on their last call status LastStatus.

We define the status_class template function in index.go after initialising our template engine:

// index.go

// ...

func main() {app := iris.New()

tmpl := iris.Handlebars("./templates", ".html")

tmpl.AddFunc("status\_class", func(status int) string {  
    if status >= 200 && status < 300 {  
        return "success"  
    } else if status >= 300 && status < 400 {  
        return "warning"  
    } else if status >= 400 {  
        return "danger"  
    }  
    return "success"  
})

app.RegisterView(tmpl)

}

Then in our view file we call the function as:

class="list-group-item list-group-item-{{status_class LastStatus}}"

💡 In the above LastStatus is the function’s parameter.

Adding realtime updates to our dashboard

So far in this article, we have monitored the calls to an API and displayed the data via a dashboard. We will now use Pusher to provide realtime data updates to our dashboard.

Sending realtime data from the backend

Earlier, we installed the Pusher Go library, which we will use to trigger an event when an endpoint is called. In the MoviesController.go file, where the API requests are handled, we will initialize the Pusher client:

// MoviesController.go

    package controllers

    import (        // ...        "github.com/pusher/pusher-http-go"    )

    // MoviesController - controller object to serve movie data    type MoviesController struct {        mvc.BaseController        Cntx iris.Context    }

    var client = pusher.Client{        AppId:   "app_id",        Key:     "app_key",        Secret:  "app_secret",        Cluster: "app_cluster",    }

    // ...

Here, we have initialized the Pusher client using the credentials from our earlier created app.

⚠️ Replace _app_id, app_key, app_secret and app_cluster_ with your app credentials.

Next, we will use our Pusher client to trigger an event, which would include the endpoint’s data to be displayed in our view. We will do this in the saveEndpointCall method, which logs an endpoint and its call:

// MoviesController.go

    // ...

    func (m MoviesController) saveEndpointCall(name string) {        endpoint := models.EndPoints{            Name: name,            URL:  m.Cntx.Path(),            Type: m.Cntx.Request().Method,        }        endpoint = endpoint.SaveOrCreate()        endpointCall := endpoint.SaveCall(m.Cntx)        endpointWithCallSummary := models.EndPointWithCallSummary{            ID:            endpoint.ID,            Name:          endpoint.Name,            URL:           endpoint.URL,            Type:          endpoint.Type,            LastStatus:    endpointCall.ResponseCode,            NumRequests:   1,            LastRequester: endpointCall.RequestIP,        }

        client.Trigger("goggles_channel", "new_endpoint_request", endpointWithCallSummary)    }

Above, we create an EndPointWithCallSummary object from EndPoints (the endpoint) and EndPointCalls. This EndPointWithCallSummary object has all the data required for display on the dashboard, so will be passed to Pusher for transmission.

Displaying data in realtime on the dashboard

To display the realtime updates of our endpoints, we will use the Pusher JavaScript client and jQuery libraries.

In our view file, templates/endpoints.html, we will first import and initialize a Pusher instance using our app’s credentials:

<!-- endpoints.html -->    <script src="//cdnjs.cloudflare.com/ajax/libs/jquery/3.2.1/jquery.min.js"></script>    <script src="//cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/4.0.0-beta.3/js/bootstrap.min.js"></script>    <script src="//js.pusher.com/4.1/pusher.min.js"></script>    <script>      const pusher = new Pusher('app_id', {cluster: "app_cluster"});    </script>

⚠️ Replace _app_id and app_cluster_ with values from your app’s credentials.

Next, we will define the following:

The template for adding new endpoints to our view.
The functions to append a new endpoint and get the status class of the endpoint.

Finally, we will subscribe to the goggles_channel and listen to the new_endpoint_request event, where our endpoint updates will be transmitted:

<!-- endpoints.html -->    <script>    // ...    const channel = pusher.subscribe("goggles_channel");    channel.bind('new_endpoint_request', function(data) {        let end_point_id = data.ID;        if ( $('#endpoint-' + end_point_id).length > 0 ) {            let status_class = getItemStatusClass(data['LastStatus']),                endpoint     = $('#endpoint-' + end_point_id);            let calls = 1 * endpoint.find('span.times_called').text()            endpoint.find('span.last_status').text(data['LastStatus']);            endpoint.find('span.times_called').text( (calls + 1) )            endpoint.removeClass('list-group-item-success');            endpoint.removeClass('list-group-item-danger');            endpoint.removeClass('list-group-item-warning');            endpoint.addClass('list-group-item-' + status_class);        } else {            addNewEndPoint(data);        }    });    // ...

In the new_endpoint_request event handler, the endpoint data is categorized into either an update scenario (where the endpoint already exists on the dashboard) or a create scenario (where a new list item is created and appended).

Finally, you can build your application and when you run it you should see something similar to what we have in the preview: