Real-time API Monitoring Using GO and Pusher

![](https://hackernoon.com/hn-images/1*GkRPNOGl8xDtiqOsUTKgTg.jpeg) > Disclosure: [Pusher](https://goo.gl/52hnDL), which provide real-time APIs for developers, has previously sponsored Hacker Noon. > A basic understanding of Go and JavaScript is needed to follow this tutorial. REST is a popular architectural style for providing standards between computer systems on the web, making it easier for systems to communicate with each other. It is mostly used by APIs to provide data to other systems requiring them. Sometimes, the providers of APIs would like to monitor its use. Monitoring APIs helps provide useful information, such as which endpoints are called most frequently, or what regions are the largest audience using request IP Addresses. This information can then be used to optimize the API. In this article, we will implement realtime monitoring of a small API built with GoLang, using Pusher. Here’s a preview of what it should look like at the end: ![](https://hackernoon.com/images/1*uxk2Dhe1apJP8iKvRbmh0A.gif) ### Requirements To follow along with this article, you will need the following: * An IDE of your choice e.g. [Visual Studio Code](https://code.visualstudio.com/). * [Go](https://golang.org/doc/install) installed on your computer. * Basic knowledge of GoLang. * Basic knowledge of JavaScript (ES6 syntax) and jQuery. * Basic knowledge of using a CLI tool or terminal. Once you have all the above requirements, let’s proceed. ### Setting up our codebase To keep things simple, we’ll be using an already written GoLang CRUD API, which is available on [GitHub](https://github.com/neoighodaro/go-pusher-api-monitor/tree/master/api-goggles). We will fork the repository and set it up following the **README.md** guidelines on installation. Next, we will set up Pusher in the API project. Pusher is a service that provides a simple implementation of realtime functionality for our web and mobile applications. We will use it in this article, to provide realtime updates to our API monitor dashboard. Let’s head over to Pusher.com, you can [create a free account](https://pusher.com/signup?plan=sandbox) if you don’t already have one. On the [dashboard](https://dashboard.pusher.com/), create a new app and copy out the app credentials (App ID, Key, Secret, and Cluster). We will use these credentials in our API. Now that we have our Pusher app, we will install the Pusher Go library by running: $ go get github.com/pusher/pusher-http-go ### Monitoring our API We have so far set up a functional CRUD API, and we will now implement monitoring calls to it. In this article, we will monitor: * The endpoints called with details like name, request type (GET, POST, etc) and URL. * For each call to an endpoint, we will also take note of: * The requesting IP address remove, * The response status code for the particular call. Now that we have defined what to monitor, we will begin by creating models to keep track of the data we acquire. ### Creating models for monitoring Based on our specifications above, we will create two new model files `EndPoints.go` and `EndPointCalls.go`. As was used in the base API, we will use the [GORM](http://jinzhu.me/gorm/) (the GoLang ORM) for managing data storage. > _💡 Our new model files will exist in the models directory and belong to the models package._ In `EndPoints.go`, we will define the `EndPoints` object and a method to save endpoints: package models import ( "github.com/jinzhu/gorm" ) // EndPoints - endpoint model type EndPoints struct { gorm.Model Name, URL string Type string \`gorm:"DEFAULT:'GET'"\` Calls \[\]EndPointCalls \`gorm:"ForeignKey:EndPointID"\` } // SaveOrCreate - save endpoint called func (ep EndPoints) SaveOrCreate() EndPoints { db.FirstOrCreate(&ep, ep) return ep } In the code block above, our model did not re-initialize the GORM instance `db`, yet it was used. This is because the instance defined in the `Movies.go` file was global to all members of the package, and so it can be referenced and used by all members of `package models`. > _💡 Our EndPoints model has an attribute_ `_Calls_` _which is an array of_ `_EndPointCalls_` _objects. This attribute signifies the_ [_one to many_](http://jinzhu.me/gorm/associations.html#has-many) _relationship between_ `_EndPoints_` _and_ `_EndPointCalls_`_. For more information on model associations and relationships see the GORM_ [_documentation_](http://jinzhu.me/gorm/associations.html)_._ Next, we’ll fill in the model definitions and methods for our `EndPointCalls` model in the `EndPointCalls.go` file: package models import ( "github.com/jinzhu/gorm" "github.com/kataras/iris" ) // EndPointCalls - Object for storing endpoints call details type EndPointCalls struct { gorm.Model EndPointID uint \`gorm:"index;not null"\` RequestIP string ResponseCode int } // SaveCall - Save the call details of an endpoint func (ep EndPoints) SaveCall(context iris.Context) EndPointCalls { epCall := EndPointCalls{ EndPointID: ep.ID, RequestIP: context.RemoteAddr(), ResponseCode: context.GetStatusCode(), } db.Create(&epCall) return epCall } As shown above, our `EndPointCalls` model defines a `SaveCall` method, which stores the requesting IP address and the response code of an existing `EndPoint` object. Finally, we will update the model migration in the `index.go` file to include our new models: // index.go // ... func main() { // ... // Initialize ORM and auto migrate models db, \_ := gorm.Open("sqlite3", "./db/gorm.db") db.AutoMigrate(&models.Movies{}, &models.EndPoints{}, &models.EndPointCalls{}) // ... } ### Saving endpoint data for monitoring Using our newly created models, we will edit the `MoviesController.go` file to save relevant data when an endpoint is called. To do this, we will add a private helper method to `MoviesController.go`, which will save endpoint data with our models. See how below: // MoviesController.go // ... func (m MoviesController) saveEndpointCall(name string) { endpoint := models.EndPoints{ Name: name, URL: m.Cntx.Path(), Type: m.Cntx.Request().Method, } endpoint = endpoint.SaveOrCreate() endpointCall := endpoint.SaveCall(m.Cntx) } The `saveEndpointCall` method takes the name of the endpoint as a parameter. Using the controller’s `iris.Context` instance, it reads and saves the endpoint path and request method. Now that this helper method is available, we will call it in each of the endpoint methods in the `MoviesController.go` file: // MoviesController.go // ... // Get - get a list of all available movies func (m MoviesController) Get() { movie := models.Movies{} movies := movie.Get() go m.saveEndpointCall("Movies List") m.Cntx.JSON(iris.Map{"status": "success", "data": movies}) } // GetByID - Get movie by ID func (m MoviesController) GetByID(ID int64) { movie := models.Movies{} movie = movie.GetByID(ID) if !movie.Validate() { msg := fmt.Sprintf("Movie with ID: %v not found", ID) m.Cntx.StatusCode(iris.StatusNotFound) m.Cntx.JSON(iris.Map{"status": "error", "message": msg}) } else { m.Cntx.JSON(iris.Map{"status": "success", "data": movie}) } name := fmt.Sprintf("Single Movie with ID: %v Retrieval", ID) go m.saveEndpointCall(name) } // ... As shown in the snippet above, the `saveEndpointCall` helper method will be called in each CRUD method. > _💡 The_ `_saveEndpointCall_` _method is called as a_ [_Goroutine_](https://golangbot.com/goroutines/)_. Calling it this way calls it concurrently with the execution of the endpoint’s method, and allows our monitoring code to not delay or inhibit the response of the API._ ### Creating the endpoint monitor dashboard Now that we have implemented monitoring our API’s calls, we will display the data we have accrued on a dashboard. ### Registering our template engine The GoLang framework, Iris, has the ability to implement a range of template engines, which we will take advantage of. In this section, we will implement the **Handlebars** template engine, and in our `index.go` file, we will register it to the app instance: // index.go package main import ( "goggles/controllers" "goggles/models" "github.com/jinzhu/gorm" "github.com/kataras/iris" ) func main() { app := iris.New() tmpl := iris.Handlebars("./templates", ".html") app.RegisterView(tmpl) // ... app.Run(iris.Addr("127.0.0.1:1234")) } > _💡 We have defined our template engine (Handlebars), to render_ `_.html_` _files contained in the_ `_templates_`_directory._ ### Creating the dashboard’s route and controller Now that we have registered our template engine to the application, we will add a route in `index.go` to render our API monitor dashboard: // index.go // ... func main() { app := iris.New() // ... app.Get("/admin/endpoints", func(ctx iris.Context) { dashBoard := controllers.DashBoardController{Cntx: ctx} dashBoard.ShowEndpoints() }) app.Run(iris.Addr("127.0.0.1:1234")) } Above, we have added definitions for the path /admin/endpoints, where we intend to render details of our API endpoints and its calls. We have also specified that the route should be handled by the ShowEndpoints method of DashBoardController. To create DashBoardController, we will create a DashBoardController.go file in the controllers directory. And in our DashBoardController.go file, we will define the DashBoardController object and its ShowEndpoints method: // DashBoardController.go package controllers import ( "goggles/models" "github.com/kataras/iris" "github.com/kataras/iris/mvc" ) // DashBoardController - Controller object for Endpoints dashboard type DashBoardController struct { mvc.BaseController Cntx iris.Context } // ShowEndpoints - show list of endpoints func (d DashBoardController) ShowEndpoints() { endpoints := (models.EndPoints{}).GetWithCallSummary() d.Cntx.ViewData("endpoints", endpoints) d.Cntx.View("endpoints.html") } In `ShowEndpoints()`, we retrieve our endpoints and a summary of their calls for display. Then we pass this data to our view using `d.Cntx.ViewData("endpoints", endpoints)`, and finally we render our view file `templates/endpoints.html` using `d.Cntx.View("endpoints.html")`. ### Retrieving endpoints and call summaries To retrieve our list of endpoints and a summary of their calls, we will create a method in the `EndPoints.go` file called `GetWithCallSummary`. Our `GetWithCallSummary` method should return the endpoints and their call summaries ready for display. For this, we will define a collection object `EndPointWithCallSummary` with the attributes we need for our display in the `EndPoints.go` file: // EndPoints.go package models import ( "github.com/jinzhu/gorm" ) // EndPoints - endpoint model type EndPoints struct { gorm.Model Name, URL string Type string \`gorm:"DEFAULT:'GET'"\` Calls \[\]EndPointCalls \`gorm:"ForeignKey:EndPointID"\` } // EndPointWithCallSummary - Endpoint with last call summary type EndPointWithCallSummary struct { ID uint Name, URL string Type string LastStatus int NumRequests int LastRequester string } And then define `GetWithCallSummary` method to use it as follows: // EndPoints.go // ... // GetWithCallSummary - get all endpoints with call summary details func (ep EndPoints) GetWithCallSummary() \[\]EndPointWithCallSummary { var eps \[\]EndPoints var epsWithDets \[\]EndPointWithCallSummary db.Preload("Calls").Find(&eps) for \_, elem := range eps { calls := elem.Calls lastCall := calls\[len(calls)-1:\]\[0\] newElem := EndPointWithCallSummary{ elem.ID, elem.Name, elem.URL, elem.Type, lastCall.ResponseCode, len(calls), lastCall.RequestIP, } epsWithDets = append(epsWithDets, newElem) } return epsWithDets } // ... Above, the `GetWithCallSummary` method leverages the `Calls` attribute of `EndPoints`, which defines its relationship with `EndPointCalls`. When retrieving our list of endpoints from the database, we eager load its `EndPointCalls` data using `db.Preload("Calls").Find(&eps)`. For more information on eager loading in GORM, see the [documentation](http://jinzhu.me/gorm/crud.html#preloading-eager-loading). `GetWithCallSummary` initializes an array of `EndPointWithCallSummary`, and loops through the `EndPoints`objects returned from our database to create `EndPointWithCallSummary` objects. These `EndPointWithCallSummary` objects are appended to the initialized array and returned. > _💡 The_ `_EndPointWithCallSummary_` _is not a model. It is a collection object and does not need to have a table in our database. This is why it does not have its own file and is not passed to_ `_index.go_` _for migration._ ### Implementing the dashboard and displaying data Now that we have the dashboard’s route, controller and data for display, we will implement the dashboard view to achieve a simple list display of endpoints and their summary data. Let’s update `templates/endpoints.html` to have the following code: Endpoints Monitor Dashboard Goggles - A Real-Time API Monitor {{#each endpoints}} {{name}} {{type}}: {{url}} | Last Status: {{LastStatus}} | Times Called: {{NumRequests}} | Last Request IP: {{LastRequester}} {{/each}} Above, we render our endpoints list using [Bootstrap](https://getbootstrap.com/) and our Handlebars template engine. We have also created and used a template function `status_class`, to colour code our list based on their last call status `LastStatus`. We define the `status_class` template function in `index.go` after initialising our template engine: // index.go // ... func main() { app := iris.New() tmpl := iris.Handlebars("./templates", ".html") tmpl.AddFunc("status\_class", func(status int) string { if status >= 200 && status = 300 && status = 400 { return "danger" } return "success" }) app.RegisterView(tmpl) } Then in our view file we call the function as: class="list-group-item list-group-item-{{status_class LastStatus}}" > _💡 In the above LastStatus is the function’s parameter._ ### Adding realtime updates to our dashboard So far in this article, we have monitored the calls to an API and displayed the data via a dashboard. We will now use [Pusher](http://pusher.com/) to provide realtime data updates to our dashboard. ### Sending realtime data from the backend Earlier, we installed the [Pusher Go library](https://github.com/pusher/pusher-http-go), which we will use to trigger an event when an endpoint is called. In the `MoviesController.go` file, where the API requests are handled, we will initialize the Pusher client: // MoviesController.go package controllers import ( // ... "github.com/pusher/pusher-http-go" ) // MoviesController - controller object to serve movie data type MoviesController struct { mvc.BaseController Cntx iris.Context } var client = pusher.Client{ AppId: "app_id", Key: "app_key", Secret: "app_secret", Cluster: "app_cluster", } // ... Here, we have initialized the Pusher client using the credentials from our earlier created app. > _⚠️ Replace_ `_app_id, app_key, app_secret and app_cluster_` _with your app credentials._ Next, we will use our Pusher client to trigger an event, which would include the endpoint’s data to be displayed in our view. We will do this in the `saveEndpointCall` method, which logs an endpoint and its call: // MoviesController.go // ... func (m MoviesController) saveEndpointCall(name string) { endpoint := models.EndPoints{ Name: name, URL: m.Cntx.Path(), Type: m.Cntx.Request().Method, } endpoint = endpoint.SaveOrCreate() endpointCall := endpoint.SaveCall(m.Cntx) endpointWithCallSummary := models.EndPointWithCallSummary{ ID: endpoint.ID, Name: endpoint.Name, URL: endpoint.URL, Type: endpoint.Type, LastStatus: endpointCall.ResponseCode, NumRequests: 1, LastRequester: endpointCall.RequestIP, } client.Trigger("goggles_channel", "new_endpoint_request", endpointWithCallSummary) } Above, we create an `EndPointWithCallSummary` object from `EndPoints` (the endpoint) and `EndPointCalls`. This `EndPointWithCallSummary` object has all the data required for display on the dashboard, so will be passed to Pusher for transmission. ### Displaying data in realtime on the dashboard To display the realtime updates of our endpoints, we will use the Pusher JavaScript client and jQuery libraries. In our view file, `templates/endpoints.html`, we will first import and initialize a Pusher instance using our app’s credentials: const pusher = new Pusher('app_id', {cluster: "app_cluster"}); > _⚠️ Replace_ `_app_id and app_cluster_` _with values from your app’s credentials._ Next, we will define the following: * The template for adding new endpoints to our view. * The functions to append a new endpoint and get the status class of the endpoint. Finally, we will subscribe to the `goggles_channel` and listen to the `new_endpoint_request` event, where our endpoint updates will be transmitted: // ... const channel = pusher.subscribe("goggles_channel"); channel.bind('new_endpoint_request', function(data) { let end_point_id = data.ID; if ( $('#endpoint-' + end_point_id).length > 0 ) { let status_class = getItemStatusClass(data['LastStatus']), endpoint = $('#endpoint-' + end_point_id); let calls = 1 * endpoint.find('span.times_called').text() endpoint.find('span.last_status').text(data['LastStatus']); endpoint.find('span.times_called').text( (calls + 1) ) endpoint.removeClass('list-group-item-success'); endpoint.removeClass('list-group-item-danger'); endpoint.removeClass('list-group-item-warning'); endpoint.addClass('list-group-item-' + status_class); } else { addNewEndPoint(data); } }); // ... In the `new_endpoint_request` event handler, the endpoint data is categorized into either an update scenario (where the endpoint already exists on the dashboard) or a create scenario (where a new list item is created and appended). Finally, you can build your application and when you run it you should see something similar to what we have in the preview: ![](https://hackernoon.com/images/1*uxk2Dhe1apJP8iKvRbmh0A.gif) ### Conclusion In this article, we were able to monitor the realtime requests to a REST API and demonstrate how Pusher works with GoLang applications. > This post first appeared on the [Pusher blog](https://blog.pusher.com/realtime-trade-platform-javascript-pusher/).

Real-time API Monitoring Using GO and Pusher

Too Long; Didn't Read

Companies Mentioned

@neo

RELATED STORIES