Disclosure: , which provide real-time APIs for developers, has previously sponsored Hacker Noon. Pusher A basic understanding of Go and JavaScript is needed to follow this tutorial. REST is a popular architectural style for providing standards between computer systems on the web, making it easier for systems to communicate with each other. It is mostly used by APIs to provide data to other systems requiring them. Sometimes, the providers of APIs would like to monitor its use. Monitoring APIs helps provide useful information, such as which endpoints are called most frequently, or what regions are the largest audience using request IP Addresses. This information can then be used to optimize the API. In this article, we will implement realtime monitoring of a small API built with GoLang, using Pusher. Here’s a preview of what it should look like at the end: Requirements To follow along with this article, you will need the following: An IDE of your choice e.g. . Visual Studio Code installed on your computer. Go Basic knowledge of GoLang. Basic knowledge of JavaScript (ES6 syntax) and jQuery. Basic knowledge of using a CLI tool or terminal. Once you have all the above requirements, let’s proceed. Setting up our codebase To keep things simple, we’ll be using an already written GoLang CRUD API, which is available on . We will fork the repository and set it up following the guidelines on installation. GitHub README.md Next, we will set up Pusher in the API project. Pusher is a service that provides a simple implementation of realtime functionality for our web and mobile applications. We will use it in this article, to provide realtime updates to our API monitor dashboard. Let’s head over to Pusher.com, you can if you don’t already have one. On the , create a new app and copy out the app credentials (App ID, Key, Secret, and Cluster). We will use these credentials in our API. create a free account dashboard Now that we have our Pusher app, we will install the Pusher Go library by running: $ go get github.com/pusher/pusher-http-go Monitoring our API We have so far set up a functional CRUD API, and we will now implement monitoring calls to it. In this article, we will monitor: The endpoints called with details like name, request type (GET, POST, etc) and URL. For each call to an endpoint, we will also take note of: The requesting IP address remove, The response status code for the particular call. Now that we have defined what to monitor, we will begin by creating models to keep track of the data we acquire. Creating models for monitoring Based on our specifications above, we will create two new model files and . As was used in the base API, we will use the (the GoLang ORM) for managing data storage. EndPoints.go EndPointCalls.go GORM 💡 Our new model files will exist in the models directory and belong to the models package. In , we will define the object and a method to save endpoints: EndPoints.go EndPoints package models import ("github.com/jinzhu/gorm") // EndPoints - endpoint modeltype EndPoints struct {gorm.ModelName, URL stringType string `gorm:"DEFAULT:'GET'"`Calls []EndPointCalls `gorm:"ForeignKey:EndPointID"`} // SaveOrCreate - save endpoint calledfunc (ep EndPoints) SaveOrCreate() EndPoints {db.FirstOrCreate(&ep, ep)return ep} In the code block above, our model did not re-initialize the GORM instance , yet it was used. This is because the instance defined in the file was global to all members of the package, and so it can be referenced and used by all members of . db Movies.go package models 💡 Our EndPoints model has an attribute _Calls_ which is an array of _EndPointCalls_ objects. This attribute signifies the one to many relationship between _EndPoints_ and _EndPointCalls_ . For more information on model associations and relationships see the GORM documentation . Next, we’ll fill in the model definitions and methods for our model in the file: EndPointCalls EndPointCalls.go package models import ("github.com/jinzhu/gorm""github.com/kataras/iris") // EndPointCalls - Object for storing endpoints call detailstype EndPointCalls struct {gorm.ModelEndPointID uint `gorm:"index;not null"`RequestIP stringResponseCode int} // SaveCall - Save the call details of an endpointfunc (ep EndPoints) SaveCall(context iris.Context) EndPointCalls {epCall := EndPointCalls{EndPointID: ep.ID,RequestIP: context.RemoteAddr(),ResponseCode: context.GetStatusCode(),} db.Create(&epCall) return epCall } As shown above, our model defines a method, which stores the requesting IP address and the response code of an existing object. EndPointCalls SaveCall EndPoint Finally, we will update the model migration in the file to include our new models: index.go // index.go// ... func main() {// ... // Initialize ORM and auto migrate models db, \_ := gorm.Open("sqlite3", "./db/gorm.db") db.AutoMigrate(&models.Movies{}, &models.EndPoints{}, &models.EndPointCalls{}) // ... } Saving endpoint data for monitoring Using our newly created models, we will edit the file to save relevant data when an endpoint is called. MoviesController.go To do this, we will add a private helper method to , which will save endpoint data with our models. See how below: MoviesController.go // MoviesController.go// ... func (m MoviesController) saveEndpointCall(name string) {endpoint := models.EndPoints{Name: name,URL: m.Cntx.Path(),Type: m.Cntx.Request().Method,} endpoint = endpoint.SaveOrCreate() endpointCall := endpoint.SaveCall(m.Cntx) } The method takes the name of the endpoint as a parameter. Using the controller’s instance, it reads and saves the endpoint path and request method. saveEndpointCall iris.Context Now that this helper method is available, we will call it in each of the endpoint methods in the file: MoviesController.go // MoviesController.go// ... // Get - get a list of all available moviesfunc (m MoviesController) Get() {movie := models.Movies{}movies := movie.Get() go m.saveEndpointCall("Movies List") m.Cntx.JSON(iris.Map{"status": "success", "data": movies}) } // GetByID - Get movie by IDfunc (m MoviesController) GetByID(ID int64) {movie := models.Movies{}movie = movie.GetByID(ID)if !movie.Validate() {msg := fmt.Sprintf("Movie with ID: %v not found", ID)m.Cntx.StatusCode(iris.StatusNotFound)m.Cntx.JSON(iris.Map{"status": "error", "message": msg})} else {m.Cntx.JSON(iris.Map{"status": "success", "data": movie})} name := fmt.Sprintf("Single Movie with ID: %v Retrieval", ID) go m.saveEndpointCall(name) } // ... As shown in the snippet above, the helper method will be called in each CRUD method. saveEndpointCall 💡 The _saveEndpointCall_ method is called as a Goroutine . Calling it this way calls it concurrently with the execution of the endpoint’s method, and allows our monitoring code to not delay or inhibit the response of the API. Creating the endpoint monitor dashboard Now that we have implemented monitoring our API’s calls, we will display the data we have accrued on a dashboard. Registering our template engine The GoLang framework, Iris, has the ability to implement a range of template engines, which we will take advantage of. In this section, we will implement the template engine, and in our file, we will register it to the app instance: Handlebars index.go // index.gopackage main import ("goggles/controllers""goggles/models""github.com/jinzhu/gorm""github.com/kataras/iris") func main() {app := iris.New() tmpl := iris.Handlebars("./templates", ".html") app.RegisterView(tmpl) // ... app.Run(iris.Addr("127.0.0.1:1234")) } 💡 We have defined our template engine (Handlebars), to render _.html_ files contained in the _templates_ directory. Creating the dashboard’s route and controller Now that we have registered our template engine to the application, we will add a route in to render our API monitor dashboard: index.go // index.go// ... func main() {app := iris.New() // ... app.Get("/admin/endpoints", func(ctx iris.Context) { dashBoard := controllers.DashBoardController{Cntx: ctx} dashBoard.ShowEndpoints() }) app.Run(iris.Addr("127.0.0.1:1234")) } Above, we have added definitions for the path /admin/endpoints, where we intend to render details of our API endpoints and its calls. We have also specified that the route should be handled by the ShowEndpoints method of DashBoardController. To create DashBoardController, we will create a DashBoardController.go file in the controllers directory. And in our DashBoardController.go file, we will define the DashBoardController object and its ShowEndpoints method: // DashBoardController.gopackage controllers import ("goggles/models""github.com/kataras/iris""github.com/kataras/iris/mvc") // DashBoardController - Controller object for Endpoints dashboardtype DashBoardController struct {mvc.BaseControllerCntx iris.Context} // ShowEndpoints - show list of endpointsfunc (d DashBoardController) ShowEndpoints() {endpoints := (models.EndPoints{}).GetWithCallSummary()d.Cntx.ViewData("endpoints", endpoints)d.Cntx.View("endpoints.html")} In , we retrieve our endpoints and a summary of their calls for display. Then we pass this data to our view using , and finally we render our view file using . ShowEndpoints() d.Cntx.ViewData("endpoints", endpoints) templates/endpoints.html d.Cntx.View("endpoints.html") Retrieving endpoints and call summaries To retrieve our list of endpoints and a summary of their calls, we will create a method in the file called . EndPoints.go GetWithCallSummary Our method should return the endpoints and their call summaries ready for display. For this, we will define a collection object with the attributes we need for our display in the file: GetWithCallSummary EndPointWithCallSummary EndPoints.go // EndPoints.gopackage models import ("github.com/jinzhu/gorm") // EndPoints - endpoint modeltype EndPoints struct {gorm.ModelName, URL stringType string `gorm:"DEFAULT:'GET'"`Calls []EndPointCalls `gorm:"ForeignKey:EndPointID"`} // EndPointWithCallSummary - Endpoint with last call summarytype EndPointWithCallSummary struct {ID uintName, URL stringType stringLastStatus intNumRequests intLastRequester string} And then define method to use it as follows: GetWithCallSummary // EndPoints.go // ... // GetWithCallSummary - get all endpoints with call summary detailsfunc (ep EndPoints) GetWithCallSummary() []EndPointWithCallSummary {var eps []EndPointsvar epsWithDets []EndPointWithCallSummary db.Preload("Calls").Find(&eps) for \_, elem := range eps { calls := elem.Calls lastCall := calls\[len(calls)-1:\]\[0\] newElem := EndPointWithCallSummary{ elem.ID, elem.Name, elem.URL, elem.Type, lastCall.ResponseCode, len(calls), lastCall.RequestIP, } epsWithDets = append(epsWithDets, newElem) } return epsWithDets } // ... Above, the method leverages the attribute of , which defines its relationship with . When retrieving our list of endpoints from the database, we eager load its data using . GetWithCallSummary Calls EndPoints EndPointCalls EndPointCalls db.Preload("Calls").Find(&eps) For more information on eager loading in GORM, see the . documentation initializes an array of , and loops through the objects returned from our database to create objects. GetWithCallSummary EndPointWithCallSummary EndPoints EndPointWithCallSummary These objects are appended to the initialized array and returned. EndPointWithCallSummary 💡 The _EndPointWithCallSummary_ is not a model. It is a collection object and does not need to have a table in our database. This is why it does not have its own file and is not passed to _index.go_ for migration. Implementing the dashboard and displaying data Now that we have the dashboard’s route, controller and data for display, we will implement the dashboard view to achieve a simple list display of endpoints and their summary data. Let’s update to have the following code: templates/endpoints.html <!-- templates/endpoints.html --><!DOCTYPE html><html><head><title>Endpoints Monitor Dashboard</title><link rel="stylesheet" type="text/css" href=" " /></head><body><div><nav class="navbar navbar-default navbar-static-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href=" ">Goggles - A Real-Time API Monitor</a></div></div></nav><div class="container"><div class="row"><div class="col-xs-12 col-lg-12"><div class="endpoints list-group">{{#each endpoints}}<a id="endpoint-{{ID}}" href="#" class="list-group-itemlist-group-item-{{status_class LastStatus}}"><strong>{{name}}</strong><span class="stats">{{type}}: <strong>{{url}}</strong> |Last Status: <span class="last_status">{{LastStatus}}</span> |Times Called: <span class="times_called">{{NumRequests}}</span> |Last Request IP: <span class="request_ip">{{LastRequester}}</span></span></a>{{/each}}</div></div></div></div></div><script src=" "></script><script src=" "></script></body></html> https://cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/4.0.0-beta.3/css/bootstrap.min.css http://127.0.0.1:1234/ https://cdnjs.cloudflare.com/ajax/libs/jquery/3.2.1/jquery.min.js https://cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/4.0.0-beta.3/js/bootstrap.min.js Above, we render our endpoints list using and our Handlebars template engine. We have also created and used a template function , to colour code our list based on their last call status . Bootstrap status_class LastStatus We define the template function in after initialising our template engine: status_class index.go // index.go // ... func main() {app := iris.New() tmpl := iris.Handlebars("./templates", ".html") tmpl.AddFunc("status\_class", func(status int) string { if status >= 200 && status < 300 { return "success" } else if status >= 300 && status < 400 { return "warning" } else if status >= 400 { return "danger" } return "success" }) app.RegisterView(tmpl) } Then in our view file we call the function as: class="list-group-item list-group-item-{{status_class LastStatus}}" 💡 In the above LastStatus is the function’s parameter. Adding realtime updates to our dashboard So far in this article, we have monitored the calls to an API and displayed the data via a dashboard. We will now use to provide realtime data updates to our dashboard. Pusher Sending realtime data from the backend Earlier, we installed the , which we will use to trigger an event when an endpoint is called. In the file, where the API requests are handled, we will initialize the Pusher client: Pusher Go library MoviesController.go // MoviesController.go package controllers import ( // ... "github.com/pusher/pusher-http-go" ) // MoviesController - controller object to serve movie data type MoviesController struct { mvc.BaseController Cntx iris.Context } var client = pusher.Client{ AppId: "app_id", Key: "app_key", Secret: "app_secret", Cluster: "app_cluster", } // ... Here, we have initialized the Pusher client using the credentials from our earlier created app. ⚠️ Replace _app_id, app_key, app_secret and app_cluster_ with your app credentials. Next, we will use our Pusher client to trigger an event, which would include the endpoint’s data to be displayed in our view. We will do this in the method, which logs an endpoint and its call: saveEndpointCall // MoviesController.go // ... func (m MoviesController) saveEndpointCall(name string) { endpoint := models.EndPoints{ Name: name, URL: m.Cntx.Path(), Type: m.Cntx.Request().Method, } endpoint = endpoint.SaveOrCreate() endpointCall := endpoint.SaveCall(m.Cntx) endpointWithCallSummary := models.EndPointWithCallSummary{ ID: endpoint.ID, Name: endpoint.Name, URL: endpoint.URL, Type: endpoint.Type, LastStatus: endpointCall.ResponseCode, NumRequests: 1, LastRequester: endpointCall.RequestIP, } client.Trigger("goggles_channel", "new_endpoint_request", endpointWithCallSummary) } Above, we create an object from (the endpoint) and . This object has all the data required for display on the dashboard, so will be passed to Pusher for transmission. EndPointWithCallSummary EndPoints EndPointCalls EndPointWithCallSummary Displaying data in realtime on the dashboard To display the realtime updates of our endpoints, we will use the Pusher JavaScript client and jQuery libraries. In our view file, , we will first import and initialize a Pusher instance using our app’s credentials: templates/endpoints.html <!-- endpoints.html --> <script src="//cdnjs.cloudflare.com/ajax/libs/jquery/3.2.1/jquery.min.js"></script> <script src="//cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/4.0.0-beta.3/js/bootstrap.min.js"></script> <script src="//js.pusher.com/4.1/pusher.min.js"></script> <script> const pusher = new Pusher('app_id', {cluster: "app_cluster"}); </script> ⚠️ Replace _app_id and app_cluster_ with values from your app’s credentials. Next, we will define the following: The template for adding new endpoints to our view. The functions to append a new endpoint and get the status class of the endpoint. Finally, we will subscribe to the and listen to the event, where our endpoint updates will be transmitted: goggles_channel new_endpoint_request <!-- endpoints.html --> <script> // ... const channel = pusher.subscribe("goggles_channel"); channel.bind('new_endpoint_request', function(data) { let end_point_id = data.ID; if ( $('#endpoint-' + end_point_id).length > 0 ) { let status_class = getItemStatusClass(data['LastStatus']), endpoint = $('#endpoint-' + end_point_id); let calls = 1 * endpoint.find('span.times_called').text() endpoint.find('span.last_status').text(data['LastStatus']); endpoint.find('span.times_called').text( (calls + 1) ) endpoint.removeClass('list-group-item-success'); endpoint.removeClass('list-group-item-danger'); endpoint.removeClass('list-group-item-warning'); endpoint.addClass('list-group-item-' + status_class); } else { addNewEndPoint(data); } }); // ... In the event handler, the endpoint data is categorized into either an update scenario (where the endpoint already exists on the dashboard) or a create scenario (where a new list item is created and appended). new_endpoint_request Finally, you can build your application and when you run it you should see something similar to what we have in the preview: Conclusion In this article, we were able to monitor the realtime requests to a REST API and demonstrate how Pusher works with GoLang applications. This post first appeared on the . Pusher blog
