How Goji Investments Enhances Developer Experience via Observability

This post was written by Dean Record, Engineer at Goji Investments.

Creating an Excellent Developer Experience

In the four years since we launched, Goji Investments’ developers have worked hard to help us expand our product offering and bring in new customers. In return, we’ve done everything to keep them happy, and that means supplying them with the right tools to help them transition from building our platform to scaling it.

Our DevOps pipeline is a complex machine with many moving parts. We are continuously refining it and adding or subtracting tools based on developer feedback and our changing deliverables.

We realised early on that the best way to ensure a superb end-user experience is to provide an outstanding developer experience.

We help our people stay focused on performing at the highest level by automating repetitive low-level tasks, thus freeing them to work on features that add value to the company. To this end, we have fine-tuned our technology stack to give our eight-member developer team everything it needs to succeed.

Automating Deployments Wasn’t Enough

Our applications are written in a modern Java stack, and we are using Dropwizard as our framework. Our developers typically write code and then push it to GitHub for review. If it’s approved, the new code is merged into master, which automatically triggers a build pipeline using ThoughtWorks GoCD. We then run automated unit, integration, and acceptance tests and our developers also have the option to run manual tests to encourage exploratory testing.

From there, we can promote new code into our pre-production environment, which mirrors our current production setup. Anything not covered in prior testing that makes its way into our production environment can be easily rolled back using the blue-green deployment functionality of HashiCorp Nomad, our container orchestration platform. We’re a small team, and our codebase is relatively compact, so we don’t need the complexity of Kubernetes GKE or EKS to deploy and manage our applications.

Combing through logs for application monitoring is a waste of valuable developer resources.

One of the benefits of our technology stack is extensive automation. Although our deployment workflow was seamless, things were not as rosy when it came to monitoring. We were using New Relic to monitor our infrastructure. When an application was eating up too many CPU cycles, or an EC2 instance crashed and rebooted, it would trigger an alert. But if we wanted to troubleshoot slow response times, latency errors, and bottlenecks at the service level, we had to comb through logs manually, and that was a waste of time and human resources.

End-to-End Traceability

The last thing you want to do with your day is analyse log files. You want to build the next killer feature, but if you have to dive into the weeds, you want to do it as quickly and as painlessly as possible. That’s where observability and application performance monitoring (APM) comes in. APM incorporates infrastructure monitoring, but also provides service-level metrics, including information about database calls and end-user activity. It allows us to locate bottlenecks and other performance issues without having to divert our developers from their primary duties.

I’d used APM software at my previous position, and when it came to time to adopt a platform at Goji Investments, I went with what I knew. Experience had shown me that Instana is a zero-touch solution that does everything we need straight out of the box. When you run 20 different microservices that cover everything from taxes, payments, and customer notification—and when you’re scaling to add even more—you need to optimise every element of your DevOps pipeline. Instana was the final piece in our puzzle, and it allowed us to automate one of the most cumbersome parts of the development cycle.

Instana gives us full traceability from our front-end to our back-end. We can track user actions on our website and trace subsequent calls across multiple services. It automates alerts and can send an alarm to our developers’ Slack channel when it encounters an issue with CPU, container memory usage, or the JVM heap size.

Instana also integrates with Humio, a real-time logging solution we recently adopted. When we spot an issue in Humio, we can jump into Instana and go directly to the Docker container or host that generated the error. We can also view AWS metrics in Instana and no longer have to pull information from the AWS console. Consolidating all of these functions saves time and simplifies troubleshooting.

Finding Our Bottlenecks

When we’re testing a new build or functionality, we can use Instana to view dozens of metrics in real-time, and that is a complete game-changer. Recently, we developed a feature that distributes dividends to investors and, prior to release we wanted to run performance tests to ensure it would scale to meet the demands of our growing client base. Initially, we ran a performance test of several hundred thousands of payments, and it took over 20 hours.

During the performance tests as we iterated on our code, we found ourselves getting halfway through the process in five or ten minutes, hitting a bottleneck, and then taking 10 to 12 hours to complete the remainder.

With Instana, we were able to see where the problem occurred. The instant Java profiling feature, AutoProfile™, showed us hotspots where the code was slow, and we determined that our database queries were the bottleneck. Once we knew where to look, we went back, fixed the code, and redeployed the application until we got it right.

Instana allowed us to compare test runs. We could pull up a previous trial run and then compare its timeframe with the current version of the code. Using this iterative process, we streamlined the code, eliminated all the bottlenecks, and reduced the dividend processing time from 20 hours to less than 1 hour.

If we didn’t have Instana, it would have easily taken us four to five times longer to get this process right.

The key to developer experience: reward them with creative work and give them every opportunity to shine.

If there’s one thing developers love, it’s rising to the occasion. The best way to keep them happy is to give them the tools to contribute to your bottom line. Treat them like internal customers by relieving their pain points. Automate repetitive tasks and eliminate tedious, time-consuming work like going through error logs. Reward them with creative work and give them every opportunity to shine.

Instana is helping Goji Investments streamline the DevOps process by giving us visibility into the inner workings of our applications and microservices. We can trace calls and database queries, and we can monitor hosts, Docker containers, and Java Virtual Machines in real-time. All of this accelerates testing and deployment. We can do more and scale faster and attract the world’s most talented developers.

After all, the smartest people want to work for companies that challenge them and also give them the tools to shine.

If there’s one thing developers love, it’s rising to the occasion. The best way to keep them happy is to give them the tools to contribute to your bottom line. Treat them like internal customers by relieving their pain points. Automate repetitive tasks and eliminate tedious, time-consuming work like going through error logs. Reward them with creative work and give them every opportunity to shine.

Instana is helping Goji Investments streamline the DevOps process by giving us visibility into the inner workings of our applications and microservices. We can trace calls and database queries, and we can monitor hosts, Docker containers, and Java Virtual Machines in real-time. All of this accelerates testing and deployment. We can do more and scale faster and attract the world’s most talented developers.

After all, the smartest people want to work for companies that challenge them and also give them the tools to shine.

If there’s one thing developers love, it’s rising to the occasion. The best way to keep them happy is to give them the tools to contribute to your bottom line. Treat them like internal customers by relieving their pain points. Automate repetitive tasks and eliminate tedious, time-consuming work like going through error logs. Reward them with creative work and give them every opportunity to shine.

Instana is helping Goji Investments streamline the DevOps process by giving us visibility into the inner workings of our applications and microservices. We can trace calls and database queries, and we can monitor hosts, Docker containers, and Java Virtual Machines in real-time. All of this accelerates testing and deployment. We can do more and scale faster and attract the world’s most talented developers.

After all, the smartest people want to work for companies that challenge them and also give them the tools to shine.

Dean Record is an Engineer at Goji Investments.

You can connect with him on LinkedIn.

P.S. You can test your observability skills with this interactive observability sandbox!

Previously published at https://www.instana.com/blog/from-shipping-to-scaling-how-goji-investments-masters-developer-experience-through-observability/