In this article, I will explain how I built COINDATAX, a cryptocurrency analytics platform to help investors analyze the market, and explain why we choose to go serverless. I will also explain our biggest pain points with AWS Lambda, and how Dashbird helped us with Lambda performance monitoring. Why serverless? When my co-founder and I decided to build a cryptocurrency web application, we immediately thought about using AWS Lambda for our integrations. After all, since we were a small team trying to create a new product, Having to select the ideal instance type, configure auto-scaling policies, and creating a deployment pipeline takes a significant amount of work, which we simply could not afford. we didn't want to spend too much time managing AWS servers. In addition, since we would like to add as many integrations as possible to our dashboards, With traditional server-based architectures, your environment scales more like a "step" function --- if, say, each instance can handle 100 clients, when you reach 101 visitors you need to spawn a new machine that will be idle most of the time. With serverless architecture, your application scales, up or down, according to each request. we wanted our application to scale linearly with product usage. Another benefit of Lambda, or more specifically of the , is infrastructure as code. With serverless, you know exactly what is going on with your infrastructure simply by looking at a configuration file. Although you can do this for regular server-based architectures, using tools such as , you usually need to think of it as something extra to your infrastructure design, while most AWS applications built on top of Lambda will use serverless from the beginning. Serverless framework Terraform Now speaking about some of the drawbacks: First of all, AWS Lambda has some inherent limitations. You can't have, for example, a long-running or expensive process. in order to use serverless. It might be necessary to use Step Functions or to break your lambdas into smaller functions so that you can pass the processed output from one to the other. If your functions last longer than 5 minutes or if they need more than 1 MB of RAM, you need to rework your architecture Secondly, Since you don't have direct access to the instance where your code is running and since AWS CloudWatch is somewhat limited in its monitoring capabilities, sometimes things break and you have no idea of what happened. This is where Dashbird comes into play, but I'll talk more about that in a moment. Lambda functions are indisputably harder to debug. Taking all that into consideration, we decided to move forward with serverless, as we believed these limitations could be managed. Architecture The architecture behind COINDATAX is pretty straightforward. We pull data from a number of external APIs, then we do some cleaning and processing to be sure that all data is consistent and reliable, and finally, we store the events in our databases. These so that we have up to date crypto information in our system. data extractors are triggered via CloudWatch events at fixed intervals In this article, I will explain how to create a monitoring platform that implements the first step of our product, the More specifically, we will go through the following: data gathering of one of an external API. Setup of the development environment (Node.js, serverless) Development of the API integration Deployment of the application Monitoring of the Lambda functions with Dashbird 1. Setup of the development environment Install serverless $ npm i -g serverless Configure the file. In this example we are using Node.js 6 and we have only one function that is triggered every 5 minutes. serverless.yml service: coindatax-dashbird-demo

provider:
  name: aws
  runtime: nodejs6.10

functions:
  coinmarketcap:
    handler: functions/coinmarketcap.handler
    events:
      - schedule: rate(5 minutes) 2. Development of the API integration Our Lambda function is also simple, as the first step of a monitoring platform is to extract data from external APIs. Here we connect our function to CoinMarketCap's API and get the ticker information of the top coins for that period. // functions/coinmarketcap.js

const request = require('request-promise');
const uri = 'https://api.coinmarketcap.com/v1/ticker/'

function handler(event, context, callback) {
  const options = {
    uri,
    json: true
  }
  request(options)
    .then(data => callback(null, data)) // insert into database
    .catch(err => callback(err))
}

module.exports = {
  handler
} 3. Deployment of the application To deploy a serverless application, simply run $ serverless deploy -v And check out the output logs Serverless: Packaging service...
Serverless: Creating Stack...
Serverless: Checking Stack create progress...
CloudFormation - CREATE_IN_PROGRESS - AWS::CloudFormation::Stack - CoindataxDashDashbirdDashDemo-dev
CloudFormation - CREATE_IN_PROGRESS - AWS::S3::Bucket - ServerlessDeploymentBucket
CloudFormation - CREATE_IN_PROGRESS - AWS::S3::Bucket - ServerlessDeploymentBucket
CloudFormation - CREATE_COMPLETE - AWS::S3::Bucket - ServerlessDeploymentBucket
CloudFormation - CREATE_COMPLETE - AWS::CloudFormation::Stack - CoindataxDashDashbirdDashDemo-dev
Serverless: Stack create finished...
Serverless: Uploading CloudFormation file to S3...
Serverless: Uploading artifacts...
Serverless: Uploading service .zip file to S3 (17.23 MB)...
Serverless: Validating template...
Serverless: Updating Stack...
Serverless: Checking Stack update progress...
CloudFormation - UPDATE_IN_PROGRESS - AWS::CloudFormation::Stack - CoindataxDashDashbirdDashDemo-dev
CloudFormation - CREATE_IN_PROGRESS - AWS::IAM::Role - IamRoleLambdaExecution
CloudFormation - CREATE_IN_PROGRESS - AWS::Logs::LogGroup - CoinmarketcapLogGroup
CloudFormation - CREATE_IN_PROGRESS - AWS::IAM::Role - IamRoleLambdaExecution
CloudFormation - CREATE_IN_PROGRESS - AWS::Logs::LogGroup - CoinmarketcapLogGroup
CloudFormation - CREATE_COMPLETE - AWS::Logs::LogGroup - CoinmarketcapLogGroup
CloudFormation - CREATE_COMPLETE - AWS::IAM::Role - IamRoleLambdaExecution
CloudFormation - CREATE_IN_PROGRESS - AWS::Lambda::Function - CoinmarketcapLambdaFunction
CloudFormation - CREATE_IN_PROGRESS - AWS::Lambda::Function - CoinmarketcapLambdaFunction
CloudFormation - CREATE_COMPLETE - AWS::Lambda::Function - CoinmarketcapLambdaFunction
CloudFormation - UPDATE_COMPLETE_CLEANUP_IN_PROGRESS - AWS::CloudFormation::Stack - CoindataxDashDashbirdDashDemo-dev
CloudFormation - UPDATE_COMPLETE - AWS::CloudFormation::Stack - CoindataxDashDashbirdDashDemo-dev
Serverless: Stack update finished...
Service Information
service: CoindataxDashDashbirdDashDemo
stage: dev
region: us-east-1
api keys:
  None
endpoints:
  None
functions:
  coinmarketcap: CoindataxDashDashbirdDashDemo-dev-coinmarketcap

Stack Outputs
CoinmarketcapLambdaFunctionQualifiedArn: arn:aws:lambda:us-east-1:123456789012:function:CoindataxDashDashbirdDashDemo-dev-coinmarketcap:1
ServerlessDeploymentBucketName: coindataxdashdashbirddashdemo-dev-serverlessdeploymentbucket-abcdefgh1234 4. Monitoring of the Lambda functions with Dashbird After you have successfully deployed your Lambda function, Dashbird tries to fill that gap, with a dashboard that groups all your lambdas in a single place, a live tailing of your application logs and more. you quickly realize that AWS CloudWatch does not offer that many monitoring features for you to be in full control of your application. What I love the most about Dashbird is that it was super easy to set it up, and at the same time, it provided very useful insights to our team. I literally spent less than 5 minutes configuring it, and we were able to have a much better understanding of our architecture immediately. When it comes to Lambda functions, Dashbird provided us an aggregated performance and resource usage metrics for easy analysis, such as: Invocations - total count of: Successful requests Errors Cold starts Memory usage Average Maximum Minimum 99th percentile Duration Average Maximum Minimum 99th percentile Cost Aggregated billed execution time Learning outcome & saving money Because of Dashbird, we noticed that The change was very simple to implement, as all we did was to update the default parameter of serverless: all our lambdas were running with a third of the allocated memory size and that we could confidently reduce that threshold in order to reduce costs. memorySize provider:
  name: aws
  runtime: nodejs6.10
  memorySize: 512 # default is 1024 With one line of code, we , and we are constantly monitoring our application to see if we can reduce it even further. reduced our billing by 50% Conclusion Serverless architecture has undoubtedly many advantages to both small and big companies. Nevertheless, you should always take into account the unexpected debugging/monitoring time that you wouldn't have with traditional server-based systems. By using Dashbird, some of that hurdle can be reduced and maybe even eliminated, so that you end up with only the benefits of serverless and AWS Lambda.

Super

How to Use AWS S3 with Python 

An Introduction to Boto3: How to Use any AWS Service with Python

Serverless af

Nominated for 2022 - HackerNoon Contributor of the Year - Cloud Computing

Nominated for 2022 - HackerNoon Contributor of the Year - Devops

Nominated for 2022 - HackerNoon Contributor of the Year - Serverless

Nominated for 2022 - HackerNoon Contributor of the Year - Aws

Too Long; Didn't Read

Make resilience your competitive advantage

I built a crypto monitoring app with serverless - this is how

I built a crypto monitoring app with serverless - this is how

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

10 Things in Engineering We Don't Spend Enough Time On

101 Stories To Learn About Cloud Infrastructure

10 Things in Engineering We Don't Spend Enough Time On

10 Things I Did To Increase CloudTrail Logs Security

10 reasons to give cloud computing a go

10 Things in Engineering We Don't Spend Enough Time On

101 Stories To Learn About Cloud Infrastructure

10 Things in Engineering We Don't Spend Enough Time On

10 Things I Did To Increase CloudTrail Logs Security

10 reasons to give cloud computing a go

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps