Build a CRUD API using DynamoDB and Serverless - Part 1

Introduction

In this post, we are going to see how to make a CRUD application using DynamoDB, AWS Serverless, and NodeJS, we will cover all CRUD operations like DynamoDB GetItem, PutItem, UpdateItem, DeleteItem, and list all the items in a table. Everything will be done using the AWS Serverless framework and on NodeJS, this is part 1 of this series, in part 2 we are going to add authentication to this application, for now, let’s get started.

Project Setup

Our project folder structure will look like this

Let’s discuss what each of these folders is doing

config – This folder will hold all the config related files, in our case, it is holding a single file that is creating a DynamoDB AWS SDK instance to use everywhere in our project, so instead of importing DynamoDB instance in each file, we are just importing it in one file and then exporting the instance from this file and importing everywhere else.

functions – This is for holding all the files related to any utility function.

post – This is our main folder which will hold all the lambda functions for our CRUD operations.

Serverless.yml file

This file is the soul and heart of every serverless project, let’s try to see in parts how this file looks like for us

service: dynamodb-crud-api

provider:
  name: aws
  runtime: nodejs12.x
  environment:
    DYNAMO_TABLE_NAME: PostsTable
  iamRoleStatements:
    - Effect: Allow
      Action:
        - dynamodb:Query
        - dynamodb:Scan
        - dynamodb:GetItem
        - dynamodb:PutItem
        - dynamodb:UpdateItem
        - dynamodb:DeleteItem
      Resource: "arn:aws:dynamodb:${opt:region, self:provider.region}:*:table/${self:provider.environment.DYNAMO_TABLE_NAME}"

Here we are defining one environment variable which will store the name of our DynamoDB table and we are also adding different permissions which our lambda functions will need to do different operations like dynamodb:GetItem to get the data item from the table, dynamodb:PutItem to insert a new entry in the table and so on.

Lambda Functions

Now we are going to define all our lambda functions with their respective configuration

functions:
  listPosts:
    handler: post/list.listPosts
    events:
      - http:
          path: posts/list
          method: get
          cors: true

  createPost:
    handler: post/create.createPost
    events:
      - http:
          path: post/create
          method: post
          cors: true

  getPost:
    handler: post/get.getPost
    events:
      - http:
          path: post/{id}
          method: get
          cors: true

  updatePost:
    handler: post/update.updatePost
    events:
      - http:
          path: post/update
          method: patch
          cors: true

  deletePost:
    handler: post/delete.deletePost
    events:
      - http:
          path: post/delete
          method: delete
          cors: true

Now we are defining all our lambda functions which are going to be called when we will send requests to our API Gateway URLs, an HTTP event is attached with each lambda function so they can get called through API Gateway.

path – This is the relative path of the endpoint which we want to use, so for example, if our API Gateway URL is https://abc.com then getPost lambda function will be called with this endpoint https://abc.com/post/{id}.

method – This is just the API request type POST, GET, DELETE, etc.

Defining DynamoDB table

Finally, we need to define our DynamoDB table and its configuration, for how pricing gets calculated in DynamoDB check out AWS DynamoDB Pricing.

resources:
  Resources:
    UsersDynamoDbTable:
      Type: AWS::DynamoDB::Table
      DeletionPolicy: Retain
      Properties:
        AttributeDefinitions:
          - AttributeName: id
            AttributeType: S
        KeySchema:
          - AttributeName: id
            KeyType: HASH
        ProvisionedThroughput:
          ReadCapacityUnits: 1
          WriteCapacityUnits: 1
        TableName: ${self:provider.environment.DYNAMO_TABLE_NAME}

AttributeDefinitions – Here we define all the key fields for our table and indices.

KeySchema – Here we set any field which we defined in AttributeDefinitions as Key field, either sort key or partition key.

ProvisionedThroughput – Here we define the number of read and write capacity units for our DynamoDB table.

Whole serverless.yml file

service: dynamodb-crud-api

provider:
  name: aws
  runtime: nodejs12.x
  environment:
    DYNAMO_TABLE_NAME: PostsTable
  iamRoleStatements:
    - Effect: Allow
      Action:
        - dynamodb:Query
        - dynamodb:Scan
        - dynamodb:GetItem
        - dynamodb:PutItem
        - dynamodb:UpdateItem
        - dynamodb:DeleteItem
      Resource: "arn:aws:dynamodb:${opt:region, self:provider.region}:*:table/${self:provider.environment.DYNAMO_TABLE_NAME}"

functions:
  listPosts:
    handler: post/list.listPosts
    events:
      - http:
          path: posts/list
          method: get
          cors: true

  createPost:
    handler: post/create.createPost
    events:
      - http:
          path: post/create
          method: post
          cors: true

  getPost:
    handler: post/get.getPost
    events:
      - http:
          path: post/{id}
          method: get
          cors: true

  updatePost:
    handler: post/update.updatePost
    events:
      - http:
          path: post/update
          method: patch
          cors: true

  deletePost:
    handler: post/delete.deletePost
    events:
      - http:
          path: post/delete
          method: delete
          cors: true

resources:
  Resources:
    UsersDynamoDbTable:
      Type: AWS::DynamoDB::Table
      DeletionPolicy: Retain
      Properties:
        AttributeDefinitions:
          - AttributeName: id
            AttributeType: S
        KeySchema:
          - AttributeName: id
            KeyType: HASH
        ProvisionedThroughput:
          ReadCapacityUnits: 1
          WriteCapacityUnits: 1
        TableName: ${self:provider.environment.DYNAMO_TABLE_NAME}

Config File

This config file will be inside a folder named config in our project as shown above in the project structure image, this file will contain the code which will export the DynamoDB AWS SDK instance so we can call DynamoDB APIs anywhere we want to in other parts of the code.

const AWS = require("aws-sdk");

const dynamo = new AWS.DynamoDB.DocumentClient();

module.exports = dynamo;

Functions File

In this project, we are using a single file that will hold all the utility/common functions which we are going to use multiple times in our project.

  const sendResponse = (statusCode, body) => {
  const response = {
    statusCode: statusCode,
    body: JSON.stringify(body),
    headers: {
      'Content-Type': 'application/json',
      'Access-Control-Allow-Origin': '*',
      'Access-Control-Allow-Credentials': true
    }
  }
  return response
}

module.exports = {
  sendResponse
};

We will call this sendResponse function from many places, in fact from all our lambda functions to return the response for a request, this will return the JSON response back to the user, it has two arguments, one is the HTTP status code and the other is the JSON body which we will pass whenever we are going to call this function, we are also passing some required headers with the response which handles the most common “access not allowed” cors issues.

Lambda Functions

Now it is time to start working on our lambda function which will hold all our logic, we will divide four lambda functions for four different CRUD operations.

DynamoDB PutItem (Insertion)

This lambda function is defined inside the create.js file, in this lambda function, we will be doing our first operation which is inserting a new data item in the table, let’s break it down into parts.

const dynamoDb = require("../config/dynamoDb");
const { sendResponse } = require("../functions/index");
const uuidv1 = require("uuid/v1");

We need to import our DynamoDB instance from the config file which we created earlier, our sendReponse function and we are using an NPM called uuid which is used to generate a random id, this id will be used as our partition key for each post.

  const body = JSON.parse(event.body);
  try {
    const { postTitle, postBody, imgUrl, tags } = body;
    const id = uuidv1();
    const TableName = process.env.DYNAMO_TABLE_NAME;
    const params = {
      TableName,
      Item: {
        id,
        postTitle,
        postBody,
        imgUrl,
        tags
      },
      ConditionExpression: "attribute_not_exists(id)"
    };

Here we are getting different properties from the request payload which we are going to insert in our post table, after that we are generating a random id by calling a function provided by uuid library.

attribute_not_exists – By default DynamoDB PutItem will overwrite the content of any item if we are trying to insert data with the same partition key, but we don’t want that so to only insert the data if the partition key is not found we are using this conditional expression.

await dynamoDb.put(params).promise();
return sendResponse(200, { message: 'Post created successfully' })

We are passing our parameters which we created in the previous step in the DynamoDB put API call and sending 200 status code with the relevant message.

Whole create.js file

"use strict";

const dynamoDb = require("../config/dynamoDb");
const { sendResponse } = require("../functions/index");
const uuidv1 = require("uuid/v1");

module.exports.createPost = async event => {
  const body = JSON.parse(event.body);
  try {
    const { postTitle, postBody, imgUrl, tags } = body;
    const id = uuidv1();
    const TableName = process.env.DYNAMO_TABLE_NAME;
    const params = {
      TableName,
      Item: {
        id,
        postTitle,
        postBody,
        imgUrl,
        tags
      },
      ConditionExpression: "attribute_not_exists(id)"
    };
    await dynamoDb.put(params).promise();
    return sendResponse(200, { message: 'Post created successfully' })
  } catch (e) {
    return sendResponse(500, { message: 'Could not create the post' });
  }
};

DynamoDB GetItem (Read)

This lambda function is defined inside get.js file, this will be doing the reading operation, meaning getting the data from the DynamoDB using the partition key.

const { id } = event.pathParameters;
const params = {
TableName: process.env.DYNAMO_TABLE_NAME,
KeyConditionExpression: "id = :id",
ExpressionAttributeValues: {
  ":id": id
  },
 Select: "ALL_ATTRIBUTES"
 };

We are getting the id from the request parameters, then we are matching this with the partition key in our table and selecting all the fields from the table.

const data = await dynamoDb.query(params).promise();
if (data.Count > 0) {
  return sendResponse(200, { item: data.Items });
} else {
  return sendResponse(404, { message: "Post not found" });
}

Now we are querying the table with the params and checking if there are any items returned or not if there are any items found then we are returning the items array otherwise we are returning an appropriate message.

Whole get.js file

"use strict";

const { sendResponse } = require("../functions/index");
const dynamoDb = require("../config/dynamoDb");

module.exports.getPost = async event => {
  try {
    const { id } = event.pathParameters;
    const params = {
      TableName: process.env.DYNAMO_TABLE_NAME,
      KeyConditionExpression: "id = :id",
      ExpressionAttributeValues: {
        ":id": id
      },
      Select: "ALL_ATTRIBUTES"
    };

    const data = await dynamoDb.query(params).promise();
    if (data.Count > 0) {
      return sendResponse(200, { item: data.Items });
    } else {
      return sendResponse(404, { message: "Post not found" });
    }
  } catch (e) {
    return sendResponse(500, { message: "Could not get the post" });
  }
};

DynamoDB UpdateItem (Updation)

This lambda is defined inside update.js file, in this lambda function we are going to do the update operation which will update the data inside the DynamoDB table.

    const body = JSON.parse(event.body);

    const { postTitle, postBody, imgUrl, tags, id } = body
    const params = {
      TableName: process.env.DYNAMO_TABLE_NAME,
      Key: {
        id
      },
      ExpressionAttributeValues: {
        ":postTitle": postTitle,
        ":postBody": postBody,
        ":imgUrl": imgUrl,
        ":tags": tags
      },
      UpdateExpression:
        "SET postTitle = :postTitle, postBody = :postBody, imgUrl = :imgUrl, tags = :tags",
      ReturnValues: "ALL_NEW"
    };

We are getting the data from the request payload, there is one additional property that we need to send with the request is id of the item which we want to update.

ExpressionAttributeValues – DynamoDB has many reserved keywords so there may be a case where our table field name matches with that reserved keyword, then in that case this update will throw an error. To avoid this DynamoDB has a system of setting the original field name with some alternate name temporarily just for this purpose, so we are setting all the fields values in this object.

UpdateExpression – To update any item in DynamoDB we need to pass the field name with their respective update expression.

ReturnValues – This is just indicating that we need the updated fields data in the response when we’ll run our update operation.

    const data = await dynamoDb.update(params).promise();
    if (data.Attributes) {
      return sendResponse(200, data.Attributes);
    } else {
      return sendResponse(404, { message: "Updated post data not found" });
    }

Now we just need to call the update API with the params, we are also checking if updated attributes data were returned or not, if yes then we are returning that data otherwise we are returning 404 status code with a message.

Whole update.js file

"use strict";

const { sendResponse } = require("../functions/index");
const dynamoDb = require("../config/dynamoDb");

module.exports.updatePost = async event => {
  try {
    const body = JSON.parse(event.body);

    const { postTitle, postBody, imgUrl, tags, id } = body
    const params = {
      TableName: process.env.DYNAMO_TABLE_NAME,
      Key: {
        id
      },
      ExpressionAttributeValues: {
        ":postTitle": postTitle,
        ":postBody": postBody,
        ":imgUrl": imgUrl,
        ":tags": tags
      },
      UpdateExpression:
        "SET postTitle = :postTitle, postBody = :postBody, imgUrl = :imgUrl, tags = :tags",
      ReturnValues: "ALL_NEW"
    };

    const data = await dynamoDb.update(params).promise();
    if (data.Attributes) {
      return sendResponse(200, data.Attributes);
    } else {
      return sendResponse(404, { message: "Updated post data not found" });
    }
  } catch (e) {
    return sendResponse(500, { message: "Could not update this post" });
  }
};

DynamoDB DeleteItem (Deletion)

This lambda function will be in delete.js file, in this lambda function we are going to delete an item from the table.

"use strict";

const { sendResponse } = require("../functions/index");
const dynamoDb = require("../config/dynamoDb");

module.exports.deletePost = async event => {
  try {
    const body = JSON.parse(event.body);
    const { id } = body;
    const params = {
      TableName: process.env.DYNAMO_TABLE_NAME,
      Key: {
        id
      }
    };
    await dynamoDb.delete(params).promise();
    return sendResponse(200, { message: "Post deleted successfully" });
  } catch (e) {
    return sendResponse(500, { message: "Could not delete the post" });
  }
};

This lambda function is self-explanatory, we are just getting the id of the item which we want to remove in the request and we are passing that as a param in the DynamoDB delete API.

So basically now we are done with all our four operations Create/Read/Update/Delete but we are still missing something, we don’t have any lambda function to list all the posts, let’s look into how we can do so.

DynamoDB Scan

We are going to use DynamoDB scan to get all the items from the table, scan operations can be costly while using DynamoDB so we need to be careful with it and try to avoid using it as much as possible and even if we have to use it we should only get the data we need and not do unnecessary scans of items.

"use strict";

const dynamoDb = require("../config/dynamoDb");
const { sendResponse } = require("../functions/index");

module.exports.listPosts = async event => {
  try {
    const params = {
      TableName: process.env.DYNAMO_TABLE_NAME,
    }
    const posts = await dynamoDb.scan(params).promise();
    return sendResponse(200, { items: posts.Items });
  } catch (e) {
    return sendResponse(500, { message: "Could not get the posts list" });
  }
};

This function will be in list.js file, we are doing a very simple DynamoDB scan here and returning the data.

Conclusion

This was a long post, but if you are able to reach up to this point then congratulations to you, because you now have a full CRUD API made using DynamoDB, AWS Serverless, and Nodejs, we can always enhance this application and make it better, here are some ideas –

• Adding authentication to the application.
• Adding user-based posting.
• Adding validations.
• Making it more cost-effective

If you have any other suggestions, please feel free to add them in the comments below, there will be more parts for this application so stay tuned.

Get this code

Source code on Github

Check out more:

AWS Cognito Pricing

DynamoDB VS MongoDB: Detailed Comparison

This article was first published here