Before you go, check out these stories!

0
Hackernoon logoUsing the Stream Real-Time Firehose with AWS SQS, Lambda, and SNS by@nickparsons

Using the Stream Real-Time Firehose with AWS SQS, Lambda, and SNS

Author profile picture

@nickparsonsNick Parsons

Director of Developer Marketing @ GetStream.io

Stream enables you to listen to fee changes in near real-time using SQS, webhooks or websockets. In this tutorial, we will discuss how to use AWS SQS & Lambda to respond to feed updates.

While websockets are the preferred method to listen to changes, SQS notifications have a special spot in the feed infrastructure. Stream really shines in its ability to provide real-time because you can interact with the results that you receive. For example, you can listen to for changes to the feed and respond with, for example, an email to a user enticing them to pull the trigger on a purchase, send an SMS message based on an action taken within your application, among hundreds of other scenarios.

The goal of this post is for you to walk away with a strong understanding of AWS SQS, Lambda, and of course, Stream.

Note: To ensure that you can follow along from start to finish, please create an account or make sure that you have the necessary permissions to use the services mentioned aboveโ€Šโ€”โ€Šwe wonโ€™t be discussing permissions within AWS.

Setting upย SQS

Setting up AWS SQS is rather straightforward. Once signed in, head on over to the SQS page. From there, weโ€™ll have the option to create a new queue. Click the โ€œGet Started Nowโ€ button.

Next, you will be presented with a settings page for your new queue. For the name, go ahead and set it to โ€œSTREAMโ€. Then, choose the option on the left (the โ€œStandard Queue).

Once created, weโ€™ll need to specify permissions for the newly created STREAM queue. For this tutorial, weโ€™re going to open it up to the world.

Note: Never do thisโ€Šโ€”โ€ŠIโ€™m only doing this for example purposes. Always keep your services locked down and only let in applications and users that require access. ๐Ÿ˜œ

Click โ€œAdd Permissionโ€ and you should be set!

Setting up Stream for Use withย SQS

Before we get started, weโ€™ll need to create a user for Stream. I prefer to create a separate user so that I can change permissions; however, youโ€™re free to use an existing key and secret if youโ€™d like (just skip the first part of this section if that is the case).

Create a new IAM User &ย Role

The first step in this process is to head over to the IAM section of AWS. Once you are there, click โ€œUsersโ€ and then โ€œAdd Userโ€.

For the โ€œUser nameโ€, enter โ€œSTREAMโ€ and for the โ€œAccess typeโ€, select โ€œProgrammatic accessโ€. Click the next button to continue to the next step.

In the above image, youโ€™ll notice that we selected AmazonSQSFullAccess as our Policy name. This will ensure that the user you are creating has access to SQS. You will need to do the same, in addition to setting the โ€œGroup nameโ€ to โ€œSTREAMโ€.

Note: You will need to search for this policy. To do so, simply drop in SQS in the search box.

With the โ€œSTREAMโ€ role created, continue on to the review stage.

Last, go ahead and create a user!

Copy the โ€œAccess key IDโ€ and โ€œSecret access keyโ€โ€Šโ€”โ€Šyouโ€™ll need them in the next step.

Hook It up toย Stream

Now, letโ€™s go ahead and drop our API Key and Secret into the Stream dashboard so that Stream backend knows where to route outbound messages.

Steps:

  1. Go to the Stream dashboard
  2. Create a flat feed with the feed name of โ€œuserโ€ (be sure to enable notifications)
  3. Click on the โ€œuserโ€ feed
  4. Enable SQS notifications by clicking the button labeled โ€œActiveโ€
  5. Fill out the SQS URL, AWS Key, and AWS Secret
  6. Click the save button in the top right-hand corner
Note: For a full explanation of the Stream SQS feature, you can view the documentation here.

Configuring Lambda

AWS Lambda lets will allow us to run code without provisioning or managing servers. This service is going to come in handy for decoding the base64 encoded string that is sent to SQS as the payload. Once the message is decoded, you can do anythingโ€Šโ€”โ€Šsend an SMS, email, etc.

Letโ€™s kick things off by setting up Lambda. First things first, youโ€™ll want to head over to the Lambda section of AWS (this can easily be found with the search bar). If youโ€™re a new user, go through the short-2 minute tutorial. If youโ€™ve done this before, jump right in.

The first step will be to create a new Lambda function as shown in the screenshot below. To copy my configuration, youโ€™ll want to do the following:

  • Select the โ€œAuthor from scratchโ€ box
  • Name your function โ€œSTREAMโ€
  • Update the Node.js runtime to Node.js v8.10 (outdated, but it is the latest version that is available on Lambda)
  • Specify your โ€œRoleโ€ name as โ€œSTREAMโ€
  • Include โ€œSQS Pollerโ€ permissions

Once youโ€™ve done that, click โ€œCreate Functionโ€ on the lower right-hand side of the screen.

All set! Letโ€™s continue to the next step. If successful, you should see a screen that looks nearly identical to this:

To allow the Lambda function to access SQS, there is are some minor configurations that need to be made. Luckily, this step only requires a couple of clicks.

  1. Under the โ€œDesignerโ€ section, click โ€œSQSโ€ in the scrollable section
  2. In the โ€œConfigure triggersโ€ section, search for your SQS queue (in our case, it will be named โ€œSTREAMโ€)
  3. Keep the batch size at 10
  4. Check โ€œEnable triggerโ€
  5. Finally, click on โ€œAddโ€

Sending SQS Messages toย Lambda

To connect SQS to Lambda, weโ€™ll use the ARN for the Lambda function. The easiest way to do this is to go to the SQS page, click on โ€œQueue Actionsโ€, and then click on โ€œTrigger a Lambda Functionโ€. Doing so will open a dialog.

Select โ€œSTREAMโ€ from the dropdown menu and the Lambda Function ARN should automatically populate. Click โ€œSaveโ€ once youโ€™re finished. It takes around 30 seconds to 1 minute for the connection to get fully up and running. You can see the status under the โ€œLambda Triggersโ€ section as shown in the screenshot below.

Testing Lambda Functionality ๐Ÿ‘

Now that we have SQS, Lambda and Stream connected, letโ€™s run an end to end test to ensure that messages are making it to the Lambda function.

On the Lambda page, specify the contents of your function to be the following:

Then click on the โ€œSaveโ€ button in the top right-hand corner.

Once youโ€™ve successfully saved your Lambda code, you will need to click โ€œMonitoringโ€ which will take you to another page. Once redirected, click โ€œView logs in CloudWatchโ€.

Head back over to the Stream dashboard where you previously configured your SQS settings. There will be a handy button labeled โ€œTest SQSโ€. Clicking on this will send a test payload to SQS, which should then be forwarded over to Lambda where it will be logged out into CloudWatch. Whew! ๐Ÿ˜…

What you should see in the payload is an example activity for a tweet activity (example shown above). If nothing showed up in CloudWatch, try reloading. If that doesnโ€™t work, run through the steps outlined above one more time to be sure that youโ€™ve checked every box.

Piping the Payload to AWSย SNS

Letโ€™s do something with the example payload. Iโ€™m going to propose that we use AWS SNS to forward the payload as a text message. To enable SNS, weโ€™ll need to add the AmazonSNSFullAccess to the STREAM user. To do that, head back to the IAM section of AWS (here). From there, click Groups > STREAM > Add Permissions > Attach Policy and choose AmazonSNSFullAccess and AWSLambdaFullAccess from the dropdown menu. Once selected, click โ€œAttach Policyโ€ in the bottom right-hand corner.

Your permissions should now look like this:

The STREAM user now has permissions for SNS, so letโ€™s go ahead and set it up. Head over to the SNS section of AWS and click โ€œGet Startedโ€. Next steps are listed below:

Steps:

  1. Click โ€œCreate topicโ€
  2. Set your โ€œTopic nameโ€ and โ€œDisplay nameโ€ to STREAM and then click โ€œCreate topicโ€
  3. Copy your โ€œTopic ARNโ€ as youโ€™ll need that in a bit

Next, we need to modify our Lambda. As you probably noticed, modifying Lambda code in the editor is pretty difficultโ€Šโ€”โ€Šor at least it would be on a larger scale. Instead, I recommend creating a new Node.js project somewhere on your computer and installing node-lambda. This awesome little module has the ability to set up, dry-run, package and deploy your Lambda to AWS without almost no effort.

Make a directory on your local machine called โ€œstreamโ€ and move into that directory. Once inside, run the command โ€œnode-lambda setupโ€ and the node-lambda package should bootstrap a Lambda for you. After running setup, itโ€™s a good idea to ignore the generated event.json andย .env files, as well asย .lambda.

Copy and Paste the Following Commands

Note: For those who are not familiar with command line, the command above creates a new file (index.js) and installs all of the node modules that weโ€™ll need.

Follow the prompts in the terminal to finish up.

Modify Your package.json File

Open package.json and change scripts section to this:

Openย .env and Modify Yourย Keys

Inside of the directory you generated, there should be aย .env file and paste your AWS credentials, role ARN (which youโ€™ve copied before) and fill rest of the parameters to match your configuration in AWS. I recommend that you specify the following settings:

Update Your index.jsย File

Update your index.js file with the following code:

Note: Weโ€™re still logging out the contents that Stream sends as an example.

And most importantly, deploy your code!

Fix the SQS Connection

When we deployed this time around, it created a Lambda called โ€œSTREAM-developmentโ€ or should have, assuming you followed the directions to a tee. If you did not name your Lambda โ€œSTREAMโ€, follow along with your naming convention.

Provided that this is a new Lambda, weโ€™ll need to add a new โ€œtriggerโ€. This is the same as we did previously in the post, so thereโ€™s nothing new here. Just click on SQS from the left-hand drawer on the AWS Lambda page (for your new Lambda), and specify the configuration if necessary.

Associate Lambda with SNSย Topic

Remember the ARN from the topic we created? Now is the time to use it. Open yourย .env file and create a new environment variable called AWS_SNS_TOPIC_ARN and set the value to your generated SNS Topic ARN.

Create Your SNS Subscription

Before we wrap up and test, we need to generate a subscription for your phone number. This can be done under the SNS section of the AWS dashboard. To create a subscription, click on topics > select your existing topic and click โ€œCreate Subscription. Specify the protocol as SMS and the endpoint should be a valid E.164 formatted phone number (e.g. 1โ€“555โ€“555โ€“5555).

Upload the Finalย Code

The second to last part of this SNS setup is uploading some updated code to your Lambda function. Simply update your index.js file with the code below and use the yarn deploy command to push it to AWS.

All Done!

Congratulations! Youโ€™ve set up an end to end test using Stream, AWS SQS, Lambda, and SNS to send an SMS message when an event happens to a feed (even if you are the one pressing the โ€œTest SQSโ€ button ๐Ÿ˜‰)

Iโ€™ve hosted the repo on GitHub, so feel free to clone it if you run into issues: https://github.com/nparsons08/stream-sqs-lambda-sns

Iโ€™m also available to answer any questions in the comments below.

Happy coding! ๐Ÿ‘‹

Tags

Join Hacker Noon

Create your free account to unlock your custom reading experience.