

Stream enables you to listen to fee changes in near real-time using SQS, webhooks or websockets. In this tutorial, we will discuss how to use AWS SQS & Lambda to respond to feed updates.
While websockets are the preferred method to listen to changes, SQS notifications have a special spot in the feed infrastructure. Stream really shines in its ability to provide real-time because you can interact with the results that you receive. For example, you can listen to for changes to the feed and respond with, for example, an email to a user enticing them to pull the trigger on a purchase, send an SMS message based on an action taken within your application, among hundreds of other scenarios.
The goal of this post is for you to walk away with a strong understanding of AWS SQS, Lambda, and of course, Stream.
Note: To ensure that you can follow along from start to finish, please create an account or make sure that you have the necessary permissions to use the services mentioned aboveโโโwe wonโt be discussing permissions within AWS.
Setting up AWS SQS is rather straightforward. Once signed in, head on over to the SQS page. From there, weโll have the option to create a new queue. Click the โGet Started Nowโ button.
Next, you will be presented with a settings page for your new queue. For the name, go ahead and set it to โSTREAMโ. Then, choose the option on the left (the โStandard Queue).
Once created, weโll need to specify permissions for the newly created STREAM queue. For this tutorial, weโre going to open it up to the world.
Note: Never do thisโโโIโm only doing this for example purposes. Always keep your services locked down and only let in applications and users that require access. ๐
Click โAdd Permissionโ and you should be set!
Before we get started, weโll need to create a user for Stream. I prefer to create a separate user so that I can change permissions; however, youโre free to use an existing key and secret if youโd like (just skip the first part of this section if that is the case).
The first step in this process is to head over to the IAM section of AWS. Once you are there, click โUsersโ and then โAdd Userโ.
For the โUser nameโ, enter โSTREAMโ and for the โAccess typeโ, select โProgrammatic accessโ. Click the next button to continue to the next step.
In the above image, youโll notice that we selected AmazonSQSFullAccess as our Policy name. This will ensure that the user you are creating has access to SQS. You will need to do the same, in addition to setting the โGroup nameโ to โSTREAMโ.
Note: You will need to search for this policy. To do so, simply drop in SQS in the search box.
With the โSTREAMโ role created, continue on to the review stage.
Last, go ahead and create a user!
Copy the โAccess key IDโ and โSecret access keyโโโโyouโll need them in the next step.
Now, letโs go ahead and drop our API Key and Secret into the Stream dashboard so that Stream backend knows where to route outbound messages.
Steps:
Note: For a full explanation of the Stream SQS feature, you can view the documentation here.
AWS Lambda lets will allow us to run code without provisioning or managing servers. This service is going to come in handy for decoding the base64 encoded string that is sent to SQS as the payload. Once the message is decoded, you can do anythingโโโsend an SMS, email, etc.
Letโs kick things off by setting up Lambda. First things first, youโll want to head over to the Lambda section of AWS (this can easily be found with the search bar). If youโre a new user, go through the short-2 minute tutorial. If youโve done this before, jump right in.
The first step will be to create a new Lambda function as shown in the screenshot below. To copy my configuration, youโll want to do the following:
Once youโve done that, click โCreate Functionโ on the lower right-hand side of the screen.
All set! Letโs continue to the next step. If successful, you should see a screen that looks nearly identical to this:
To allow the Lambda function to access SQS, there is are some minor configurations that need to be made. Luckily, this step only requires a couple of clicks.
To connect SQS to Lambda, weโll use the ARN for the Lambda function. The easiest way to do this is to go to the SQS page, click on โQueue Actionsโ, and then click on โTrigger a Lambda Functionโ. Doing so will open a dialog.
Select โSTREAMโ from the dropdown menu and the Lambda Function ARN should automatically populate. Click โSaveโ once youโre finished. It takes around 30 seconds to 1 minute for the connection to get fully up and running. You can see the status under the โLambda Triggersโ section as shown in the screenshot below.
Now that we have SQS, Lambda and Stream connected, letโs run an end to end test to ensure that messages are making it to the Lambda function.
On the Lambda page, specify the contents of your function to be the following:
Then click on the โSaveโ button in the top right-hand corner.
Once youโve successfully saved your Lambda code, you will need to click โMonitoringโ which will take you to another page. Once redirected, click โView logs in CloudWatchโ.
Head back over to the Stream dashboard where you previously configured your SQS settings. There will be a handy button labeled โTest SQSโ. Clicking on this will send a test payload to SQS, which should then be forwarded over to Lambda where it will be logged out into CloudWatch. Whew! ๐
What you should see in the payload is an example activity for a tweet activity (example shown above). If nothing showed up in CloudWatch, try reloading. If that doesnโt work, run through the steps outlined above one more time to be sure that youโve checked every box.
Letโs do something with the example payload. Iโm going to propose that we use AWS SNS to forward the payload as a text message. To enable SNS, weโll need to add the AmazonSNSFullAccess to the STREAM user. To do that, head back to the IAM section of AWS (here). From there, click Groups > STREAM > Add Permissions > Attach Policy and choose AmazonSNSFullAccess and AWSLambdaFullAccess from the dropdown menu. Once selected, click โAttach Policyโ in the bottom right-hand corner.
Your permissions should now look like this:
The STREAM user now has permissions for SNS, so letโs go ahead and set it up. Head over to the SNS section of AWS and click โGet Startedโ. Next steps are listed below:
Steps:
Next, we need to modify our Lambda. As you probably noticed, modifying Lambda code in the editor is pretty difficultโโโor at least it would be on a larger scale. Instead, I recommend creating a new Node.js project somewhere on your computer and installing node-lambda. This awesome little module has the ability to set up, dry-run, package and deploy your Lambda to AWS without almost no effort.
Make a directory on your local machine called โstreamโ and move into that directory. Once inside, run the command โnode-lambda setupโ and the node-lambda package should bootstrap a Lambda for you. After running setup, itโs a good idea to ignore the generated event.json andย .env files, as well asย .lambda.
Note: For those who are not familiar with command line, the command above creates a new file (index.js) and installs all of the node modules that weโll need.
Follow the prompts in the terminal to finish up.
Open package.json and change scripts section to this:
Inside of the directory you generated, there should be aย .env file and paste your AWS credentials, role ARN (which youโve copied before) and fill rest of the parameters to match your configuration in AWS. I recommend that you specify the following settings:
Update your index.js file with the following code:
Note: Weโre still logging out the contents that Stream sends as an example.
And most importantly, deploy your code!
When we deployed this time around, it created a Lambda called โSTREAM-developmentโ or should have, assuming you followed the directions to a tee. If you did not name your Lambda โSTREAMโ, follow along with your naming convention.
Provided that this is a new Lambda, weโll need to add a new โtriggerโ. This is the same as we did previously in the post, so thereโs nothing new here. Just click on SQS from the left-hand drawer on the AWS Lambda page (for your new Lambda), and specify the configuration if necessary.
Remember the ARN from the topic we created? Now is the time to use it. Open yourย .env file and create a new environment variable called AWS_SNS_TOPIC_ARN and set the value to your generated SNS Topic ARN.
Before we wrap up and test, we need to generate a subscription for your phone number. This can be done under the SNS section of the AWS dashboard. To create a subscription, click on topics > select your existing topic and click โCreate Subscription. Specify the protocol as SMS and the endpoint should be a valid E.164 formatted phone number (e.g. 1โ555โ555โ5555).
The second to last part of this SNS setup is uploading some updated code to your Lambda function. Simply update your index.js file with the code below and use the yarn deploy command to push it to AWS.
Congratulations! Youโve set up an end to end test using Stream, AWS SQS, Lambda, and SNS to send an SMS message when an event happens to a feed (even if you are the one pressing the โTest SQSโ button ๐)
Iโve hosted the repo on GitHub, so feel free to clone it if you run into issues: https://github.com/nparsons08/stream-sqs-lambda-sns
Iโm also available to answer any questions in the comments below.
Happy coding! ๐
Create your free account to unlock your custom reading experience.