I’ve felt like I’ve been having a hard time finding music I like. I used to fill my iTunes with tracks I found on rap blogs like Rap Radar and Hypetrak, but lately it feels like the actual content gets drowned out by editorial posts. One of my favorite sources of music discovery used to be Complex’s which was a daily post with 5 mp3’s from around the internet, but unfortunately it doesn’t exist anymore. To fill this gap, I thought I’d create a chatbot that grabs content from rap blogs and sends out a daily update. And so I made stunts.fm: 5 O’Clock Shuffle Facebook One of the things I wanted to do when making this chatbot was to use Amazon Web Service’s Lambda service. Lambda allows you to deploy code without having to provision any servers. This was appealing to me because I wanted to minimize the operational overhead of this side project. The functions I wanted my chatbot to have were: Collect content from rap blogs Onboard new users Allow users to request their update through the bot Send out the update automatically every day Track what content users click on Analyze usage patterns Here’s what the entire architecture looks like: I’ll break down how I implemented each of my requirements below. Collecting content I collected content by writing scripts to check the RSS feeds for the rap blogs 2dopeboyz and Pigeons & Planes. Each script was deployed as a separate Lambda function. These scripts would look at each post from the past 24 hours and parse out any Soundcloud or YouTube links. They would then save this content along with metadata like the title, description, and preview image into a table hosted by AWS’ DynamoDB. These scripts get triggered automatically every day by the Lambda service. Onboarding new users Onboarding new users is relatively simple using Facebook’s Messenger API. A Facebook chatbot is essentially an HTTP endpoint, known as a webhook, that receives message events from users, processes them, and then sends a response to the Messenger API. In order to set up this webhook, I wrote a Python function to process message events and deployed it as a Lambda function. I used AWS’ API Gateway to set up a url with a /webhook endpoint backed by this Lambda function. Any time a user interacts with my chatbot, this endpoint will receive a message. The first thing the webhook code does is check if we’ve seen the sender before. If not, it knows we need to onboard the user. It’ll save the user’s details in the user table in DynamoDB so we can send them messages later. Allow users to request updates The webhook endpoint is also responsible for processing all messages from the user. One thing a user can ask the chatbot to do is to send them the content for today. When the webhook code sees that a user requested this, it calls another Lambda function which queries the content table in DynamoDB and then sends a response to the Messenger API. future Send out the update automatically every day The bot will also send out this update by itself every day. A Lambda function is triggered automatically which queries the user table in DynamoDB for all the subscribed users. It then sends a notification for every user to AWS’s Simple Notification Service. These notifications trigger the same “update sender” Lambda function the webhook uses to send the update to every user. Track what content users click on In order to track what content users click on, every time the “update sender” sends a message to a user, it generates a unique id for every user/content combination and saves that to a click tracker table in DynamoDB. Then instead of sending the actual content links to the user, it sends URLs for our API with the generated ids. When the user clicks these URLs in their Facebook Messenger app, our API Gateway calls a Lambda function which tracks this click in the click tracker table and then redirects them to the actual content. Analyze usage patterns In order to analyze content, users, and clicks, each of the DynamoDB tables is set up to write to an Elasticsearch cluster every time a row is created or updated.
