We all know that building content platforms is hard. You are constantly fighting misinformation, spam, and undesired ads—mostly from bots, the plague of the modern web. Some estimate that two-thirds of links posted on Twitter can come from bots.
So how can you fight them?
You could constantly monitor your website, manually checking for new entries, or you could leave it up to your users, waiting for them to report platform abuse. However, both solutions are slow and tedious.
A type of automated filtering is probably the best, if imperfect, solution. However, if you’re running a smaller site, the best way may be to receive automated notifications about new content!
In this article, I’ll show how to build such a tool (using MuleSoft, Heroku, and this guide) to send notifications to Slack when new content is entered into a content platform.
Using MuleSoft to Monitor Data Changes
Some artists say that everything has already been created. Some agree, and some don't. However, if developers are artists, then one thing is certain: In 2020, much of an apps core functionality has probably already been developed. If your core business is a bespoke content platform, why would you invest your time building integration software? A better strategy is to use software that has already been battle-tested and is just a few clicks away from working.
Anypoint by MuleSoft is that platform. It creates connections between services like CRMs, communication tools, and databases. Then it allows you to transfer data between those services when certain events take place. For example:
When your app has a new user, create a new lead in SalesforceWhen a photo is sent in Slack, save it to S3 and add to your website when new content is uploaded to your Heroku app, send its summary to Slack so the moderators can review it as quickly as possible
... and you probably already know which use case we will explore in the rest of the tutorial 😉
Introducing Pythonic News!
For the purpose of this article, we will deploy an open-source project, Pythonic News on Heroku. Pythonic News is a clone of Hacker News written in Python/Django. With slight modifications, I made the app Heroku-deployable without any additional configuration. The detailed process is described below. Run in production at your own risk! (Please don't 😅)
Hacker News gets an enormous amount of traffic every day and content quality is a serious business for the moderators. Fortunately, we already have our strategy to deal with this problem, so let's go ahead and build the content moderation system.
MuleSoft and Heroku Integration
To take full advantage of the Anytime platform, we need to run a JDBC compliant database like Postgres. Fortunately, Heroku allows us to run a Postgres database for free and there is even documentation on plugging it to MuleSoft's Database Connector. The modifications mentioned above include running Postgres instead of Django's default Sqlite, so we are good to go. 👍
Now let’s look at the steps needed to build the whole system.
Heroku App Deployment
(Note: You can skip this part and use my deployed fork of pythonic-news.)
Let's start by cloning the original Pythonic News repo:
git clone [email protected]:sebst/pythonic-news.gitThe project is complete and working, however, before we deploy it, we need to take three additional steps.
1. Install Gunicorn and add Procfile for Heroku
We’ll use Gunicorn as our web server (because you should never run ./manage.py runserver in production).
pip install gunicornProcfile is the configuration file for Heroku. We will guide our Heroku app to apply migrations on the release and use Gunicorn to serve the app.
2. Change the Database to Use Postgres
I prefer to run the same database locally as in production, so I added the support for the Postgres config. The snippet below will:
release: python manage.py migrate
web: gunicorn hnclone.wsgiLocally search for POSTGRES*** environmental variable and if none is found, connect to an open database ‘pythonic_news’.In Heroku, it will read the DATABASE_URL variable, which will automatically show up in our Heroku Config Varsafter setting up a Postgres Add On.
import environ
# settings.py
env = environ.Env()
DB_USER = env("POSTGRES_USER", default="postgres")
DB_PASSWORD = env("POSTGRES_PASSWORD", default="")
DB_PORT = env("POSTGRES_PORT", default="")
DB_NAME = env("POSTGRES_NAME", default="pythonic_news")
DB_HOST = env("POSTGRES_HOST", default="")
DB_CONFIG_URL = f"postgres://{DB_USER}:{DB_PASSWORD}@{DB_HOST}:{DB_PORT}/{DB_NAME}"
DATABASES = {"default": env.db("DATABASE_URL", default=DB_CONFIG_URL)}3. Do Some Miscellaneous Changes That Are Definitely NOT Production Ready But Worked for Me 🤷🏻♂️
For me it was DEBUG=True, so the static files work locally. For you it may be something totally different! The project setup is not the essence of this tutorial, so we will skip to the next part.
Slack Setup
We’ll assume that you already have a Slack workspace (you probably wouldn’t read this article otherwise), so the next step is to create a Slack app and connect it to a selected channel. To do this, let’s head over to https://api.slack.com/appsand click the "Create App" button.
Then, when the app is created, you will be prompted to select the functions and permissions that your bot needs.
In our case, we will need to enable incoming webhooks and chat:write. In true Linux fashion, always remember to use only the minimal set of permissions that the app needs!
Looks great! Now let’s connect the app to the channel in the menu in the right-hand side panel.
Ready! Now, when MuleSoft sends a message to the chat, we are ready to receive it.
MuleSoft Setup
Now on to the fun part! We have our application with a database. We have Slack. Let's connect them!
Login to MuleSoft Anypoint and create a new project.
After choosing "Create new application", we will be prompted to choose an event listener and a receiver.
In the first case, choose a "Database connector" that will listen on new row entries.
For the receiver, choose Slack and "Post new message" (duh!).
Now it’s configuration time!
For database configuration, we will use the tutorial written by Heroku on connecting their data products with MuleSoft. Once it's ready, we should be able to choose a table that we want to listen on and select which column plays the role of an id. A watermark column will mark which rows have already been seen by our connector. In this case, we will select the item_ptr_id for both cases.
🎉 Congratulations! The database is now properly configured! Let's set up the Slack messages.
First, let’s connect to the Slack app that we created. Again, for this, we will use an existing tutorial.
Once the Slack connection is live, MuleSoft will automatically detect what data we can obtain from the selected table and will present it in the right-hand panel. Now you just need to select a channel to post the messages to, write the message you want to post, and drag-and-drop the values from the right to the correct places. And once you're done...
The whole flow is done as well! Now head over to test it (with a short-term MuleSoft deployment) or deploy it so the connection lives after you exit the platform.
Let's take it for a spin and receive some Slack notifications.
Demo
Pythonic News UI is really simple — by design. To submit a link, the user clicks the "submit" button and proceeds with filling out the form.
Message posted, and boom 💥
Our app informed me about the new submission. Now, I can go to the admin panel, and mark it as "spam", because it doesn't have anything to do with Python. 😅
(Actually, it wasn't this easy...)
But, hey, we got there eventually!
Challenges and Issues I faced with the Setup
Naturally, the process wasn’t as smooth as it looks in this write-up. During development, the following issues forced me to stop and rethink some details:
1. You need to upload your own JDBC driver
While Postgres is JDBC-compliant, you need to upload and specify the driver on your own. This has proven to be problematic, because once you enter the official Postgres website, there are over 50 different available drivers to download. I went with version 42.2.1, one recommended by MuleSoft. However, it is not the newest one.
2. Slack token
While creating the Slack app, you may very easily over complicate things by choosing the OAuth authentication flow instead of the token. While OAuth is certainly recommended for more complex projects, the token authentication will work just fine in our case. Simply copy the access token from OAuth & Permissions tab. It starts with xoxb- followed by a random string and needs to be passed to MuleSoft during Slack integration.
3. Watermark
When the whole flow worked correctly for the first time, my Slack was bombarded with tens of messages per second. This was caused by not setting up the Watermark field in Database Connector. It caused MuleSoft to forget which rows had been seen before. Because of that, with every iteration, a message was sent to the channel for each database entry. It was a complete mess! Putting the watermark on a column with a unique ID solved this problem.
Summary
Automating one’s business is easier than ever. Tools like Heroku allow us not to worry about infrastructure management, and MuleSoft will simplify complex business integrations to drag-and-drop fun. What a time to be alive!
I hope that this tutorial will help you set up your own integrations and save you a ton of precious time in the outcome. Good luck!
Also published on: https://dev.to/wkulikowski/using-mulesoft-to-send-slack-notifications-when-data-changes-3i34