Update: Add to Slack_Tickerbot is now live on the Slack app directory. Try it out!_slack.com
Conversational apps (a.k.a “chat-bots”) have recently been given life thanks to messaging companies opening up their APIs. All major platforms, with the notable exception of WhatsApp, allow developers to build and publish apps. Because of ever-increasing difficulty of making people download new apps, what better way to get an audience than to leverage the popularity of existing apps!
Slack is one of those messaging platforms. It is an amazing tool for teams, and is now widely used by organisations of all kinds and sizes. It is different from others though, as once a team admin installs the app, it is available for all users on the team. I decided to leverage this to build a conversation game for slack teams. Its called TickerBot. Basically it allows you to invest in stocks using realtime prices (fake $$$, of course). You can manage your portfolio by buying and selling stocks, get realtime and historical price plots for them, and get daily updates on top performers in your team.
My aim was to build a neat and friendly app with minimum effort and at no cost. Now let’s deep dive into the tools used for various parts.
API.AI — Natural Language Processing (NLP) and conversation management
What good is a conversational app with no NLP? I don’t want the users typing exact commands to perform an action. The users should be free to type whatever they want, and the app needs to understand the intent. API.AI (recently acquired by Google) has a brilliant machine learning based NLP model, that allows you to manage the conversation flow with contexts. The free plan allows you to create unlimited private agents, however restricted to one query per second (QPS). It also has one-click integration with Slack which allows you to quickly deploy your app. The downside? API.AI is unstable. Random bugs, sudden changes to backend with no prior information are very common.
Scaling considerations — 1 QPS is not enough for thousands of users. So what’s the workaround? For this app, a lot of user input is simple, which doesn’t need any advanced NLP. Only sending input which is more than 2 words will save a lot of API calls. If need be, I can store processed patterns and responses from API.AI, thereby reducing it’s need over time. As far as conversation management is concerned, it’s not rocket science. I am already partially doing it myself because of some restrictions in their framework.
Slack — API for asynchronous responses, authentication and of course, the Slack App itself
API.AI requires quick responses (max 5 seconds). Also, I can only respond once per user input. Unfortunately, the external APIs I used, took a lot of processing time, and I needed to break down my response in multiple messages. Since they don’t give you the OAuth credentials of the slack team that’s installed your app (that’s sad, since technically you own the credentials), I had to turn my back on the one-click integration, and use Slack API directly to send asynchronous responses. To send these messages, I needed the OAuth credentials, and the user ID of the recipient. The documentation was a pain to go through. I had to read each page multiple times to understand which API I need to use where. Not their fault I guess, they have grown really fast recently.
Scaling considerations — The slack app store provides a great place to scale your app, and grow your audience. OAuth needs to be implemented correctly to allow new teams to use your app.
plot.ly — Beautiful plots
Plot.ly has THE BEST python library (at least according to me) for plotting simple, feature rich and beautiful graphs. They have clear documentation and their APIs are pleasantly simple to understand. I used it for plotting historical stock prices and portfolio performance. The downside is that plotting and image conversion is done on their server. It takes around 10–30 seconds to send graph data and receive the link to the image. It’s not possible to get a plot image offline directly, and the free online version restricts it to 50 API calls per day (still generous though).
Scaling considerations — Either purchase plot.ly pro, or find a workaround. In the limited time I had, I couldn’t get this to work offline. I have a few more ideas though, such as converting the graph to matplotlib, or hacking the offline HTML version they support.
Yahoo Finance, Quandl — Financial data
I started with good old Yahoo Finance. It provides historical daily prices of various shares worldwide and also gives access to current prices realtime. It’s free of course. But the catch is their documentation is unclear and now even the future of this service is unclear. I started getting some errors randomly as well. Quandl was only other service providing free data. However, it offers only historical prices for U.S., India stocks free of cost. I ended up using quandl (with yahoo as backup) for historical prices , and yahoo for realtime.
Scaling considerations — Quandl free API limit is 2K calls / 10 min and 50K / day, which is very generous. Yahoo’s own future is unknown, so I don’t think it is the right choice for serious apps. Either magically there is an open source movement for free and easy access to financial data soon, or I have to pay exorbitant amount of money to get access. Seriously, access to such data is THE bottleneck and most frustrating.
Django, AWS — Development and deployment
I only have good things to say in this section, thanks to the open source community and amazon. Python makes life easy. I used the standard django+gunicorn+nginx setup. For sending messages asynchronously, I used celery + rabbitMQ extensively. Celery is very convenient. I also tried using Amazon SQS, but it didn’t work reliably at my first attempt, so I switched back to rabbitMQ as the message broker. I needed SSL for communicating with Slack API, so I used Let’s Encrypt, which provides free SSL certification. It’s super quick to setup and it’s amazing that it costs nothing at all. Also, AWS free-tier for their server (EC2) and database (RDS) services.
Scaling considerations — Scaling will be required for the infrastructure. Means giving $$$ to AWS. I can spawn more celery workers on EC2 instances, use AWS auto-scaling, load balancing etc. AWS is convenient and inexpensive though.
Post project thoughts
I developed this while I was building a company at Entrepreneur First. This project got inspiration while doing technical research on conversational bots. It was completely unnecessary for the task at hand, but as a software engineer who likes to put pieces together to make something cool, I just couldn’t help myself. Building a company requires more concrete business and market development, and I also ended up not pursuing this space (oops!). I realised that it’s not that hard to build production level bots. You don’t have to deal with convoluted front-end development (no HTML, CSS, JS; hehe).
You can checkout the code at Github. Feel free to ask, suggest, comment or contribute :)
Update: Add to Slack_Tickerbot is now live on the Slack app directory. Try it out!_slack.com