Analyzing over a thousand tweets a minute to find out how President Donald Trump is faring in real-time
Introduction
The President of the United States, Donald Trump is arguably the most polarizing figure in the world today. No matter where your political allegiances lie, you likely have atleast a somewhat strong opinion on Trump. Whether it’s the volatility of his actions or the brashness of the way he does what he does, it seems everyone is fixated on him.
This has been reflected in tons of interesting projects such as Trump Tracker, FiveThirtyEights Popularity Ratings, and Track Trump. With these projects in mind, I thought I would throw my hat in the ring and ship something of my own.
Overview
The Trump Sentiment Tracker uses real-time twitter data to determine the current public perception of President Donald Trump. The web application streams all tweets containing the word ‘Trump’ within the last five minutes of being launched. The tweets are then put through sentiment analysis in order to determine how positive or negative they are. Lastly, the sentiment scores are averaged and reported to you using an intuitive visualization.
Not doing so hot right now…
Outline
The full process is outlined below and explained in more detail throughout the rest of the article:
- Created a twitter streamer that collects tweets with the word ‘Trump’ and transfers them into a dataframe
- Utilized the VADER Sentiment Analysis package in order to speculate how positive or negative each given tweet was
- Adjusted the time frame to the last five minutes and averaged the sentiment scores for all the tweets
- Transferred the back-end to a Flask application for deployment
- Planned out how I wanted to effectively convey insights and designed a user interface
- Put it all together and hosted the web app on Heroku
See the Project
If you aren’t interested in any of the behind the scenes stuff, no worries. Feel free to skip all the detail below and check out the code on my Github or the work itself at this link below:
Goals
I came into this project wanting to accomplish several things. First and foremost, I wanted to improve my full-stack development related skill sets. Prior to this, I had done back-end and front-end programming separately but I had never taken a project the full distance. On top of this, I was also eager to take the opportunity to design something. I’ve recently developed an interest in design and I had never developed a web application user interface before diving into this project.
Tools and Software
Coming into this project with little development background with Python, my first task was to decide on an environment. After a bit of consideration, I ended up going with Pycharm (Version 2017.1.5). I found Pycharm to be very intuitive and effective for web application development. I’m sure I’ll be going back to it sometime in the near future. If you are a student and are interested in Pycharm, you can get it for free here.
I also used Python 3.5.2 and several other dependencies that you can find specified in the Github repository under the requirements.txt file. Lastly, I used Heroku to host the web application.
Quick look at the Pycharm environment and workflow
Back-End
The first step of the project was to figure out a way to stream tweets and then format them properly for analysis. For the streaming, I choose to use Tweepy and implemented a basic listener that while being run would collect every tweet with the word ‘Trump’ in it. The class then takes the current tweet and formats it properly to be added to the ongoing pandas DataFrame. The DataFrame is constantly updated to only keep data within a certain time threshold (5 minutes) in order to get virtually real-time insights.
After taking in data and formatting it correctly, sentiment analysis is performed. Following a bit of research, I decided to go with VADER (Valence Aware Dictionary and Sentiment Reasoner). I choose VADER because of it’s aptitude for social media data specifically.
Front-End
As far as the user interface goes, it was created using D3.js and HTML primarily. The meter was developed using Matt Magoffin’s open source project as a jumping off point and the general feel of the website was in large part motivated by Track Trump.
One of the things that I struggled with the most was integrating the back-end with the front-end so that the meter would correspond with the current mean sentiment score. Eventually, I was able to work around this problem by combining different JavaScript files and using getters and setters to my advantage. As far as the user interface goes, you can see the transformation below:
Deployment
Once the project was working locally, the next step was to figure out how I wanted to deploy it. I considered a few different alternatives but ended up going with Heroku due to the abundance of great tutorials out there. That, and the fact that it’s free. I found the following video especially helpful, if any of you plan on checking out heroku anytime soon, I highly recommend it.
Learning Resources
As you can probably tell, I came into this journey needing to acquire some skills that weren’t quite where I wanted them to be. In order to do this, I used a seemingly infinite amount of online resources. However, there were two that really stood out to me in the form of the free Full Stack Development course by Udacity and various Youtube tutorials by sentdex.
Reflection
Now that the project is shipped, I can look back and be very proud of what I was able to accomplish. There was times throughout the learning process where I undoubtedly struggled and thought about quitting but I’m glad that I stuck it out. By sticking with it and staying patient, I ended up acquiring some skills and techniques that I wouldn’t have right now otherwise.
Coming into this endeavor, I was admittedly quite inexperienced in full-stack development and web design. However, thanks to this project and all the steps I had to take in order to complete it, I can say that I am a much more competent and confident developer now.
Moving forward, I’m excited to see if Trump Sentiment Tracker gets any interest and most of all, I’m excited to start my next project.
Wrapping Things Up
I hope you enjoyed hearing about the making of Trump Sentiment Tracker. If you’re interested in the code, be sure to check it out on Github. If you have any feedback or inquiries, feel free to reach out to me on Twitter and LinkedIn. You can also see more of my work on my website.
Lastly, be sure to subscribe to my weekly data science newsletter below. Thanks for reading!
Originally published at conordewey.com on August 31, 2017.
