About two months ago, I wanted to go on a vacation. I had the hotel more or less picked out, but the transportation was still up in the air. I began by scouring the web for cheap tickets, like normal seem to do. I scoured all of the fancy airlines, but all of their fares were too high for my liking. plane travel folks I decided to look into Southwest, the de facto airline for low fares (not really). I had heard from a few internet-goers that prices would drop on Tuesday nights, so I set out to test that claim. I started checking fare prices periodically throughout the day. ticket Days passed. Reload. Reload. “Would you like to resubmit this form?” Yes. Reload. Nothing. I started to get annoyed with the repetition (surprise, surprise!), so I set out to automate things as best I could. What if I wrote a script to do these annoying tasks for me? Yeah. I liked that idea. Crawling the Southwest website A quick look at Southwest’s site revealed no API. (No surprise there.) I figured that straight up scraping their website would be an effective approach. I could write a script which would, you know, scrape their website for the data I wanted. I chose Node because I figured they would have great tooling for web scraping, and I was more or less correct on that one. As I began experimenting with the algorithm… err, script, I realized that it was a lot harder than I thought it would be to fake a form submission on Southwest’s website so that I could get to the results page. I tried cURL. I tried “copy as cURL.” I tried faking the user agent. I tried all the things. The solution was to use to literally fill out and submit the search form, scrape the results page. osmosis and then So, I built a quick and dirty command line script that scraped their website every so often. I then let it do its thing every night while I slept. It wasn’t fancy, but it did what I told it to do. Analyzing the Fares I wrote a Node script … wait I already said that. Okay. So, I have a script running. It’s printing out fare prices, but I’m not really sure what to do with all of this data. It’s not like I want to sit there and watch it all night. So what did I do? I hooked up Twilio to send me a text message whenever a fare hit a certain threshold. Twilio offers a free account as long as you verify the phone number that you’re texting to, so that’s pretty rad. So I verified my number and let the texts come flowing in. And since I didn’t think it through, I also chose to send the text message at a low interval so that it would keep pinging me until I stopped the script, even if I was out on a date with my lovely wife. It wasn’t annoying at all. /s But it worked quite well. I could see that fares , especially late at night. The internet-goers didn’t lie to me this time. did indeed drop Plotting the Data After a few days, I got kind of annoyed (what’s new?) with looking at a stream of text and thought of complex ways to make it more appealing to look at for the few seconds that I looked at it every day. So what did I do? I hooked up to plot me some graphs and make things look pretty. I had seen blessed be put to use in the lovely , so I thought “why ever not?”. blessed webpack-dashboard It was awesome. Thanks, Ken Wheeler. Booking the Flight Days later, on a normal Tuesday night, it happened: the prices dropped dramatically, something that happen every Tuesday, but rather quite randomly. I snagged our tickets for under . (I, of course, booked them manually because my script is not fancy.) did not $100 that Sweet, Sweet Victory A month later, I went on vacation. And that’s about it. You can check out if you’d also like to go on a nice vacation. the project on GitHub Note: as of July, 2017, the project is no longer available on GitHub due to a takedown request by Southwest.
