When Tom and I left Asana to build Pod, one of our biggest decisions was which cloud we wanted to build on. It’s a decision with huge path-dependency; changing infrastructure is tremendously costly, and each stack has quirks and limits that you’ll only discover after you’re locked in. In many ways, it’s a marriage. We decided on a GAE marriage — putting the entire system on the Google Cloud.
We had several reasons for doing this:
In many ways, we felt like we were choosing the underdog. Sure, Snapchat uses Google, but we didn’t know many others. At Asana, we were on a mostly-AWS stack. AWS is powerful, but it can also be hands-on: frequently updating EC2 instances with security patches, seeing remote jobs back up, and witnessing the sharding of the Asana database left deep impressions on me. Even Elastic Beanstalk can be manual. We needed something simpler for Pod but didn’t want to sacrifice power and scalability.
Months later, with the beta version of Pod released, we are happy with our GAE decision. There are a few challenges, but most of the “quirks” we have experienced have been positive.
Here’s what we like about GAE:
The cost of GAE scales with the amount of computing your app is doing —completely shutting down instances when you aren’t using them. So far, the web servers have been the major component of our cost, even though we do a lot of writing to Datastore, BigQuery, and Cloud Storage.
It wouldn’t be a marriage without a few problems, and we’ve definitely had moments of angst with Google. Google uses an internal issue tracker for managing bugs and feature requests, and certain teams do a very good job of triaging, whereas other teams seem to pay little attention.
Another challenging spot has been BigQuery. Similar to Redshift, it’s difficult to avoid duplicate data due to the fact that BigQuery is append-only and does not have support for primary keys. This can be addressed by app design, however. Another issue with BigQuery has been reliability and lack of communication by Google. We’ve found that the status page can be completely wrong — sometimes for hours. When BigQuery goes down, we usually end up contacting friends at Google to see if they know about it.
Finally, we can’t figure out any way that Google’s stack supports substring search without a hack on our part. If you want to search for partial strings (e.g., a “Justi” to find “Justin” in a user typeahead), you’re simply out of luck — the Search API doesn’t do that (nor does NDB).
Overall, however, we are extremely happy we got into bed with Google. Compared with AWS, Google feels simpler, value-added, and, more than anything, accessible. It’s almost as if Google has experience building cloud applications! All the hard technical issues around storing and moving data have been solved; by sharing these solutions, they enable developers like us to stop thinking about infrastructure and instead focus on building great products.
And that is a lot more fun.