2,912 reads

Top 5 lessons learned from trying to build my own Serverless website

by Joseph NielsenSeptember 9th, 2016

Too Long; Didn't Read

A little over a month ago I started with a dream- aka self-imposed delusion ;)

Companies Mentioned

featured image - Top 5 lessons learned from trying to build my own Serverless website

A little over a month ago I started with a dream- aka self-imposed delusion ;)

Build a mobile responsive serverless web app on AWS that integrates with social media to show events as they happen in real time- and why not throw in React while I’m at it.

(If you are curious as to what “serverless” means, check out this handy dandy medium article)

Yes, I have built other applications, so I was “intellectually aware” that I was setting on something that would normally take three months with three people- at least- no small task for myself and my partner in crime plasx. But back in June I had been part of a team that had won the grand prize for the Manhattan Angelhack Hackathon. We were invited to join a hackcelerator program. See your dreams come to life and become a millionaire they said!

(Add flourish and waving of hands)

(Angelhack didn’t say that or deliver any such overly-optimistic nonsense but it makes my story better if I pretend that they did.)

These are the technologies that I set out to use

AWS Cognito for Identity Authorization and Authentication
AWS S3, CloudFront and Route 53 for static content hosting
AWS API Gateway, Lambda, DynamoDB, ElasticSearch for supporting my CQRS and REST styled data architecture
And then Bootstrap, React and whatever for the front. I’m not a front-end developer. I’m an architect. I’m here to make things sophisticated. (“Don’t worry, I’m an engineer, I can make this process complicated.”) And real-time messaging would be enabled by MQTT and/or WebSockets.

Where am I now?

I’ve got the second and third items done, almost got the first one done. The last one I’m still working on. Check it out here. https://gogobanana.net/

Top lessons learned so far on Serverless?

AWS Route 53

If you’ve registered some domains that you don’t care about, like maybe you decided at 2AM during a hackathon that your product should be called jumpingspider.info- not that that was me- but later you realize that you had had too many Monster energy drinks. You think in the back of your mind that it shouldn’t cost anything to own a domain, right?

Here is the caveat. By default, Amazon Route 53 creates a DNS hosted zone for you when you purchase a domain on Amazon Route 53. This is what allows for such great things as quick and free SSL certificates for your domain name. But the DNS management costs maybe around 50 cents a month. I’ve got 9 domain names registered and last month I used around $5.30 in credits on DNS hosted zones that largely only had the two default records. (I fixed it by deleting the hosted zones I wasn’t using)

AWS Cognito

This is not an easy to setup out of the box service. The easiest helper guide I found was the developer hundred something page SDK document and this lab for zombies. (No really, this is an awesome lab that covers a lot of other serverless topics as well, check it out.)

API Gateway, Lambda and DynamoDb

My goal was to pull about a thousand records from EventBrite and store them in my DynamoDb which would then stream out to ElasticSearch. (This DynamoDb streams feature is amazing and the Python template is easy to adjust for your own type and index scheme on ElasticSearch)

However this was my main challenge with Lambda and DynamoDb. Leaving my Lambda at 128mb of RAM and DynamoDb at a low write capacity would soon cripple my process, causing timeouts. I first tried incorporating the node npm package async into my process to throttle it down to only one PUT at a time.

However even this still caused timeouts of greater than 3 seconds. I also tried raising the timeout on Lambda. In the end I had to force one PUT operation at a time, set the RAM to 512mb on my Lambda, timeout of 60 seconds and set the Write capacity to 25 on DynamoDb. My biggest problem is now is that DynamoDb is no longer a cheap and easy way to persist my JSON document data. It’s still cheapish, mind, like $25 or less a month. But I get anxious speculating how much I might have to scale this in the future when others start using my database.

DynamoDb does also have a write batch items feature, which can allow inserting up to 25 documents at a time. (No more than 25)

Update: posted a resolution blog entry here

ElasticSearch

This isn’t really a surprise but ElasticSearch is costing me around $26/month for two t2.small nodes on my cluster. I could probably take it back down to the unsafe level of just one node, but I’d hate to ruin my data before I can start applying anything useful to it. It’s also insanely fast. I use a geohash filter on my query along with custom function scoring using a Gauss function on proximity and recency.

Google Maps API V3

This may or may not be a Google Maps problem but I thought I’d tag this along just in case. I just got back from vacation in England. (I know, great idea to book a vacation during a 13 week hackcelerator. But I bought the plane tickets back in April and won the hackathon in June, and I have issues with believing I can fly around the world and turn back time like Superman. You can do anything if you just believe!)

When I was in England, the only place I could use my app was in London for a couple of hours. I don’t know what was up. The map wouldn’t even draw most of the time. I thought maybe it was React or something else that was causing it but after removing these the problem still happened. Certainly I’ll have to convince my wife to let me fly back to England just to test out this bug again, but I’m guessing that I couldn’t get Google Maps to load on my phone in England because of some sort of regionally caused timeout. (Though it worked just fine on my laptop while in England.)

Try my website

Still it was great to use the app. It was very handy for touristing. We started with taking the London Underground (aka The Tube) to the London Bridge stop. Then we were going to walk to the Tower of London.

While using my app on the way I was able to discover “The Monument” — dedicated to the lives lost during the Great Fire of 1666 (If I can read my Roman numerals correctly.) They were handing out kids workbooks with stickers, which was a great win for my kids. I wouldn’t have stopped by The Monument otherwise but stuck to the river on the way to the Tower of London.