Dealing with a nasty AWS Billing Surprise: Beware the defaults.

The architecture of my Amazon Web Services application was supposed to be simple. It’s two independent Lambda functions with a Redis cache between them for persistence. By purchasing Lambda compute power by the millisecond, I thought my app would be stupid cheap, possibly even free! I hadn’t even launched into production when I got my first bill for my partial month of development: $26.97 and if that wasn’t bad enough, my forecast for the following month was supposed to be $40.78 and nobody had even used my app!

The shock of this discovery taught me a valuable lesson about creating hobby projects on AWS: Beware of over-provisioning. In the rest of this essay I’ll walk through how I identified the costlier parts of my app, and what I did to trim costs.

Go to the billing console! When I received an unexpectedly nasty bill from Amazon, my first reaction was to not reopen the email while cursing. “There has to be a mistake…” I say in complete denial. The first step should be to use the billing console on the AWS website to identify where the costs are actually coming from. When I went to mine, I found the following scary picture:

$40 for an app I haven’t even launched, yikes!

From there, I noticed two immediate cost centers: EC2 and Elasticache. That was weird, I thought. I’m not even using any EC2 instances, I’m only using Lambda, and my Redis node in Elasticache only stores 5 strings. How could that be costing me almost $41 a month?!

From the main page I jumped to the dashboard where I could clearly see what was costing me the money:

Aha! My two targets.

I could see that it was clearly two problematic services, my NAT Gateway and my Redis node. With cost reduction in mind, I knew I had my work cut out for me.

Rolling my own NAT Gateway: My first job was to replace the costly NAT gateway that allowed my Lambda to connect to external networks. Amazon provisions these gateways to deal with bursts of up to 10 Gbps. That was way over provisioned for my simple needs. So I found an interesting solution: setting up a free t2.micro with a NAT AMI. Luckily Amazon provides a preconfigured image exactly for this purpose (amzn-ami-vpc-nat-hvm-2016.03.0.x86_64-ebs). I spun up an instance, and put that it into my VPC in a subnet with an internet gateway.

The new instance goes into the subnet with the internet gateway as a default route.

Amazingly this worked right out of the box! I’m thankful that Amazon provides a machine image to duplicate their NAT gateway functionality, but I find it unfortunate that it is so difficult to set up a Lambda function to make outbound requests. If you are having trouble with Lambdas and VPCs, I recommend checking out this AWS article, and Vincent de Lagabbe’s very helpful StackOverflow answer.

Cost of running new t2.micro instance: $0

Running Redis on the cheap: The second cost center was my Elasticache Redis instance. The reason it cost me $6 was that it was running on a t2.small instance. Considering my simple use case, I just needed to move it to a t2.micro instance. Just change the machine type in the Elasticache console right? Wrong. Unknown to me, I had used the default config for Redis on a t2.small Elasticache cluster with two nodes. A t2.micro only supports one node.

This wasn’t documented anywhere and to add to the frustration the AWS console would allow me to change the cluster machine type, but just wouldn’t show the T2.micro anywhere in the list. It took me running commands directly from the AWS CLI, to discover that in fact, I could not change my Node type to micro:

aws elasticache list-allowed-node-type-modifications — cache-cluster-id roadbotomicro

The solution was simply standing up an entirely new Elasticache instance with one node on a t2.micro instance**.**

Cost of Redis on t2.micro: $o

Moral of the story: AWS is a fantastic platform for hobbyists. It is also a financial behemoth that makes gobs of money by over provisioning its resources for big companies that forget about how much these little expense add up. However, as a hobbyist you need to be extra careful about your own budget. Much the same way we have learned to be cautious with installers or download buttons, we need to be careful about the default configuration for AWS products.

When you avoid the shiny defaults…