Welcome all my new follow_ers, it’s a pleasure to have you aboard and I hope you enjoy this latest attempt at me becoming a millionaire!_ The Week’s Million Dollar Idea So last weeks’ million dollar idea of creating a highly scalable website monitor seems to have somewhat failed. _Lambda functions are a truly awesome bit of technology, if you haven’t read up about my theory that serverless and more…_hackernoon.com Creating A Highly Scalable Website Monitoring Service in half an hour using AWS Lambda I don’t think it was enough for these billionaire angel investors but whatever… This time round we are going to crack the nut and I am going to buy myself a big yacht. “disruptive” This week, we are going to pivot on our initial idea, we are going to create a service that, get this, analyses websites. Sounds cool right? The best part about this new service is going to meet the following requirements: it’s going to be done on the cheap. Our frontend will be hosted in S3. Unless we hit the truly big time our lambda’s will not cost a dime. It’s going to be highly scalable! This will be ready for the prime time once we IPO. It’s going to be stable and extendible — I can’t have the platform falling over if new requirements come in. If it meets these requirements it’s sure to be a hit! So, in order to achieve these hefty requirements we are going to be leveraging AWS Lambda, the Gateway and S3. We are going to be creating a series of API endpoints that will be hit by our frontend written in Vue.JS. These endpoints will then go on to hit a number of different Lambda functions written in any language we wish! I’m going to be leveraging Go, Python and Node because, why not? it’s all a bit of fun! API The Frontend — In Vue.JS on S3 So first things first, we need a flashy looking frontend. If it doesn’t look cool, the investors won’t be enticed to part ways with their hard earned money. This will be a very simplistic frontend that will take in a webpage URL through a simple input and it will then hit the 3 distinct endpoints that we are going to be defining in our API Gateway. These 3 endpoints are going to calculate things like, the total page load speed, the first time to byte, and the number of links and 404s present on that page. The finished product for this looks a little like this: Inspired By: https://dribbble.com/shots/3203637-Dashboard-Report/attachments/683409 Full Source code for the frontend as well as all of the following Lambda Functions can be found here: https://github.com/elliotforbes/website-analyzer Time to First Byte — In Go So, the first Lambda function will utilize the very recently announced Go runtime! If you’ve never written a Go Lambda before then I recommend you check out my very brief introductory Youtube video on how to do so. (Remember to like and subscribe to support the channel! :D) In our Lambda function we are going to be taking in a URL through our Request object and then trying to accurately calculate the time to first byte. Our finished version of our Go based endpoint will eventually look like this: This essentially parses the URL that is passed in through the request and then attempts to establish a TCP connection with that URL on port 80. We then send a GET HTTP 1.0 request down this TCP connection and we calculate the length of time it takes for 1 byte to travel back. This calculated time value is our time-to-first-byte. Once it’s calculated this time to first byte it returns this back to the caller of our Lambda function. Page Speed — In Python The second lambda function we’ll be defining is the one that calculates our page load speed. This is going to be rather crude and will simply calculate the time taken to perform a call and return that to the caller. urllib.request.urlopen() You’ll notice that there is no fancy error handling to check if the passed in is valid, this could be done in the future! url 404 Checker — In Node Finally, we come to our last Lambda function. Our 404 checker endpoint will simply take in the same URL as the other two functions and it will utilize the node module in order to adequately test our page for links and broken links. broken-link-checker The broken-link-checker module made this endpoint ridiculously simple to implement. I would have saved a lot of time just assuming such a node module already existed as opposed to trying to come up with my own implementation… Key Takeaway — If you are writing anything in node, chances are there is already a package out there that does what you want. Our API — Using API Gateway So now that we have our 3 distinct Lambda functions we need to define a way to access these through a HTTP Endpoint. The best way to do this is through the API Gateway! Essentially this allows you to pair up paths and methods with their respective lambda functions. It’s incredibly simple and once you are happy with your endpoints, you simply click the ‘actions’ button and click deploy. HTTP Deployments are done to particular stages, so if you wish to deploy to a test environment you have to deploy to a stage, for production you deploy to a stage and so on. test prod After a little bit of tweaking and enabling things like the dreaded , I had a working API that I could point my frontend at. CORS Once I was happy with my frontend locally, I pushed it up to a public S3 bucket and made it accessible by the public. The URL for this newly deployed site can be found here: https://s3-eu-west-1.amazonaws.com/website-analyzer/index.html Note: I’m positive that with a little tinkering you could break this in some way, making it production-worthy will be done after I’ve gone public! Conclusion And there you have it folks! We now have a fully functioning website analysis that we can demonstrate to investors and hopefully attract investments! tool My and my team after our first seed round. This website is not only incredibly resilient, it’s also incredibly cheap. The frontend is hosted on a service that features a ridiculously high SLA in terms of up time and the backend is hosted on Lambda functions that are not only massively scalable, but also incredibly resilient. There isn’t much that could bring down this site that easily. This will not only insure that our customers don’t see any major impact, it will also keep management and investors happy as they have a product that can be relied upon. Hopefully you found this article super useful as well as somewhat entertaining! I also hope that it’s demonstrated to you just how powerful the likes of Lambda and the API Gateway can be when used in conjunction with one another! If you enjoyed this then please feel free to let me know on twitter: . I’m also on LinkedIn should you wish to connect: . I’m also currently working on a new book titled “An Introduction to Cloud Development” — if you wish to support me and learn more about the cloud then check it out here: Elliot Forbes https://linkedin.com/in/elliotforbes _This book provides a gentle introduction to the world of cloud development. We will be taking an in-depth look at…_leanpub.com An Introduction to Cloud Development

Super

Twitter

YouTube

Too Long; Didn't Read

Developers: Build Less. Deliver More. [Learn how]	

Creating a Scalable Website Analysis Tool using Lambda and API Gateway and S3

Too Long; Didn't Read

People Mentioned

Companies Mentioned

Elliot Forbes

About Author

TOPICS

THIS ARTICLE WAS FEATURED IN...

Creating a Scalable Website Analysis Tool using Lambda and API Gateway and S3

Too Long; Didn't Read

People Mentioned

Companies Mentioned

Elliot Forbes

About Author

TOPICS

THIS ARTICLE WAS FEATURED IN...

RELATED STORIES