Serverless — Beyond the over simplistic todo apps into the world of reality (part 1)

Besides the shitty todo apps that are so conveniently demonstrated for all new technologies I wondered what the would be like. world of serverless really This is of building a real world example using the and how you can monetize your new infinitely scalable solutions. part 1 of 3 serverless framework In this part I talk about how to use (the framework) to build an ffmpeg wrapper based on bucket events. The code for my . serverless open sourced serverless-ffmpeg can be found here Intro Whenever a media file is uploaded to an s3 bucket, we want to post-process that into various formats using ffmpeg. What we want to do: Serverless allows us to trigger a function once a file has been added/delete to a bucket, perfect for our post-processing action. Also we pay for the time we process, no more, no less. Last but not least, we can batch process and not worry about how many files are uploaded at once. Why serverless: The code 1. The serverless.yml This is THE file where it all is tied together and if you’ve ever built anything bigger than a todo app, you know this will be a bottleneck — and it is. However! there are some fancy stuff you can do to make it a bit better, such as reference another file (see notes below) or use . javascript to generate parts of the yml A couple of notes: As you can see I’m using things like: to grab values out of another yaml file. This is one way to abstract away the complexity of the serverless.yml and focus on a few parameters that may be important for deployment. ${file(./config.yml):source_bucket} Line 29, we define the parameters that will be passed to the ffmpeg binary. They’re passed as env variables in the lambda function! (as you’ll see in the coming sections) Line 30–60 is for having access to the buckets that we need for the app. Line 67 defines the event handler, funny enough it also creates the bucket 2. The webpack config We need webpack to bundle our javascript nicely, as well as use es6/7 features to write some modern day JS (rather than being bound to node 6.10 that lambda allows us). Let’s take a look at the webpack config + babelrc files to make that happen. On line 8, we’re using the in order to auto generate entries based on our functions in serverless. Thanks to the smarts of the serverless-webpack lib. slsw.lib.entries On line 23–24 we’re copying the binaries that will be used in our app. The reason we’re doing that here, is because we need the binaries to have executable permissions (+x). 3. The handler code This is where serverless starts! As we specified in our we want a function called to run from our file, whenever there is an object created event in our source bucket. serverless.yml main handler.js Line 7 calls a function to extract the newly uploaded file’s location in the source bucket. Thanks to our fancy webpack config, we can now use await/async which is what we do while we download the recently uploaded file from the source bucket into the lambda process. Line 14 we run the ffmpeg function, which calls the binary in turn (see next section). Line 15 we upload the results into the destination bucket. Line 23 showcases how we use the predefined parameters as environment variables. 4. The ffmpeg app code Let’s take a look at some of the interesting parts of the app code to see how this is all tied together. Line 4 Make sure our lambda root is in the PATH env variable, this way we can call binaries with directly located in our project root! spawn Line 21 we’re merging the arguments that we pass to the ffmpeg process. Line 29 we call spawn with our arguments and hook up some event handlers Line 34–35 we make sure to capture stdout and stderr, without this we literally have no idea what’s going on, if anything goes wrong. 4. Unit testing of course! As part of any real world application unit testing is important. But how can we test serverless functions? We can unit test parts of them! But how do we unit test calling a binary to do things? We set up our testing infrastructure to support this! In order for us to achieve this we’ll need (ava is great!), it allows us to quickly get started with a minimalistic configuration (see this for how to configure it properly). ava package.json Import ava and some helpers to deal with files. Line 8 Run our ffprobe function which returns a json result ( ) see it’s code to see how Line 16 Run ffmpeg on a test file and expect a file to exist at the end of the test Line 26 clean up after testing Deploying One of the great things about serverless is how easy it is to deploy and tear down. Since serverless framework does it all in a cloudformation templates it’s fast and easy. No more “what command did I run where”. To deploy the whole stack, we run: . This will show the output of the cloudformation status so that we can make sure all resources are created as they should. sls deploy -v That’s it! The Result You can view this little video to see how it all works in the end. _Recorded by kvaggelakos_asciinema.org serverless-ffmpeg Conclusion I hope you enjoyed this quick deep dive into serverless and a real world example! In the next parts we’ll be looking at more fun real world scenarios and how to monetize them, so stick around! A couple of real life problems we encountered in this part: We want to add webpack to make this worth our time We need to use a plugin with webpack and keep permissions for this binary to actually work (together with serverless framework) A bucket is created if we attach events to it! Seems random, I know. But I don’t make the rules. In the next parts I’ll reveal more life scenarios and problems solved. If you have any questions or comments please post them here and I’ll do my best to help. Appendix — find them, use them Serverless Plugins One thing that wasn’t obvious to me when I started out was how important plugins were for the serverless framework. The two plugins we’ll be using in this part are (more plugins in part 2 and 3): Because as of today lambda supports node 6.10, which does not fully support es6. We want that fancy es6/7 and some other nice packaging features. serverless-webpack aws You’ll quickly realize you’re deploying and tearing down your stack with the serverless framework many times as you’re testing things. If you’re working with buckets, this simply helps us remove the buckets and the content in them when we destroy our stack. ( ) serverless-s3-remover WARNING: You will loose content with this