Zeet is sponsoring this blogpost for the next month. I tried it out the other day. It's like serverless but for running entire back ends. You can host and scale apps automagically. Pretty neat. Handling large images has always been a pain in my side since I started writing code. Lately, it has started to have a huge impact on page speed and SEO ranking. If your website has poorly optimized images it won’t score well on . If it doesn’t score well, it won’t be on the first page of Google. That sucks. Google Lighthouse TL;DR I’ve built and open-sourced a snippet of code that automates the process of creating and deploying an image resize function and an S3 bucket with one simple command. Check out . the code here But if you want to follow along and learn how to do it yourself, keep reading. Where do we start? Luckily enough, there is a way to solve the issue of bad image optimization with little to no hassle. Today we’ll build an function to resize images on-the-fly. AWS Lambda The images will be stored in an bucket and, once requested, will be served from it. If you need a resized version, you’ll request the image and supply the height and width. This will trigger a function. It’ll grab the existing image, resize it, return it back to the bucket and then serve it from the bucket. S3 This scenario will only resize an image for a given set of dimensions once. Every subsequent request to the image of that size will be served from the bucket. Pretty cool huh? Here’s a diagram, because who doesn’t love diagrams. Because I already assume you know how to use the and have already been introduced to the basics of , and , I’ll immediately jump into the gist of things. Here’s an overview of what we’ll be doing. Serverless Framework serverless Docker AWS Create the project structure Create a secrets file Write the AWS Lambda function configuration Write AWS Lambda function source code Write S3 Bucket configuration Deploy with DockerTest with Dashbird Note : Please install Docker and Docker Compose before continuing with this tutorial. The funny thing here is that we need to use Docker to deploy this service because of . This image-resize module has binaries that need to be built on the same operating system as it will run on. Because AWS Lambda is running on Amazon Linux, we need to the packages on an Amazon Linux instance before running . Sharp npm install sls deploy With that all out of the way let’s build some stuff. Create the project structure The best way of explaining this complex structure will be with an image. Folders in green and files in blue. Check out the for further insight. repo Create a secrets file We’ll create a secrets.env file in the folder to hold our secrets and inject them into our Docker container once it spins up. secrets SLS_KEY=XXX # replace your IAM User key 
SLS_SECRET=YYY # replace your IAM User secret 
STAGE=dev 
REGION=us-east BUCKET=images.yourdomain.com with with -1 Note : The deploy.sh script will create a secondary secrets.json file only to hold the domain name of our API Gateway endpoint. Write the AWS Lambda function configuration Open up the folder, and start by creating the file. Paste this in. functions package.json { : , : , : , : , : { : , : }
} "name" "image-resize" "version" "1.0.0" "description" "Serverless image resize on-the-fly." "main" "resize.js" "dependencies" "serverless-plugin-tracing" "^2.0.0" "sharp" "^0.18.4" No need to run any install, because we need to run the Docker container first. Note that we’re adding a tracing plugin to enable because it’s awesome. X-Ray Moving on, let’s create the file to configure our function. serverless.yml service: image-resize-on-the-fly-functions

plugins:
  - serverless-plugin-tracing

provider:
  name: aws
  runtime: nodejs8 stage: ${ :STAGE, }
  profile: serverless-admin # or change to tracing: region: ${ :REGION, }
  environment:
    BUCKET: ${ :BUCKET}
    REGION: ${ :REGION}
  iamRoleStatements:
    - Effect: Action:
        - Resource: - Effect: Action:
        - Resource: - Effect: Action:
        - - Resource:
        - functions:
  resize:
    handler: resize.handler
    events:
      - http:
          path: resize
          method: get .10 env 'dev' 'default' true env 'us-east-1' env env "Allow" "s3:ListBucket" "arn:aws:s3:::${env:BUCKET}" "Allow" "s3:PutObject" "arn:aws:s3:::${env:BUCKET}" "Allow" "xray:PutTraceSegments" "xray:PutTelemetryRecords" "*" As you can see we’re grabbing a bunch of values from the and also allowing the function to access the we specified and some X-Ray telemetry. env BUCKET Write AWS Lambda function source code Only the code left now. Here’s what we’ll do. Because images can get quite large and we don’t want to risk loading several megabytes into the lambda function’s memory, I’ll show you how to use to read the image from an S3 stream, pipe it to Sharp, and then write it back to S3, once again, as a stream. Node.js streams Ready? Let’s do it in two steps. First, define the helper functions we need in order to create the streams, then create the handler function itself which our lambda will invoke. stream = ( ) AWS = ( ) S3 = AWS.S3({ : }) sharp = ( ) BUCKET = process.env.BUCKET URL = readStreamFromS3 = { S3.getObject({ Bucket, Key }).createReadStream()
} writeStreamToS3 = { pass = stream.PassThrough() { : pass, : S3.upload({ : pass,
      Bucket, : ,
      Key
    }).promise()
  }
} streamToSharp = { sharp()
    .resize(width, height)
    .toFormat( )
} // resize.js // require modules const require 'stream' const require 'aws-sdk' const new signatureVersion 'v4' const require 'sharp' // create constants const const `http:// .s3-website. .amazonaws.com` ${process.env.BUCKET} ${process.env.REGION} // create the read stream abstraction for downloading data from S3 const ( ) => { Bucket, Key } return // create the write stream abstraction for uploading data to S3 const ( ) => { Bucket, Key } const new return writeStream uploadFinished Body ContentType 'image/png' // sharp resize stream const ( ) => { width, height } return 'png' Here we’re defining constants and the constructor functions for our streams. What’s important to note is that S3 doesn’t have a default method for writing to storage with a stream. So you need to create one. It’s doable with the helper. The abstraction above will write the data with a stream and resolve a promise once it's done. stream.PassThrough() Moving on, let’s check out the handler. Just underneath the code above, paste this in. exports.handler = (event) => { key = event.queryStringParameters.key match = key.match( ) width = (match[ ], ) height = (match[ ], ) originalKey = match[ ] newKey = + width + + height + + originalKey imageLocation = { readStream = readStreamFromS3({ : BUCKET, : originalKey }) resizeStream = streamToSharp({ width, height }) { 
      writeStream, 
      uploadFinished 
    } = writeStreamToS3({ : BUCKET, : newKey }) readStream
      .pipe(resizeStream)
      .pipe(writeStream) uploadedData = uploadFinished .log( , {
      ...uploadedData, : URL, : imageLocation
    }) { : , : { : imageLocation }, : }
  } (err) { .error(err) { : , : err.message
    }
  }
} async const // match a string like: '/1280x720/image.jpg' const /(\d+)x(\d+)\/(.*)/ // get the dimensions of the new image const parseInt 1 10 const parseInt 2 10 const 3 // create the new name of the image, note this has a '/' - S3 will create a directory const '' 'x' '/' const ` / ` ${URL} ${newKey} try // create the read and write streams from and to S3 and the Sharp resize stream const Bucket Key const const Bucket Key // trigger the stream // wait for the stream to finish const await // log data to Dashbird console 'Data: ' BucketEndpoint ImageURL // return a 301 redirect to the newly created resource in S3 return statusCode '301' headers 'location' body '' catch console return statusCode '500' body The query string parameters will look like this . Which means we grab the image dimensions from the parameters by using a regex match. The value is set to the dimensions followed by a and the original name of the image. This will create a new folder named with the image in it. Pretty cool. /1280x720/image.jpg newKey / 1280x720 image.jpg Once we trigger the streams and await the promise to resolve we can log out the data and return a 301 redirect to the location of the image in the bucket. Note that we will enable static website hosting on our bucket in the next paragraph. Write S3 Bucket configuration The bucket configuration will mostly consist of default templates, so if you’re a bit rusty, I’d advise you brush up a bit. But, the gist of it is simple enough to understand. Open up the folder and create a file. CloudFormation bucket serverless.yml service: image-resize-on-the-fly-bucket

custom:
  secrets: ${file(../secrets/secrets.json)} # will be created our deploy.sh script

provider:
  name: aws
  runtime: nodejs8 stage: ${ :STAGE, }
  profile: serverless-admin # or change to region: us-east environment:
    BUCKET: ${ :BUCKET}

resources:
  Resources:
    ImageResizeOnTheFly:
      Type: AWS::S3::Bucket # creating an S3 Bucket
      Properties:
        AccessControl: PublicReadWrite
        BucketName: ${ :BUCKET}
        WebsiteConfiguration: # enabling the website option
          ErrorDocument: error.html
          IndexDocument: index.html
          RoutingRules: # telling it to redirect every error
            - 
              RedirectRule:
                HostName: ${ :custom.secrets.DOMAIN} # API Gateway domain
                HttpRedirectCode: # temporary redirect HTTP code
                Protocol: ReplaceKeyPrefixWith: # route
              RoutingRuleCondition:
                HttpErrorCodeReturnedEquals: KeyPrefixEquals: ImageResizeOnTheFlyPolicy:
      Type: AWS::S3::BucketPolicy # add policy public read access
      Properties: 
        Bucket: 
          Ref: ImageResizeOnTheFly
        PolicyDocument: 
          Statement: 
            - 
              Action: 
                - Effect: Resource:
                Fn::Join: 
                  - - 
                    - - 
                      Ref: ImageResizeOnTheFly
                    - Principal: in .10 env 'dev' 'default' -1 env env static for 404 self "307" "https" "${self:provider.stage}/resize?key=" "404" "" for "s3:*" "Allow" "" "arn:aws:s3:::" "/*" "*" Once we enable the static website hosting option on our bucket, it will behave like any website. This lets us serve images from it and utilize the 404 error redirect rule. If no image is found, the bucket will trigger the 404, which redirects to our lambda function. Then resize the original image to the requested dimensions, and put it back to the bucket at the exact route it was requested from. Just like magic! Deploy with Docker Here comes the fun part! We’ll create a and file to create our Amazon Linux container and load it with values. That's easy. The hard part will be writing the bash script to run all commands and deploy our function and bucket. Dockerfile docker-compose.yml .env Starting with the , here's what you need to add. Dockerfile FROM amazonlinux

# Create deploy directory
WORKDIR /deploy

# Install system dependencies
RUN yum -y install make gcc*
RUN curl --silent --location https: RUN yum -y install nodejs

# Install serverless
RUN npm install -g serverless

# Copy source
COPY . .

# Install app dependencies
RUN cd /deploy/functions && npm i --production && cd /deploy

#  Run deploy script
CMD ./deploy.sh ; sleep m //rpm.nodesource.com/setup_8.x | bash - 5 Because Amazon Linux is rather basic, we need to install both gcc and Node.js at the get-go. Then it’s as simple as any Dockerfile you’ve seen. Install the Serverless Framework globally, copy over the source code, install the npm modules in the directory and run the script. functions deploy.sh The file is literally only used to load the values. docker-compose.yml .env version: services:
  image-resize-on-the-fly:
    build: .
    volumes:
      - ./secrets: secrets
    env_file:
      - ./secrets/secrets.env "3" /deploy/ That’s it. The Docker part is done. Let’s write some bash. Starting out simple, we’ll define the initial variables, configure our serverless installation and deploy our function. # deploy.sh

# variables
stage=${STAGE}
region=${REGION}
bucket=${BUCKET}
secrets= # Configure your Serverless installation to talk to your AWS account
sls config credentials \
  --provider aws \
  --key ${SLS_KEY} \
  --secret ${SLS_SECRET} \
  --profile serverless-admin

# cd into functions dir
cd /deploy/functions

# Deploy '/deploy/secrets/secrets.json' "------------------" ' ...' "------------------" function echo echo Deploying function echo sls deploy Once we have deployed it, we need to grab the domain name of our API Gateway endpoint and put it in a file which will be loaded into our file from the directory. To do this I just did some regex magic. Add this at the bottom of your . secrets.json serverless.yml bucket deploy.sh # find and replace the service endpoint [ -z ${stage+dev} ]; then echo ; echo ; fi

sls info -v | grep ServiceEndpoint > domain.txt
sed -i domain.txt
sed -i domain.txt
domain=$(cat domain.txt)
sed domain.txt > id.txt
id=$(cat id.txt)

echo echo echo echo echo echo rm domain.txt
rm id.txt

echo > $secrets if "Stage is unset." else "Stage is set to '$stage'." 's@ServiceEndpoint:\ https:\/\/@@g' "s@/$stage@@g" "s@.execute-api.$region.amazonaws.com@@g" "------------------" "Domain:" "  $domain" "------------------" "API ID:" "  $id" "{\"DOMAIN\":\"$domain\"}" Now you have a file in the directory. All that's left is to run the bucket deployment. Paste this final snippet at the bottom of the script. secrets.json secrets deploy.sh cd /deploy/bucket

# Deploy bucket config
echo echo sls deploy

echo echo echo echo echo "------------------" 'Deploying bucket...' "------------------" 'Bucket endpoint:' "  http://$bucket.s3-website.$region.amazonaws.com/" "------------------" "Service deployed. Press CTRL+C to exit." There we have it! The coding part is done. Would you believe me we only need to run one command now? Open up a terminal window in the root of your project and run: $ docker-compose up --build Let it do its magic, and you’ll see everything get created automagically! Please note that the terminal will show you the bucket endpoint which you’ll use to access the images. Test with Dashbird The final step is to check if it all works. Let’s upload an image to the bucket, so we have something to resize. Go ahead and find an image you like and want to resize, or just take . this one It’s rather huge and around 6 MB in size. Here’s the command you want to run to upload it. $ aws s3 cp --acl public-read IMAGE s3: //BUCKET So, if your bucket name is and your image name is , then it should look like this if you run the command from the directory where the image is located. images the-earth.jpg $ aws s3 cp --acl public-read the-earth.jpg s3: //images Keep in mind you need to have the installed on your machine, or upload the image through the AWS Console. AWS CLI Now, try out to request the image through the bucket. Type into your browser the S3 bucket URL. http: //BUCKET.s3-website.REGION.amazonaws.com/the-earth.jpg You’ll see the original image appear. But, now add dimensions to the URL. http: //BUCKET.s3-website.REGION.amazonaws.com/400x400/the-earth.jpg This will trigger the resize function and create a 400x400 pixel version of the image. It’ll take a few hundred milliseconds to create and once it’s done, the browser will redirect you to the newly created image. When you try to refresh the URL you’ll see that it’s now fetching the new image, without calling the resize function. Let’s check the logs in and make sure it all works fine under the hood. Dashbird Looks fine, but I did make some beginner mistakes when I first tried setting up the function. One of them was forgetting to require a module before using it. Luckily I immediately got an alert that explained what was wrong. are life-savers. Slack alerts The way I fixed the issue was with the feature. It let me check the invocation logs with a few seconds of latency so I could debug the issue. Pretty cool. live-tailing Wrapping up To conclude this little session, I’d like to point out using serverless as a helper to support your existing infrastructure is incredible. It’s language agnostic and easy to use. showdev use clusters of containers for our core features that interact heavily with our databases while offloading all the rest to lambda functions, queues, streams and other serverless services on AWS. We at Dashbird And, of course, once again, give it a star if you want more people to see it on GitHub. If you follow along with the instructions there, you’ll be able to have this image resize on-the-fly microservice up and running in no time at all. here’s the repo If you want to read some of my previous serverless musings head over to or my profile join my serverless newsletter! I had an absolute blast writing this open-source code snippet. Writing the article wasn’t that bad either! Hope you guys and girls enjoyed reading it as much as I enjoyed writing it. If you liked it, slap that tiny unicorn so more people here on dev.to will see this article. Until next time, be curious and have fun. Originally published at dev.to .

Amazon

BUNCH

Google

Slack

A crash course on Serverless APIs with Express and MongoDB

Building a Serverless REST API with Node.js and MongoDB

This Month's Sponsor is Tracetest!

Read My Stories

Too Long; Didn't Read

A crash course on Serverless with AWS — Image resize on-the-fly with Lambda and S3

A crash course on Serverless with AWS — Image resize on-the-fly with Lambda and S3

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

A crash course on Serverless APIs with Express and MongoDB

101 Stories To Learn About Cloud Infrastructure

10 Things in Engineering We Don't Spend Enough Time On

10 Things I Did To Increase CloudTrail Logs Security

10 reasons to give cloud computing a go

A crash course on Serverless APIs with Express and MongoDB

101 Stories To Learn About Cloud Infrastructure

10 Things in Engineering We Don't Spend Enough Time On

10 Things I Did To Increase CloudTrail Logs Security

10 reasons to give cloud computing a go

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps