Lazy Image Optimiser Using AWS S3 and NodeJs

Below I will show you how we can simply optimise and edit images in our media library, thus dramatically reducing the time required for our website to load. Our optimised version of the image will only be created at the moment of the first user request. The Case Imagine that in our media library momentarily we have one image. name: grumpy_cat.jpgurl: media.YOUR__WEBSITE.com/grumpy_cat.jpgwidth: 2500pxsize: 2mb We would like to use our picture in the following places: As a blog list item (Recommended image width 150px) As a featured image in a blog post page (Recommended image width 760px) As part of the mobile version of the blog post page (Recommended image width 200px) Below I will describe issues of the two most common ways of handling this task: The css way We can easily implement image modification with using css, but it doesn’t resolve the issue of loading time, because even though our image is displayed small but the file is still the same size. The hard coded way Why don’t we create three different sizes of the image and upload it to the media library? This solution is better, but think about what happens when we don’t have one post but thousands of posts. The Solution — Make it easier We can implement the css functions into an image modification process (crop, cover, etc.), we can create smaller, more optimised pictures with the real width that we wish to display it in. (In our case the three new images: 150px, 760px, 200px ) We can automate image uploading into our media library The optimised version of the image will only be created at the moment of the first user request Sounds good right? Definitions Storage Server Our media library, where we can handle our images (In our case it’s an S3 Bucket) AWS In this tutorial we are using the following url as our media library domain: media.YOUR__WEBSITE.com Image Processor Service Of the endpoint of our API, which in our case does all the resource heavy file actions The API url is api.YOUR__WEBSITE.com Client A user who send’s a request to our media library Task --key-value key value pattern between the original image path and the file extension media.YOUR__WEBSITE.com/grumpy_cat--width-100.jpg In this tutorial we are using AWS S3 as the Storage Server and express Application as our Image Processor Service. These can be replaced with your personal preferences. You can use your local file server or other file storage provider (Azure, Google Cloud, etc.) instead of S3. You can also replace NodeJs with an image processor written in a different programming language or you can use a serverless function provider, for example AWS LAMBDA. NodeJs The process First Part — Setting up AWS S3 bucket The first thing we are going to do, is to set up our bucket and configure it. To redirect all 404 requests to our Image Processor Service Add permissions to upload and fetch objects using AWS SDK 1 . Register on AWS and create a new bucket In this tutorial I will not give you all the details, if you need more information I recommend the following article https://docs.aws.amazon.com/quickstarts/latest/s3backup/step-1-create-bucket.html 2. Select Your bucket 3. Configure Bucket Permissions and CORS parameters Bucket permissions and the CORS settings are instrumental in order to be able to upload optimised images from our image processor service using AWS SDK. I. Create GetObject Bucket policy permissions (make s3 bucket public) “Version”: “2012–10–17”,“Statement”: [{“Effect”: “Allow”,“Principal”: “*”,“Action”: “s3:GetObject”,“Resource”: “arn:aws:s3:::YOUR__BUCKET__NAME/*”}]} If you need more information about the Bucket Policies: https://aws.amazon.com/blogs/security/writing-iam-policies-how-to-grant-access-to-an-amazon-s3-bucket/ II. Configure CORS * GET 3000 Authorization https://api.YOUR__WEBSITE.com PUT 3000 * This setting enables the Image Processor Service to make modifications in Our Storage Server using AWS SDK and to set our media library public. 4. Enable and configure static website hosting mode Setting Our bucket to Website Hosting Mode is imperative in order to process 404 requests to Our Image Processor Service. Redirection rules: 404 https api.YOUR__WEBSITE.com media/optimiser?url= 302 5. How to use your S3 in Website Hosting Mode When your bucket is running in Website Hosted Mode, you need to use it with a different endpoint url. AWS S3 has 2 different usages: Using it as an API (https and .s3.) https://s3.REGION.amazonaws.com/YOUR_BUCET_NAME/grumpy_cat.jpghttps://YOUR_BUCET_NAME.s3.REGION.amazonaws.com/grumpy_cat.jpg Using it as a Website (http and .s3-website.) http://YOUR_BUCET_NAME.s3-website.REGION.amazonaws.com/grumpy_cat.jpg In this tutorial we need to use our Bucket in website mode, otherwise the redirection will not work. 6. Using our bucket in website hosting mode with https protocol (optional) Unfortunately the S3 in website mode doesn’t support HTTPS protocol. If you are using https protocol in your website you need to use a proxy server to your bucket url. (AWS cloudfront, nginx, express-proxy, etc…) Nginx example: server_name www.media.YOUR__WEBSITE.com media.YOUR__WEBSITE.com;return 301 https://$host$request_uri;} server {listen 443 ssl;listen [::]:443 ssl; ssl\_certificate CERTIFICATE\_\_PATH; ssl\_certificate\_key CERTIFICATE\_\_PATH;

server\_name www.media.YOUR\_\_WEBSITE.com media.YOUR\_\_WEBSITE.com;

location / { proxy\_pass [http://YOUR\_BUCET\_NAME.s3-website.REGIO.amazonaws.com ](https://s3.eu-central-1.amazonaws.com/unleash-image-resizer-test/grumpy_cat.jpg) }} You can now use your bucket with your address https://media.YOUR__WEBSITE.com **_7. Testing our S3 setup_**Now when someone navigates to our s3 bucket url and the media asset does not exist, our bucket will automatically navigate to the redirect url.(api.YOUR__WEBSITE.com/media/optimiser?url=) Please see the process (above) to see exactly how it works. Of course at this stage you will receive an error message as our Image Processor Service is not complete yet. Example: fetched image url:http://YOUR_BUCET_NAME.s3-website.REGIO.amazonaws.com/grumpy_cat--width-100.jpg or fetched image url with using https protocol and proxy server: https://media.YOUR__WEBSITE.com/grumpy_cat--width-100.jpg --------- redirect endpoint = https://api.YOUR__WEBSITE.com/media/optimiser?url=grumpy_cat--width-100.jpg Now we can start writing our Image Processor ServiceEnd of the first part… Second Part —Writing Image Processor Service in NodeJs with Express and Sharp Definitions: url:ORIGINAL_IMAGE--key-Value--key-value.FILE_EXTENSION original url:ORIGINAL_IMAGE.FILE_EXTENSION bucketUrl:media.YOUR__WEBSITE.com orYOUR_BUCET_NAME.s3-website.REGION.amazonaws.com 1. Install and configure Express In this tutorial we are using the starter example with ES6 import syntax http://expressjs.com/en/starter/hello-world.html import express from 'express' const app = express()const port = 3000 app.get('/', (req, res) => res.send('Hello World!')) app.listen(port, () => console.log(`Example app listening on port ${port}!`)) 2. Create The Function (simplified) Warning! This code snippet is not a production ready script, if you’d like to use it, you need to extend it with validators, error handlers, more tasks, etc… index.js import Boom from 'boom' // Super useful error handler libraryimport { putObject, getObject } from './aws.helper'import { sharpImageConverter } from './sharp.helper' const bucketUrl = process.env.bucketUrl app.get('/media/optimise', async (req, res) => {try {const { query } = reqconst { url } = queryconst acceptedFileExtensions = /\.(jpg|jpeg|png)$/iconst acceptedTasks = ['width'] // check the original request url if (!url) throw Boom.notFound()

// check the file extension if (!acceptedFileExtensions.test(url)) throw Boom.notFound()

// check the tasks const \[rawFilename, fileExtension\] = url.split('.') const splittedFilenameByTasks = rawFilename.split('--')

// if we don't have tasks if (splittedFilenameByTasks.length === 1) throw Boom.notFound()

// checking tasks const tasks = {} splittedFilenameByTasks.splice(1) .forEach((task) => { const \[key, value\] = task.split('-')

// if we have unknown tasks if (!acceptedTasks.includes(key)) throw Boom.notFound();

tasks\[key\] = value return });

// download the original image const originalFilename = \`${splittedFilenameByTasks\[0\]}.${fileExtension}\` const fetchedFileDataFromAWS = await getObject(originalFilename)

// create buffer const { Body, ContentType } = fetchedFileDataFromAWS const image = new Buffer.from(Body)

// filesystem tasks const convertedImageBuffer = await sharpImageConverter(image, tasks)

// upload to the s3 await putObject(url, convertedImageBuffer, ContentType);

// redirect to the new image url return res.redirect(\`${bucketUrl}/${url}\`); } catch (_err) {...handle errors}}) aws.helper.js https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html import AWS from 'aws-sdk'import Boom from 'boom' const s3Service = new AWS.S3({accessKeyId: process.env.AWS_ACCESS_KEY_ID,secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,region: process.env.AWS_REGION,}) export const getObject = key =>new Promise((resolve, reject) => {const params = { Bucket: bucketName, Key: key } if (!key) reject(Boom.notFound())

s3Service.getObject(params, (err, data) => { if (err) reject(err) resolve(data) }) }) export const putObject = (key, body, ContentType) =>new Promise((resolve, reject) => {const params = {Bucket: bucketName,ContentType,Key: key,Body: body,} if (!key) reject(Boom.notFound())

s3Service.putObject(params, (err, data) => { if (err) reject(err) resolve(data) }) }) sharp.helper.js https://github.com/lovell/sharp import sharp from 'sharp' export const sharpImageConverter = async (_image, params) => {const { width } = params const modifiedImage = await sharp(_image).resize(width ? parseInt(width, 10) : undefined).crop().toBuffer() return modifiedImage} 3. Deploy the script We can now deploy our system which will enable us to speed up our website loading speed. We can guarantee your SEO results will improve. I hope you find this tutorial useful, If you have any questions feel free to comment below.