Programmatic S3 Object Storage Integration With NodeJS

Introduction

S3 compatible storage services are available from a number of cloud providers such as Vultr and Linode. This guide demonstrates how the AWS SDK for JavaScript and the Multer storage engine can be used to create a Node.js application to upload files to S3-compatible storage service buckets. The application is comprised of a Node.js server component and a web-based client component.

Prerequisites

To complete this guide, you will need to have:

• Set up your desired cloud infrastructure account for S3-compatible object storage.
• The access key and secret key for your S3 compatible object storage instance.
• Installed Node.js version 13.x or higher.
• Installed Node Package Manager, npm.

The operating system commands in this guide are for Ubuntu. However, since the programming steps are identical between Ubuntu and Windows, Windows users can also complete this guide by substituting the analogous Windows operating system commands for the listed Ubuntu commands.

Step 1 - Creating a New Node.js Project Folder

Create a new folder for the new Node.js project and change it to the directory once created. This guide will use the folder name s3-object-storage-nodejs.

sudo mkdir s3-object-storage-nodejs
sudo cd s3-object-storage-nodejs

Step 2 - Creating a Node.js package.json File in the New Project Folder

The package.json file contains basic data about the new Node.js project, such as the project name, as well as a list of project dependencies.

Step 2a - Run the package.json Questionnaire Script

Ensure you are in the new project directory created in the previous step. Run the npm init command to start a questionnaire script that will guide you through creating a new package.json file.

sudo npm init

Step 2b - Enter package.json Project Values

When the questionnaire script prompts you to enter:

• package name, use Enter to select the default value, which is the folder name s3-object-storage-nodejs set in Step 1.
• version, use Enter to select the default value 1.0.0.
• description, use Enter to leave the description value empty.
• entry point, type server.js , and then Enter.
• test command, use Enter to leave the test command value empty.
• git repository, use Enter to leave the git repository value empty.
• keywords, use Enter to leave the value of the keyword empty.
• author, type any value (e.g. S3 User) and then Enter.
• license, use Enter to select the default value ISC.

The questionnaire script will output a summary of the chosen package.json values.

{
  "name": "s3-object-storage-nodejs",
  "version": "1.0.0",
  "description": "",
  "main": "server.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "S3 User",
  "license": "ISC"
}

When prompted Is this OK? type yes and then Enter to save the new package.json file.

Step 2c - Add Start Key to packakge.json

Use a text editor to add "start": "node server.js", above the test key in package.json. The updated package.json file will have the following structure:

{
  "name": "s3-object-storage-nodejs",
  "version": "1.0.0",
  "description": "",
  "main": "server.js",
  "scripts": {
    "start": "node server.js",
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "S3 User",
  "license": "ISC"
}

Step 3 - Installing Node.js Project Dependencies

Navigate to the main project folder s3-object-storage-nodejs.

Step 3a - Install Express.js

Express.js provides the framework for the server component of the application.

npm install express

Step 3b - Install AWS SDK for Javascript for S3 Clients

The AWS SDK for Javascript provides modular access to AWS resources, including S3-compatible storage services.

npm install @aws-sdk/client-s3

Step 3c - Install Multer

Multer middleware for Node.js is used for handling multipart/form-data, which is typically used for uploading files.

npm install multer

Step 3d - Install Multer S3

Multer S3 provides streaming-based S3 client integration with the Multer storage engine.

npm install multer-s3

Step 4 - Setting S3 Object Storage Credentials

Step 4a - Create .aws Folder and Empty credentials File

Create a new folder named .aws with an empty text file credentials in your user directory, not the project folder created in Step 1.

The path to the empty credentials file will be located at ~\.aws\credentials on Ubuntu and C:\Users\USERNAME\.aws\credentials on Windows. The AWS SDK for Javascript follows a default credentials provider chain and will look for your S3-compatible object storage credentials in either of the preceding directories depending on your operating system.

sudo mkdir .aws
sudo touch .aws\credentials

Step 4b - Edit credentials File to Set S3 Object Storage Credentials

Open the empty credentials file with a text editor and enter the profile name [default] followed by your S3-compatible object storage credentials.

[default]
aws_access_key_id=YOUR_S3_OBJECT_STORAGE_ACCESS_KEY_GOES_HERE
aws_secret_access_key=YOUR_S3_OBJECT_STORAGE_SECRET_KEY_GOES_HERE

Do not enclose the access key or the secret key with quotes or any other character. Also, the credentials file should be saved without a file extension.

Step 5 - Creating the Server-Side of the Application

In this step, you will create the server-side component of the application in your desired development environment (e.g. Visual Studio Code).

Step 5a - Create server.js File and Specify Required Project Modules

Create a new Javascript file in the project folder set in Step 1 named server.js. Add the following require statements at the beginning of the file:

const { S3, ListBucketsCommand } = require("@aws-sdk/client-s3");
const multer = require("multer");
const multerS3 = require("multer-s3");
const express = require("express");

Step 5b - Initialize a New Instance of the AWS SDK S3 Class

Initialization of the AWS SDK S3 class requires that you specify the endpoint and the region of the S3 service. Add the following code to the server.js file using the specified endpoint and region values for your S3-compatible object storage service. For example, the endpoint and region values used with Vultr Object Storage are https://ewr1.vultrobjects.com and ewr1 respectively.

// Enter the appropriate endpoint and region values for your service provider.
const s3 = new S3({
  endpoint: "YOUR_S3_SERVICE_ENDPOINT_URL_GOES_HERE",
  region: "YOUR_S3_SERVICE_REGION_GOES_HERE"
});

Step 5c - Define upload Method

Add the upload method which will call the Multer storage engine using client-side file upload form data.

const upload = multer({
    storage: multerS3({
    s3: s3,
    bucket: (request, file, cb) => {
      console.log(request.body.selectBucket);
      cb(null, request.body.selectBucket);
    },
    key: (request, file, cb) => {
      console.log(file);
      cb(null, file.originalname);
    }
  })
});

Step 5d - Add a New Express Application and Routes

Use the express() method to create a new Express application instance.

const app = express();

When the client-side (i.e. browser-side) of the application is created in Step 6, those static files will be placed in a public sub-folder within the project folder. The express.static middleware function is used to set the directory from which to serve these static assets.

app.use(express.static("public"));

The routes of the Express application need to be defined. In the following code, the first route sets the root of the application and serves the client-side index.html file, which will display a file upload form. The second route retrieves a list of your S3 compatible storage service buckets and returns that data to the client for use in the file upload form. The third route handles POST requests from the browser, passing file upload form data to the upload method defined earlier, and sending a response back to the client.

app.get("/", (request, response) => {
  response.sendFile(__dirname + "/public/index.html");
});

app.get("/buckets", (request, response) => {
  let listBuckets = async () => {
    try{
      const data = await s3.send(new ListBucketsCommand({}));
      console.log("Success fetching buckets:", data.Buckets);
      response.send(data);
    }
    catch(e){
      console.log("There was an error when trying to fetch object store buckets: ", e);
    }
  };
  listBuckets();
});

app.post("/upload", upload.array("uploadFilesInput"), (request, response) => {
  console.log("File(s) uploaded successfully.");
  response.send(request.files.length + " file(s) successfully uploaded.");
});

Step 5e - Set the Express Application to Listen for Connections

The last part of the server-side application sets the Express listen method so that server.js can listen for client connections on port 8080.

app.listen(8080, () => {
  console.log("The server is listening on port 8080.");
});

Step 5f - Complete server.js File

This is the completed server.js file after following steps 5a through 5e above.

const { S3, ListBucketsCommand } = require("@aws-sdk/client-s3");
const multer = require("multer");
const multerS3 = require("multer-s3");
const express = require("express");

const s3 = new S3({
  endpoint: "YOUR_S3_SERVICE_ENDPOINT_URL_GOES_HERE",
  region: "YOUR_S3_SERVICE_REGION_GOES_HERE"
});

const upload = multer({
  storage: multerS3({
    s3: s3,
    bucket: (request, file, cb) => {
      console.log(request.body.selectBucket);
      cb(null, request.body.selectBucket);
    },
    key: (request, file, cb) => {
      console.log(file);
      cb(null, file.originalname);
    }
  })
});

const app = express();

app.use(express.static("public"));

app.get("/", (request, response) => {
  response.sendFile(__dirname + "/public/index.html");
});

app.get("/buckets", (request, response) => {
  let listBuckets = async () => {
    try{
      const data = await s3.send(new ListBucketsCommand({}));
      console.log("Success fetching buckets: ", data.Buckets);
      response.send(data);
    }
    catch(e){
      console.log("There was an error when trying to fetch object store buckets: ", e);
    }
  };
  listBuckets();
});

app.post("/upload", upload.array("uploadFilesInput"), (request, response) => {
  console.log("File(s) uploaded successfully.");
  response.send(request.files.length + " file(s) successfully uploaded.");
});

app.listen(8080, () => {
  console.log("Server listening on port 8080.");
});

Step 5g - Testing server.js

Navigate to the main project folder s3-object-storage-nodejs. Start server.js using npm start.

sudo npm start

The application will output its listen method message to the console.

Server is listening on port 8080.

Use CTRL-C to terminate the server.js application.

Step 6 - Creating the Client-Side of the Application

In this step, you will create the client-side of the application which consists of an HTML form for uploading files, a CSS file for styling, and a client-side Javascript file to programmatically manipulate the HTML form based on server responses and user selections.

Step 6a - Create public Sub-Folder

Navigate to the main project folder s3-object-storage-nodejs and create a public sub-folder.

mkdir public

As mentioned in Step 5d, the client-side application files will be stored in and served from the public sub-folder.

Step 6b - Create index.html HTML File

Create an empty HTML file within the public sub-folder and name it index.html. Copy and paste the following HTML code into the file.

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>S3 Object Storage Upload with Node.js</title>
    <link rel="stylesheet" href="./styles.css">
  </head>
  <body>
      <h2>S3 Object Storage Upload with Node.js</h2>
      <form id="uploadForm" method="post" enctype="multipart/form-data">
        <div class="form-hint">Please select a bucket using the dropdown menu.</div>
        <select name="selectBucket" id="selectBucket">
          <option>Select a Bucket</option>
        </select>
        <div class="form-hint">Please select one or more files to upload.</div>
        <label class="upload">
          <input type="file" multiple name="uploadFilesInput" id="uploadFilesInput"/>
          <span>Click Here to Select Files</span>
        </label>
        <div id="filesListCount" class="form-hint" for="filesList"></div>
        <ul id="filesList"></ul>
        <label id="submitButton" class="submit">
          <input type="submit" value="submit">
          <span>Submit</span>
        </label>
      </form>
      <div id="uploadMessage" class="message"></div>
      <label id="resetForm" class="reset">
        <span>Click Here to Reset the Form</span>
      </label>
      <script src='client.js'></script>
  </body>
</html>

The HTML code above displays an HTML form for uploading files. The <select id="selectBucket"> and <ul id="filesList"> elements are dynamically populated to display available S3 storage service buckets and the files chosen for upload respectively.

The <label id="resetForm"> element is initially hidden and then displayed upon a successful upload response from the server. A client-side Javascript file, client.js, will be created to manage the dynamic components of the HTML form.

Step 6c - Create client.js Javascript File

Create an empty Javascript file within the public sub-folder and name it client.js. Copy and paste the following Javascript code into the file.

(() => {
    "use strict";

    const formElem = document.getElementById("uploadForm");
    const selectBucketsElem = document.getElementById("selectBucket");
    const filesListElem = document.getElementById("filesList");
    const filesListLabelElem = document.getElementById("filesListCount");
    const messageElem = document.getElementById("uploadMessage");
    const resetLabelElem = document.getElementById("resetForm");

    document.getElementById("uploadFilesInput").addEventListener("change", (e) => {filesToUploadHandler(e)});
    document.getElementById("submitButton").addEventListener("click", (e) => {submitFormHandler(e)});
    document.getElementById("resetForm").addEventListener("click", (e) => {resetFormHandler(e)});

    let filesToUploadHandler = (e) => {
        let numOfFiles = e.target.files.length;

        filesListLabelElem.innerText = "You have selected " + numOfFiles + " file(s) for uploading.";

        for(let fileKey in e.target.files){
            if((undefined !== e.target.files[fileKey].name) && (Number.isInteger(parseInt(fileKey)))){
                filesListElem.innerHTML += "<li>" + e.target.files[fileKey].name + "</li>";
            }
        };
    }

    let submitFormHandler = (e) => {
        e.preventDefault();
        messageElem.innerText = "Uploading...";
        fetch("http://localhost:8080/upload", {
            method: "POST",
            body: new FormData(uploadForm)
          })
          .then(response => response.text())
          .then(text => {
            messageElem.innerText = text;
            resetLabelElem.style.display = "block";
          })
          .catch(e => {
            messageElem.innerText = "An error occurred during upload; check your network connection.";
            resetLabelElem.style.display = "block";
          })
   }

    let resetFormHandler = (e) => {
        resetLabelElem.style.display = "none";
        filesListLabelElem.innerText = "";
        filesListElem.innerHTML = "";
        messageElem.innerText = "";
        formElem.reset();
    }

    let fetchBuckets = () => {
        fetch("http://localhost:8080/buckets", {
            method: "GET"
        })
        .then(response => response.json())
        .then(json => {
            let buckets = json.Buckets;
        
            buckets.forEach(bucket => {
                selectBucketsElem.innerHTML += "<option value='" + bucket["Name"] + "'>" + bucket["Name"] + "</option>";
            })
        })
        .catch((e) => console.log("There was an error when trying to fetch buckets: ", e));
    }

    fetchBuckets();

})()

The fetchBuckets method is executed immediately to fetch your available S3 storage service buckets and dynamically populates the <select id="selectBucket"> element of the HTML form with an <option> for each bucket. The filesToUpload method, and its associated event handler, are triggered when files are selected for uploading.

Each filename is dynamically added to the <ul id="filesList"> element of the HTML form with a <li> list element corresponding to each file. The submitFormHandler method is triggered when the Submit button is clicked in the HTML form and submits the upload form data to the server via a POST request. The submitFormHandler method also receives the server response and outputs the response to the client browser.

Step 6d - Create styles.css CSS File

Create an empty CSS file within the public sub-folder and name it styles.css. Copy and paste the following CSS styles into the file.

:root{
  --dark-grey: #333333;
  --fancy-blue: #0066FF;
  --fancy-blue-light: #4D94FF;
  --white: #FFFFFF;
  --aqua-light: #05BCB6;
  --aqua: #04A29D;
}

html{
    font-family: 'Segoe UI', Tahoma, Geneva, Verdana, sans-serif;
    font-size: 14px;
    box-sizing: border-box;
}

*, *:before, *:after {
   box-sizing: inherit;
}

body {
   margin: 1rem;
}

.form-hint{
   margin: 1rem 0 0.5rem 0;
}

select[name="selectBucket"], label{
   display: block;
   width: 25rem;
   height: 3rem;
   border-radius: 0.25rem;
   padding: 0.5rem 0;
   cursor: pointer;
 }

 label{
   line-height: 1.75;
   text-align: center;
   transition: all ease 0.25s;
 }

 input[name="uploadFilesInput"]{
   display: none;
 }

 .upload{
   background-color: var(--white);
   border: 1px solid var(--dark-grey);
   color: var(--dark-grey);
 }

 .upload:hover{
   background-color: var(--dark-grey);
   color: var(--white);
   font-weight: bolder;
 }

 input[type="submit"]{
   display: none;
 }

.submit{
   margin-top: 1rem;
   background-color: var(--fancy-blue-light);
   border: 1px solid var(--fancy-blue-light);
   color: var(--white);
 }

.submit:hover{
   background-color: var(--fancy-blue);
   border: 1px solid var(--fancy-blue);
   font-weight: bolder;
 }

 .message{
   margin-top: 1rem;
   color: var(--dark-grey);
 }

 .reset{
   display: none;
   margin-top: 1rem;
   background-color: var(--aqua-light);
   border: 1px solid var(--aqua-light);
   color: var(--white);
 }

 .reset:hover{
   background-color: var(--aqua);
   border: 1px solid var(--aqua);
   font-weight: bolder;
 }

These CSS styles generally modify the default browser styling for HTML forms.

Step 7 - Running the Application and Uploading Files

• Navigate to the main project folder s3-object-storage-nodejs. Start server.js on the server using sudo npm start.
• Verify the server listening message Server is listening on port 8080. using the server console.
• Open a browser and navigate to http://localhost:8080. The HTML file upload form will be displayed.
• Select an S3 storage service bucket to upload to using the dropdown select menu.
• Click on Click Here to Select Files to open the file explorer. Select one or more files for uploading via the file explorer.
• Click on Submit to upload the selected files.
• The upload status of Uploading... will be displayed below the Submit button while files are uploaded. When a response is received from the server, it will replace the upload status message, e.g. 2 file(s) were successfully uploaded..
• Click on Click Here to Reset the Form to reset the form and upload more files.

Conclusion

You created a Node.js server and web client application to programmatically upload files to an S3-compatible storage service. However, this guide only scratches the surface of what is possible with programmatic control over S3-compatible storage services. The server and client code examples above could be easily extended to provide additional functionality such as deleting and creating buckets. Please refer to the following resources for more information.

• AWS SDK for Javascript
• AWS SDK for Javascript for S3 Clients
• Multer storage engine
• Multer S3 client integration