Programmatic S3 Object Storage Integration With NodeJS

## Introduction S3 compatible storage services are available from a number of cloud providers such as [Vultr](https://www.vultr.com/en/docs/vultr-object-storage/?gspk=dHJveWNsYXJrNzM4Ng&gsxid=wQpnVuVRFBud) and [Linode](https://www.linode.com/products/object-storage/). This guide demonstrates how the [AWS SDK for JavaScript](https://aws.amazon.com/sdk-for-javascript/) and the [Multer storage engine](https://expressjs.com/en/resources/middleware/multer.html) can be used to create a Node.js application to upload files to S3-compatible storage service buckets. The application is comprised of a Node.js server component and a web-based client component. \ ## Prerequisites To complete this guide, you will need to have: \ * Set up your desired cloud infrastructure account for S3-compatible object storage. * The **access key** and **secret key** for your S3 compatible object storage instance. * Installed [Node.js](https://nodejs.org/en/download/) version 13.x or higher. * Installed Node Package Manager, `npm`. \ The operating system commands in this guide are for Ubuntu. However, since the programming steps are identical between Ubuntu and Windows, Windows users can also complete this guide by substituting the analogous Windows operating system commands for the listed Ubuntu commands. \ ## Step 1 - Creating a New Node.js Project Folder Create a new folder for the new Node.js project and change it to the directory once created. This guide will use the folder name `s3-object-storage-nodejs`. \ ```bash sudo mkdir s3-object-storage-nodejs sudo cd s3-object-storage-nodejs ``` \ ## Step 2 - Creating a Node.js `package.json` File in the New Project Folder The `package.json` file contains basic data about the new Node.js project, such as the project name, as well as a list of project dependencies. \ ### Step 2a - Run the `package.json` Questionnaire Script Ensure you are in the new project directory created in the previous step. Run the `npm init` command to start a questionnaire script that will guide you through creating a new `package.json` file. \ ```bash sudo npm init ``` \ ### Step 2b - Enter `package.json` Project Values When the questionnaire script prompts you to enter: * **package name**, use `Enter` to select the default value, which is the folder name `s3-object-storage-nodejs` set in Step 1. * **version**, use `Enter` to select the default value `1.0.0`. * **description**, use `Enter` to leave the description value empty. * **entry point**, type `server.js` , and then `Enter`. * **test command**, use `Enter` to leave the test command value empty. * **git repository**, use `Enter` to leave the git repository value empty. * **keywords**, use `Enter` to leave the value of the keyword empty. * **author**, type any value (e.g. `S3 User`) and then `Enter`. * **license**, use `Enter` to select the default value `ISC`. \ The questionnaire script will output a summary of the chosen `package.json` values. \ ```json { "name": "s3-object-storage-nodejs", "version": "1.0.0", "description": "", "main": "server.js", "scripts": { "test": "echo \"Error: no test specified\" && exit 1" }, "author": "S3 User", "license": "ISC" } ``` \ When prompted **Is this OK?** type `yes` and then `Enter` to save the new `package.json` file. \ ### Step 2c - Add Start Key to `packakge.json` Use a text editor to add `"start": "node server.js",` above the `test` key in `package.json`. The updated `package.json` file will have the following structure: \ ```json { "name": "s3-object-storage-nodejs", "version": "1.0.0", "description": "", "main": "server.js", "scripts": { "start": "node server.js", "test": "echo \"Error: no test specified\" && exit 1" }, "author": "S3 User", "license": "ISC" } ``` \ ## Step 3 - Installing Node.js Project Dependencies Navigate to the main project folder `s3-object-storage-nodejs`. \ ### Step 3a - Install Express.js [Express.js](http://expressjs.com/) provides the framework for the server component of the application. \ ```bash npm install express ``` \ ### Step 3b - Install AWS SDK for Javascript for S3 Clients The [AWS SDK for Javascript](https://aws.amazon.com/sdk-for-javascript/) provides modular access to AWS resources, including S3-compatible storage services. \ ```bash npm install @aws-sdk/client-s3 ``` \ ### Step 3c - Install Multer [Multer](https://www.npmjs.com/package/multer) middleware for Node.js is used for handling `multipart/form-data`, which is typically used for uploading files. \ ```bash npm install multer ``` \ ### Step 3d - Install Multer S3 [Multer S3](https://www.npmjs.com/package/multer-s3) provides streaming-based S3 client integration with the Multer storage engine. \ ```bash npm install multer-s3 ``` \ ## Step 4 - Setting S3 Object Storage Credentials ### Step 4a - Create `.aws` Folder and Empty `credentials` File Create a new folder named `.aws` with an empty text file `credentials` in your *user directory*, **not** the project folder created in Step 1. \ The path to the empty credentials file will be located at `~\.aws\credentials` on Ubuntu and `C:\Users\USERNAME\.aws\credentials` on Windows. The AWS SDK for Javascript follows a default credentials provider chain and will look for your S3-compatible object storage credentials in either of the preceding directories depending on your operating system. \ ```bash sudo mkdir .aws sudo touch .aws\credentials ``` \ ### Step 4b - Edit `credentials` File to Set S3 Object Storage Credentials Open the empty `credentials` file with a text editor and enter the profile name `[default]` followed by your S3-compatible object storage credentials. \ ``` [default] aws_access_key_id=YOUR_S3_OBJECT_STORAGE_ACCESS_KEY_GOES_HERE aws_secret_access_key=YOUR_S3_OBJECT_STORAGE_SECRET_KEY_GOES_HERE ``` \ Do not enclose the **access key** or the **secret key** with quotes or any other character. Also, the `credentials` file should be saved *without* a file extension. \ ## Step 5 - Creating the Server-Side of the Application In this step, you will create the server-side component of the application in your desired development environment (e.g. Visual Studio Code). \ ### Step 5a - Create `server.js` File and Specify Required Project Modules Create a new Javascript file in the project folder set in Step 1 named `server.js`. Add the following `require` statements at the beginning of the file: \ ```javascript const { S3, ListBucketsCommand } = require("@aws-sdk/client-s3"); const multer = require("multer"); const multerS3 = require("multer-s3"); const express = require("express"); ``` \ ### Step 5b - Initialize a New Instance of the AWS SDK S3 Class Initialization of the AWS SDK S3 class requires that you specify the `endpoint` and the `region` of the S3 service. Add the following code to the `server.js` file using the specified `endpoint` and `region` values for your S3-compatible object storage service. For example, the `endpoint` and `region` values used with Vultr Object Storage are `https://ewr1.vultrobjects.com` and `ewr1` respectively. \ ```javascript // Enter the appropriate endpoint and region values for your service provider. const s3 = new S3({ endpoint: "YOUR_S3_SERVICE_ENDPOINT_URL_GOES_HERE", region: "YOUR_S3_SERVICE_REGION_GOES_HERE" }); ``` \ ### Step 5c - Define `upload` Method Add the `upload` method which will call the Multer storage engine using client-side file upload form data. \ ```javascript const upload = multer({ storage: multerS3({ s3: s3, bucket: (request, file, cb) => { console.log(request.body.selectBucket); cb(null, request.body.selectBucket); }, key: (request, file, cb) => { console.log(file); cb(null, file.originalname); } }) }); ``` \ ### Step 5d - Add a New Express Application and Routes Use the `express()` method to create a new Express application instance. \ ``` const app = express(); ``` \ When the client-side (i.e. browser-side) of the application is created in Step 6, those static files will be placed in a `public` sub-folder within the project folder. The `express.static` middleware function is used to set the directory from which to serve these static assets. \ ```javascript app.use(express.static("public")); ``` \ The routes of the Express application need to be defined. In the following code, the first route sets the root of the application and serves the client-side `index.html` file, which will display a file upload form. The second route retrieves a list of your S3 compatible storage service buckets and returns that data to the client for use in the file upload form. The third route handles `POST` requests from the browser, passing file upload form data to the `upload` method defined earlier, and sending a response back to the client. \ ```javascript app.get("/", (request, response) => { response.sendFile(__dirname + "/public/index.html"); }); app.get("/buckets", (request, response) => { let listBuckets = async () => { try{ const data = await s3.send(new ListBucketsCommand({})); console.log("Success fetching buckets:", data.Buckets); response.send(data); } catch(e){ console.log("There was an error when trying to fetch object store buckets: ", e); } }; listBuckets(); }); app.post("/upload", upload.array("uploadFilesInput"), (request, response) => { console.log("File(s) uploaded successfully."); response.send(request.files.length + " file(s) successfully uploaded."); }); ``` \ ### Step 5e - Set the Express Application to Listen for Connections The last part of the server-side application sets the Express `listen` method so that `server.js` can listen for client connections on port 8080. \ ```javascript app.listen(8080, () => { console.log("The server is listening on port 8080."); }); ``` \ ### Step 5f - Complete `server.js` File This is the completed `server.js` file after following steps 5a through 5e above. \ ```javascript const { S3, ListBucketsCommand } = require("@aws-sdk/client-s3"); const multer = require("multer"); const multerS3 = require("multer-s3"); const express = require("express"); const s3 = new S3({ endpoint: "YOUR_S3_SERVICE_ENDPOINT_URL_GOES_HERE", region: "YOUR_S3_SERVICE_REGION_GOES_HERE" }); const upload = multer({ storage: multerS3({ s3: s3, bucket: (request, file, cb) => { console.log(request.body.selectBucket); cb(null, request.body.selectBucket); }, key: (request, file, cb) => { console.log(file); cb(null, file.originalname); } }) }); const app = express(); app.use(express.static("public")); app.get("/", (request, response) => { response.sendFile(__dirname + "/public/index.html"); }); app.get("/buckets", (request, response) => { let listBuckets = async () => { try{ const data = await s3.send(new ListBucketsCommand({})); console.log("Success fetching buckets: ", data.Buckets); response.send(data); } catch(e){ console.log("There was an error when trying to fetch object store buckets: ", e); } }; listBuckets(); }); app.post("/upload", upload.array("uploadFilesInput"), (request, response) => { console.log("File(s) uploaded successfully."); response.send(request.files.length + " file(s) successfully uploaded."); }); app.listen(8080, () => { console.log("Server listening on port 8080."); }); ``` \ ### Step 5g - Testing `server.js` Navigate to the main project folder `s3-object-storage-nodejs`. Start `server.js` using `npm start`. \ ```bash sudo npm start ``` \ The application will output its `listen` method message to the console. \ ```bash Server is listening on port 8080. ``` \ Use `CTRL-C` to terminate the `server.js` application. \ ## Step 6 - Creating the Client-Side of the Application In this step, you will create the client-side of the application which consists of an HTML form for uploading files, a CSS file for styling, and a client-side Javascript file to programmatically manipulate the HTML form based on server responses and user selections. \ ### Step 6a - Create `public` Sub-Folder Navigate to the main project folder `s3-object-storage-nodejs` and create a `public` sub-folder. \ ```bash mkdir public ``` \ As mentioned in Step 5d, the client-side application files will be stored in and served from the `public` sub-folder. \ ### Step 6b - Create `index.html` HTML File Create an empty HTML file within the `public` sub-folder and name it `index.html`. Copy and paste the following HTML code into the file. \ ```markup S3 Object Storage Upload with Node.js S3 Object Storage Upload with Node.js Please select a bucket using the dropdown menu. Select a Bucket Please select one or more files to upload. Click Here to Select Files Submit Click Here to Reset the Form ``` \ The HTML code above displays an HTML form for uploading files. The ` ` and ` ` elements are dynamically populated to display available S3 storage service buckets and the files chosen for upload respectively. \ The ` ` element is initially hidden and then displayed upon a successful upload response from the server. A client-side Javascript file, `client.js`, will be created to manage the dynamic components of the HTML form. \ ### Step 6c - Create `client.js` Javascript File Create an empty Javascript file within the `public` sub-folder and name it `client.js`. Copy and paste the following Javascript code into the file. \ ```javascript (() => { "use strict"; const formElem = document.getElementById("uploadForm"); const selectBucketsElem = document.getElementById("selectBucket"); const filesListElem = document.getElementById("filesList"); const filesListLabelElem = document.getElementById("filesListCount"); const messageElem = document.getElementById("uploadMessage"); const resetLabelElem = document.getElementById("resetForm"); document.getElementById("uploadFilesInput").addEventListener("change", (e) => {filesToUploadHandler(e)}); document.getElementById("submitButton").addEventListener("click", (e) => {submitFormHandler(e)}); document.getElementById("resetForm").addEventListener("click", (e) => {resetFormHandler(e)}); let filesToUploadHandler = (e) => { let numOfFiles = e.target.files.length; filesListLabelElem.innerText = "You have selected " + numOfFiles + " file(s) for uploading."; for(let fileKey in e.target.files){ if((undefined !== e.target.files[fileKey].name) && (Number.isInteger(parseInt(fileKey)))){ filesListElem.innerHTML += " " + e.target.files[fileKey].name + " "; } }; } let submitFormHandler = (e) => { e.preventDefault(); messageElem.innerText = "Uploading..."; fetch("http://localhost:8080/upload", { method: "POST", body: new FormData(uploadForm) }) .then(response => response.text()) .then(text => { messageElem.innerText = text; resetLabelElem.style.display = "block"; }) .catch(e => { messageElem.innerText = "An error occurred during upload; check your network connection."; resetLabelElem.style.display = "block"; }) } let resetFormHandler = (e) => { resetLabelElem.style.display = "none"; filesListLabelElem.innerText = ""; filesListElem.innerHTML = ""; messageElem.innerText = ""; formElem.reset(); } let fetchBuckets = () => { fetch("http://localhost:8080/buckets", { method: "GET" }) .then(response => response.json()) .then(json => { let buckets = json.Buckets; buckets.forEach(bucket => { selectBucketsElem.innerHTML += " " + bucket["Name"] + " "; }) }) .catch((e) => console.log("There was an error when trying to fetch buckets: ", e)); } fetchBuckets(); })() ``` \ The `fetchBuckets` method is executed immediately to fetch your available S3 storage service buckets and dynamically populates the ` ` element of the HTML form with an ` ` for each bucket. The `filesToUpload` method, and its associated event handler, are triggered when files are selected for uploading. \ Each filename is dynamically added to the ` ` element of the HTML form with a ` ` list element corresponding to each file. The `submitFormHandler` method is triggered when the **Submit** button is clicked in the HTML form and submits the upload form data to the server via a POST request. The `submitFormHandler` method also receives the server response and outputs the response to the client browser. \ ### Step 6d - Create `styles.css` CSS File Create an empty CSS file within the `public` sub-folder and name it `styles.css`. Copy and paste the following CSS styles into the file. \ ```css :root{ --dark-grey: #333333; --fancy-blue: #0066FF; --fancy-blue-light: #4D94FF; --white: #FFFFFF; --aqua-light: #05BCB6; --aqua: #04A29D; } html{ font-family: 'Segoe UI', Tahoma, Geneva, Verdana, sans-serif; font-size: 14px; box-sizing: border-box; } *, *:before, *:after { box-sizing: inherit; } body { margin: 1rem; } .form-hint{ margin: 1rem 0 0.5rem 0; } select[name="selectBucket"], label{ display: block; width: 25rem; height: 3rem; border-radius: 0.25rem; padding: 0.5rem 0; cursor: pointer; } label{ line-height: 1.75; text-align: center; transition: all ease 0.25s; } input[name="uploadFilesInput"]{ display: none; } .upload{ background-color: var(--white); border: 1px solid var(--dark-grey); color: var(--dark-grey); } .upload:hover{ background-color: var(--dark-grey); color: var(--white); font-weight: bolder; } input[type="submit"]{ display: none; } .submit{ margin-top: 1rem; background-color: var(--fancy-blue-light); border: 1px solid var(--fancy-blue-light); color: var(--white); } .submit:hover{ background-color: var(--fancy-blue); border: 1px solid var(--fancy-blue); font-weight: bolder; } .message{ margin-top: 1rem; color: var(--dark-grey); } .reset{ display: none; margin-top: 1rem; background-color: var(--aqua-light); border: 1px solid var(--aqua-light); color: var(--white); } .reset:hover{ background-color: var(--aqua); border: 1px solid var(--aqua); font-weight: bolder; } ``` \ These CSS styles generally modify the default browser styling for HTML forms. \ ## Step 7 - Running the Application and Uploading Files * Navigate to the main project folder `s3-object-storage-nodejs`. Start `server.js` on the server using `sudo npm start`. * Verify the server listening message `Server is listening on port 8080.` using the server console. * Open a browser and navigate to `http://localhost:8080`. The HTML file upload form will be displayed. * Select an S3 storage service bucket to upload to using the dropdown select menu. * Click on **Click Here to Select Files** to open the file explorer. Select one or more files for uploading via the file explorer. * Click on **Submit** to upload the selected files. * The upload status of `Uploading...` will be displayed below the **Submit** button while files are uploaded. When a response is received from the server, it will replace the upload status message, e.g. `2 file(s) were successfully uploaded.`. * Click on **Click Here to Reset the Form** to reset the form and upload more files. \ ## Conclusion You created a Node.js server and web client application to programmatically upload files to an S3-compatible storage service. However, this guide only scratches the surface of what is possible with programmatic control over S3-compatible storage services. The server and client code examples above could be easily extended to provide additional functionality such as deleting and creating buckets. Please refer to the following resources for more information. \ * [AWS SDK for Javascript](https://aws.amazon.com/sdk-for-javascript/) * [AWS SDK for Javascript for S3 Clients](https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/clients/client-s3/index.html) * [Multer storage engine](https://expressjs.com/en/resources/middleware/multer.html) * [Multer S3 client integration](https://www.npmjs.com/package/multer-s3) \

Programmatic S3 Object Storage Integration With NodeJS

Too Long; Didn't Read

Companies Mentioned

Coin Mentioned

@pictureinthenoise

Credibility

RELATED STORIES