Programmatic S3 Object Storage Integration With NodeJS

Introduction S3 compatible storage services are available from a number of cloud providers such as and . This guide demonstrates how the and the can be used to create a Node.js application to upload files to S3-compatible storage service buckets. The application is comprised of a Node.js server component and a web-based client component. Vultr Linode AWS SDK for JavaScript Multer storage engine Prerequisites To complete this guide, you will need to have: Set up your desired cloud infrastructure account for S3-compatible object storage. The and for your S3 compatible object storage instance. access key secret key Installed version 13.x or higher. Node.js Installed Node Package Manager, . npm The operating system commands in this guide are for Ubuntu. However, since the programming steps are identical between Ubuntu and Windows, Windows users can also complete this guide by substituting the analogous Windows operating system commands for the listed Ubuntu commands. Step 1 - Creating a New Node.js Project Folder Create a new folder for the new Node.js project and change it to the directory once created. This guide will use the folder name . s3-object-storage-nodejs sudo mkdir s3-object-storage-nodejs sudo cd s3-object-storage-nodejs Step 2 - Creating a Node.js File in the New Project Folder package.json The file contains basic data about the new Node.js project, such as the project name, as well as a list of project dependencies. package.json Step 2a - Run the Questionnaire Script package.json Ensure you are in the new project directory created in the previous step. Run the command to start a questionnaire script that will guide you through creating a new file. npm init package.json sudo npm init Step 2b - Enter Project Values package.json When the questionnaire script prompts you to enter: , use to select the default value, which is the folder name set in Step 1. package name Enter s3-object-storage-nodejs , use to select the default value . version Enter 1.0.0 , use to leave the description value empty. description Enter , type , and then . entry point server.js Enter , use to leave the test command value empty. test command Enter , use to leave the git repository value empty. git repository Enter , use to leave the value of the keyword empty. keywords Enter , type any value (e.g. ) and then . author S3 User Enter , use to select the default value . license Enter ISC The questionnaire script will output a summary of the chosen values. package.json { "name": "s3-object-storage-nodejs", "version": "1.0.0", "description": "", "main": "server.js", "scripts": { "test": "echo \"Error: no test specified\" && exit 1" }, "author": "S3 User", "license": "ISC" } When prompted type and then to save the new file. Is this OK? yes Enter package.json Step 2c - Add Start Key to packakge.json Use a text editor to add above the key in . The updated file will have the following structure: "start": "node server.js", test package.json package.json { "name": "s3-object-storage-nodejs", "version": "1.0.0", "description": "", "main": "server.js", "scripts": { "start": "node server.js", "test": "echo \"Error: no test specified\" && exit 1" }, "author": "S3 User", "license": "ISC" } Step 3 - Installing Node.js Project Dependencies Navigate to the main project folder . s3-object-storage-nodejs Step 3a - Install Express.js provides the framework for the server component of the application. Express.js npm install express Step 3b - Install AWS SDK for Javascript for S3 Clients The provides modular access to AWS resources, including S3-compatible storage services. AWS SDK for Javascript npm install @aws-sdk/client-s3 Step 3c - Install Multer middleware for Node.js is used for handling , which is typically used for uploading files. Multer multipart/form-data npm install multer Step 3d - Install Multer S3 provides streaming-based S3 client integration with the Multer storage engine. Multer S3 npm install multer-s3 Step 4 - Setting S3 Object Storage Credentials Step 4a - Create Folder and Empty File .aws credentials Create a new folder named with an empty text file in your , the project folder created in Step 1. .aws credentials user directory not The path to the empty credentials file will be located at on Ubuntu and on Windows. The AWS SDK for Javascript follows a default credentials provider chain and will look for your S3-compatible object storage credentials in either of the preceding directories depending on your operating system. ~\.aws\credentials C:\Users\USERNAME\.aws\credentials sudo mkdir .aws sudo touch .aws\credentials Step 4b - Edit File to Set S3 Object Storage Credentials credentials Open the empty file with a text editor and enter the profile name followed by your S3-compatible object storage credentials. credentials [default] [default] aws_access_key_id=YOUR_S3_OBJECT_STORAGE_ACCESS_KEY_GOES_HERE aws_secret_access_key=YOUR_S3_OBJECT_STORAGE_SECRET_KEY_GOES_HERE Do not enclose the or the with quotes or any other character. Also, the file should be saved a file extension. access key secret key credentials without Step 5 - Creating the Server-Side of the Application In this step, you will create the server-side component of the application in your desired development environment (e.g. Visual Studio Code). Step 5a - Create File and Specify Required Project Modules server.js Create a new Javascript file in the project folder set in Step 1 named . Add the following statements at the beginning of the file: server.js require const { S3, ListBucketsCommand } = require("@aws-sdk/client-s3"); const multer = require("multer"); const multerS3 = require("multer-s3"); const express = require("express"); Step 5b - Initialize a New Instance of the AWS SDK S3 Class Initialization of the AWS SDK S3 class requires that you specify the and the of the S3 service. Add the following code to the file using the specified and values for your S3-compatible object storage service. For example, the and values used with Vultr Object Storage are and respectively. endpoint region server.js endpoint region endpoint region https://ewr1.vultrobjects.com ewr1 // Enter the appropriate endpoint and region values for your service provider. const s3 = new S3({ endpoint: "YOUR_S3_SERVICE_ENDPOINT_URL_GOES_HERE", region: "YOUR_S3_SERVICE_REGION_GOES_HERE" }); Step 5c - Define Method upload Add the method which will call the Multer storage engine using client-side file upload form data. upload const upload = multer({ storage: multerS3({ s3: s3, bucket: (request, file, cb) => { console.log(request.body.selectBucket); cb(null, request.body.selectBucket); }, key: (request, file, cb) => { console.log(file); cb(null, file.originalname); } }) }); Step 5d - Add a New Express Application and Routes Use the method to create a new Express application instance. express() const app = express(); When the client-side (i.e. browser-side) of the application is created in Step 6, those static files will be placed in a sub-folder within the project folder. The middleware function is used to set the directory from which to serve these static assets. public express.static app.use(express.static("public")); The routes of the Express application need to be defined. In the following code, the first route sets the root of the application and serves the client-side file, which will display a file upload form. The second route retrieves a list of your S3 compatible storage service buckets and returns that data to the client for use in the file upload form. The third route handles requests from the browser, passing file upload form data to the method defined earlier, and sending a response back to the client. index.html POST upload app.get("/", (request, response) => { response.sendFile(__dirname + "/public/index.html"); }); app.get("/buckets", (request, response) => { let listBuckets = async () => { try{ const data = await s3.send(new ListBucketsCommand({})); console.log("Success fetching buckets:", data.Buckets); response.send(data); } catch(e){ console.log("There was an error when trying to fetch object store buckets: ", e); } }; listBuckets(); }); app.post("/upload", upload.array("uploadFilesInput"), (request, response) => { console.log("File(s) uploaded successfully."); response.send(request.files.length + " file(s) successfully uploaded."); }); Step 5e - Set the Express Application to Listen for Connections The last part of the server-side application sets the Express method so that can listen for client connections on port 8080. listen server.js app.listen(8080, () => { console.log("The server is listening on port 8080."); }); Step 5f - Complete File server.js This is the completed file after following steps 5a through 5e above. server.js const { S3, ListBucketsCommand } = require("@aws-sdk/client-s3"); const multer = require("multer"); const multerS3 = require("multer-s3"); const express = require("express"); const s3 = new S3({ endpoint: "YOUR_S3_SERVICE_ENDPOINT_URL_GOES_HERE", region: "YOUR_S3_SERVICE_REGION_GOES_HERE" }); const upload = multer({ storage: multerS3({ s3: s3, bucket: (request, file, cb) => { console.log(request.body.selectBucket); cb(null, request.body.selectBucket); }, key: (request, file, cb) => { console.log(file); cb(null, file.originalname); } }) }); const app = express(); app.use(express.static("public")); app.get("/", (request, response) => { response.sendFile(__dirname + "/public/index.html"); }); app.get("/buckets", (request, response) => { let listBuckets = async () => { try{ const data = await s3.send(new ListBucketsCommand({})); console.log("Success fetching buckets: ", data.Buckets); response.send(data); } catch(e){ console.log("There was an error when trying to fetch object store buckets: ", e); } }; listBuckets(); }); app.post("/upload", upload.array("uploadFilesInput"), (request, response) => { console.log("File(s) uploaded successfully."); response.send(request.files.length + " file(s) successfully uploaded."); }); app.listen(8080, () => { console.log("Server listening on port 8080."); }); Step 5g - Testing server.js Navigate to the main project folder . Start using . s3-object-storage-nodejs server.js npm start sudo npm start The application will output its method message to the console. listen Server is listening on port 8080. Use to terminate the application. CTRL-C server.js Step 6 - Creating the Client-Side of the Application In this step, you will create the client-side of the application which consists of an HTML form for uploading files, a CSS file for styling, and a client-side Javascript file to programmatically manipulate the HTML form based on server responses and user selections. Step 6a - Create Sub-Folder public Navigate to the main project folder and create a sub-folder. s3-object-storage-nodejs public mkdir public As mentioned in Step 5d, the client-side application files will be stored in and served from the sub-folder. public Step 6b - Create HTML File index.html Create an empty HTML file within the sub-folder and name it . Copy and paste the following HTML code into the file. public index.html S3 Object Storage Upload with Node.js S3 Object Storage Upload with Node.js Please select a bucket using the dropdown menu. Select a Bucket Please select one or more files to upload. Click Here to Select Files Submit Click Here to Reset the Form The HTML code above displays an HTML form for uploading files. The and elements are dynamically populated to display available S3 storage service buckets and the files chosen for upload respectively. The element is initially hidden and then displayed upon a successful upload response from the server. A client-side Javascript file, , will be created to manage the dynamic components of the HTML form. client.js Step 6c - Create Javascript File client.js Create an empty Javascript file within the sub-folder and name it . Copy and paste the following Javascript code into the file. public client.js (() => { "use strict"; const formElem = document.getElementById("uploadForm"); const selectBucketsElem = document.getElementById("selectBucket"); const filesListElem = document.getElementById("filesList"); const filesListLabelElem = document.getElementById("filesListCount"); const messageElem = document.getElementById("uploadMessage"); const resetLabelElem = document.getElementById("resetForm"); document.getElementById("uploadFilesInput").addEventListener("change", (e) => {filesToUploadHandler(e)}); document.getElementById("submitButton").addEventListener("click", (e) => {submitFormHandler(e)}); document.getElementById("resetForm").addEventListener("click", (e) => {resetFormHandler(e)}); let filesToUploadHandler = (e) => { let numOfFiles = e.target.files.length; filesListLabelElem.innerText = "You have selected " + numOfFiles + " file(s) for uploading."; for(let fileKey in e.target.files){ if((undefined !== e.target.files[fileKey].name) && (Number.isInteger(parseInt(fileKey)))){ filesListElem.innerHTML += " " + e.target.files[fileKey].name + " "; } }; } let submitFormHandler = (e) => { e.preventDefault(); messageElem.innerText = "Uploading..."; fetch("http://localhost:8080/upload", { method: "POST", body: new FormData(uploadForm) }) .then(response => response.text()) .then(text => { messageElem.innerText = text; resetLabelElem.style.display = "block"; }) .catch(e => { messageElem.innerText = "An error occurred during upload; check your network connection."; resetLabelElem.style.display = "block"; }) } let resetFormHandler = (e) => { resetLabelElem.style.display = "none"; filesListLabelElem.innerText = ""; filesListElem.innerHTML = ""; messageElem.innerText = ""; formElem.reset(); } let fetchBuckets = () => { fetch("http://localhost:8080/buckets", { method: "GET" }) .then(response => response.json()) .then(json => { let buckets = json.Buckets; buckets.forEach(bucket => { selectBucketsElem.innerHTML += " " + bucket["Name"] + " "; }) }) .catch((e) => console.log("There was an error when trying to fetch buckets: ", e)); } fetchBuckets(); })() The method is executed immediately to fetch your available S3 storage service buckets and dynamically populates the element of the HTML form with an for each bucket. The method, and its associated event handler, are triggered when files are selected for uploading. fetchBuckets filesToUpload Each filename is dynamically added to the element of the HTML form with a list element corresponding to each file. The method is triggered when the button is clicked in the HTML form and submits the upload form data to the server via a POST request. The method also receives the server response and outputs the response to the client browser. submitFormHandler Submit submitFormHandler Step 6d - Create CSS File styles.css Create an empty CSS file within the sub-folder and name it . Copy and paste the following CSS styles into the file. public styles.css :root{ --dark-grey: #333333; --fancy-blue: #0066FF; --fancy-blue-light: #4D94FF; --white: #FFFFFF; --aqua-light: #05BCB6; --aqua: #04A29D; } html{ font-family: 'Segoe UI', Tahoma, Geneva, Verdana, sans-serif; font-size: 14px; box-sizing: border-box; } *, *:before, *:after { box-sizing: inherit; } body { margin: 1rem; } .form-hint{ margin: 1rem 0 0.5rem 0; } select[name="selectBucket"], label{ display: block; width: 25rem; height: 3rem; border-radius: 0.25rem; padding: 0.5rem 0; cursor: pointer; } label{ line-height: 1.75; text-align: center; transition: all ease 0.25s; } input[name="uploadFilesInput"]{ display: none; } .upload{ background-color: var(--white); border: 1px solid var(--dark-grey); color: var(--dark-grey); } .upload:hover{ background-color: var(--dark-grey); color: var(--white); font-weight: bolder; } input[type="submit"]{ display: none; } .submit{ margin-top: 1rem; background-color: var(--fancy-blue-light); border: 1px solid var(--fancy-blue-light); color: var(--white); } .submit:hover{ background-color: var(--fancy-blue); border: 1px solid var(--fancy-blue); font-weight: bolder; } .message{ margin-top: 1rem; color: var(--dark-grey); } .reset{ display: none; margin-top: 1rem; background-color: var(--aqua-light); border: 1px solid var(--aqua-light); color: var(--white); } .reset:hover{ background-color: var(--aqua); border: 1px solid var(--aqua); font-weight: bolder; } These CSS styles generally modify the default browser styling for HTML forms. Step 7 - Running the Application and Uploading Files Navigate to the main project folder . Start on the server using . s3-object-storage-nodejs server.js sudo npm start Verify the server listening message using the server console. Server is listening on port 8080. Open a browser and navigate to . The HTML file upload form will be displayed. http://localhost:8080 Select an S3 storage service bucket to upload to using the dropdown select menu. Click on to open the file explorer. Select one or more files for uploading via the file explorer. Click Here to Select Files Click on to upload the selected files. Submit The upload status of will be displayed below the button while files are uploaded. When a response is received from the server, it will replace the upload status message, e.g. . Uploading... Submit 2 file(s) were successfully uploaded. Click on to reset the form and upload more files. Click Here to Reset the Form Conclusion You created a Node.js server and web client application to programmatically upload files to an S3-compatible storage service. However, this guide only scratches the surface of what is possible with programmatic control over S3-compatible storage services. The server and client code examples above could be easily extended to provide additional functionality such as deleting and creating buckets. Please refer to the following resources for more information. AWS SDK for Javascript AWS SDK for Javascript for S3 Clients Multer storage engine Multer S3 client integration