File uploads are a common feature in many web applications, but handling them efficiently and securely can be challenging. In this article, we will explore how to implement efficient file uploads using Node.js, Express.js, and MongoDB. We will cover the entire process, from setting up the server to storing files in MongoDB GridFS, ensuring that your application can handle large files without sacrificing performance. Problem Statement Handling file uploads efficiently involves several challenges: Scalability: Managing large files without overwhelming server memory.
Security: Preventing unauthorized access and ensuring data integrity.
Performance: Uploading and retrieving files quickly.
Storage Management: Efficiently storing and retrieving files from the database. Solution We will use the following stack to address these challenges: Node.js: For the server-side environment.
Express.js: As the web framework.
Multer: Middleware for handling multipart/form-data, which is used for uploading files.
MongoDB: For storing metadata and files.
GridFS: A specification for storing and retrieving large files in MongoDB. Step-by-Step Implementation 1. Setting Up the Project First, let’s create a new Node.js project and install the necessary dependencies. mkdir file-upload-app
cd file-upload-app
npm init -y
npm install express multer mongoose gridfs-stream 2. Creating the Server Next, we will set up a basic Express server. // server.js
const express = require('express');
const mongoose = require('mongoose');
const multer = require('multer');
const { GridFsStorage } = require('multer-gridfs-storage');
const Grid = require('gridfs-stream');
const methodOverride = require('method-override');
const bodyParser = require('body-parser');
const crypto = require('crypto');
const path = require('path');

const app = express();

// Middleware
app.use(bodyParser.json());
app.use(methodOverride('_method'));
app.set('view engine', 'ejs');

// Mongo URI
const mongoURI = 'mongodb://localhost:27017/fileupload';

// Create mongo connection
const conn = mongoose.createConnection(mongoURI, { useNewUrlParser: true, useUnifiedTopology: true });

// Init gfs
let gfs;
conn.once('open', () => {
  // Init stream
  gfs = Grid(conn.db, mongoose.mongo);
  gfs.collection('uploads');
});

// Create storage engine
const storage = new GridFsStorage({
  url: mongoURI,
  file: (req, file) => {
    return new Promise((resolve, reject) => {
      crypto.randomBytes(16, (err, buf) => {
        if (err) {
          return reject(err);
        }
        const filename = buf.toString('hex') + path.extname(file.originalname);
        const fileInfo = {
          filename: filename,
          bucketName: 'uploads'
        };
        resolve(fileInfo);
      });
    });
  }
});
const upload = multer({ storage });

// @route POST /upload
// @desc  Uploads file to DB
app.post('/upload', upload.single('file'), (req, res) => {
  res.json({ file: req.file });
});

// @route GET /files
// @desc  Display all files in JSON
app.get('/files', (req, res) => {
  gfs.files.find().toArray((err, files) => {
    // Check if files
    if (!files || files.length === 0) {
      return res.status(404).json({
        err: 'No files exist'
      });
    }

    // Files exist
    return res.json(files);
  });
});

// @route GET /files/:filename
// @desc  Display single file object
app.get('/files/:filename', (req, res) => {
  gfs.files.findOne({ filename: req.params.filename }, (err, file) => {
    if (!file || file.length === 0) {
      return res.status(404).json({
        err: 'No file exists'
      });
    }
    return res.json(file);
  });
});

// @route GET /image/:filename
// @desc Display Image
app.get('/image/:filename', (req, res) => {
  gfs.files.findOne({ filename: req.params.filename }, (err, file) => {
    if (!file || file.length === 0) {
      return res.status(404).json({
        err: 'No file exists'
      });
    }

    // Check if image
    if (file.contentType === 'image/jpeg' || file.contentType === 'image/png') {
      // Read output to browser
      const readstream = gfs.createReadStream(file.filename);
      readstream.pipe(res);
    } else {
      res.status(404).json({
        err: 'Not an image'
      });
    }
  });
});

// @route DELETE /files/:id
// @desc  Delete file
app.delete('/files/:id', (req, res) => {
  gfs.remove({ _id: req.params.id, root: 'uploads' }, (err, gridStore) => {
    if (err) {
      return res.status(404).json({ err: err });
    }
    res.redirect('/');
  });
});

const port = 5000;

app.listen(port, () => console.log(`Server started on port ${port}`)); 3. Testing the Upload Feature We can test our implementation using tools like Postman. Upload a File: Use a POST request to http://localhost:5000/upload with form data containing the file.
Retrieve Files: Use a GET request to http://localhost:5000/files to get the list of all files.
Retrieve a Single File: Use a GET request to http://localhost:5000/files/:filename to get a specific file’s details.
Display Image: Use a GET request to http://localhost:5000/image/:filename to view the image in the browser.
Delete a File: Use a DELETE request to http://localhost:5000/files/:id to delete a file. Conclusion In this article, I demonstrated how to handle file uploads efficiently using Node.js, Express.js, and MongoDB. By leveraging GridFS, we can store and manage large files seamlessly, ensuring scalability, security, and performance. This approach can be further customized and expanded based on specific application requirements, making it a versatile solution for file management in modern web applications. File uploads are a common feature in many web applications, but handling them efficiently and securely can be challenging. In this article, we will explore how to implement efficient file uploads using Node.js, Express.js, and MongoDB. We will cover the entire process, from setting up the server to storing files in MongoDB GridFS, ensuring that your application can handle large files without sacrificing performance. Problem Statement Handling file uploads efficiently involves several challenges: Scalability: Managing large files without overwhelming server memory. Security: Preventing unauthorized access and ensuring data integrity. Performance: Uploading and retrieving files quickly. Storage Management: Efficiently storing and retrieving files from the database. Scalability : Managing large files without overwhelming server memory. Scalability Security : Preventing unauthorized access and ensuring data integrity. Security Performance : Uploading and retrieving files quickly. Performance Storage Management : Efficiently storing and retrieving files from the database. Storage Management Solution We will use the following stack to address these challenges: Node.js: For the server-side environment. Express.js: As the web framework. Multer: Middleware for handling multipart/form-data, which is used for uploading files. MongoDB: For storing metadata and files. GridFS: A specification for storing and retrieving large files in MongoDB. Node.js : For the server-side environment. Node.js Express.js : As the web framework. Express.js Multer : Middleware for handling multipart/form-data , which is used for uploading files. Multer multipart/form-data MongoDB : For storing metadata and files. MongoDB GridFS : A specification for storing and retrieving large files in MongoDB. GridFS Step-by-Step Implementation 1. Setting Up the Project First, let’s create a new Node.js project and install the necessary dependencies. mkdir file-upload-app
cd file-upload-app
npm init -y
npm install express multer mongoose gridfs-stream mkdir file-upload-app
cd file-upload-app
npm init -y
npm install express multer mongoose gridfs-stream 2. Creating the Server Next, we will set up a basic Express server. // server.js
const express = require('express');
const mongoose = require('mongoose');
const multer = require('multer');
const { GridFsStorage } = require('multer-gridfs-storage');
const Grid = require('gridfs-stream');
const methodOverride = require('method-override');
const bodyParser = require('body-parser');
const crypto = require('crypto');
const path = require('path');

const app = express();

// Middleware
app.use(bodyParser.json());
app.use(methodOverride('_method'));
app.set('view engine', 'ejs');

// Mongo URI
const mongoURI = 'mongodb://localhost:27017/fileupload';

// Create mongo connection
const conn = mongoose.createConnection(mongoURI, { useNewUrlParser: true, useUnifiedTopology: true });

// Init gfs
let gfs;
conn.once('open', () => {
  // Init stream
  gfs = Grid(conn.db, mongoose.mongo);
  gfs.collection('uploads');
});

// Create storage engine
const storage = new GridFsStorage({
  url: mongoURI,
  file: (req, file) => {
    return new Promise((resolve, reject) => {
      crypto.randomBytes(16, (err, buf) => {
        if (err) {
          return reject(err);
        }
        const filename = buf.toString('hex') + path.extname(file.originalname);
        const fileInfo = {
          filename: filename,
          bucketName: 'uploads'
        };
        resolve(fileInfo);
      });
    });
  }
});
const upload = multer({ storage });

// @route POST /upload
// @desc  Uploads file to DB
app.post('/upload', upload.single('file'), (req, res) => {
  res.json({ file: req.file });
});

// @route GET /files
// @desc  Display all files in JSON
app.get('/files', (req, res) => {
  gfs.files.find().toArray((err, files) => {
    // Check if files
    if (!files || files.length === 0) {
      return res.status(404).json({
        err: 'No files exist'
      });
    }

    // Files exist
    return res.json(files);
  });
});

// @route GET /files/:filename
// @desc  Display single file object
app.get('/files/:filename', (req, res) => {
  gfs.files.findOne({ filename: req.params.filename }, (err, file) => {
    if (!file || file.length === 0) {
      return res.status(404).json({
        err: 'No file exists'
      });
    }
    return res.json(file);
  });
});

// @route GET /image/:filename
// @desc Display Image
app.get('/image/:filename', (req, res) => {
  gfs.files.findOne({ filename: req.params.filename }, (err, file) => {
    if (!file || file.length === 0) {
      return res.status(404).json({
        err: 'No file exists'
      });
    }

    // Check if image
    if (file.contentType === 'image/jpeg' || file.contentType === 'image/png') {
      // Read output to browser
      const readstream = gfs.createReadStream(file.filename);
      readstream.pipe(res);
    } else {
      res.status(404).json({
        err: 'Not an image'
      });
    }
  });
});

// @route DELETE /files/:id
// @desc  Delete file
app.delete('/files/:id', (req, res) => {
  gfs.remove({ _id: req.params.id, root: 'uploads' }, (err, gridStore) => {
    if (err) {
      return res.status(404).json({ err: err });
    }
    res.redirect('/');
  });
});

const port = 5000;

app.listen(port, () => console.log(`Server started on port ${port}`)); // server.js
const express = require('express');
const mongoose = require('mongoose');
const multer = require('multer');
const { GridFsStorage } = require('multer-gridfs-storage');
const Grid = require('gridfs-stream');
const methodOverride = require('method-override');
const bodyParser = require('body-parser');
const crypto = require('crypto');
const path = require('path');

const app = express();

// Middleware
app.use(bodyParser.json());
app.use(methodOverride('_method'));
app.set('view engine', 'ejs');

// Mongo URI
const mongoURI = 'mongodb://localhost:27017/fileupload';

// Create mongo connection
const conn = mongoose.createConnection(mongoURI, { useNewUrlParser: true, useUnifiedTopology: true });

// Init gfs
let gfs;
conn.once('open', () => {
  // Init stream
  gfs = Grid(conn.db, mongoose.mongo);
  gfs.collection('uploads');
});

// Create storage engine
const storage = new GridFsStorage({
  url: mongoURI,
  file: (req, file) => {
    return new Promise((resolve, reject) => {
      crypto.randomBytes(16, (err, buf) => {
        if (err) {
          return reject(err);
        }
        const filename = buf.toString('hex') + path.extname(file.originalname);
        const fileInfo = {
          filename: filename,
          bucketName: 'uploads'
        };
        resolve(fileInfo);
      });
    });
  }
});
const upload = multer({ storage });

// @route POST /upload
// @desc  Uploads file to DB
app.post('/upload', upload.single('file'), (req, res) => {
  res.json({ file: req.file });
});

// @route GET /files
// @desc  Display all files in JSON
app.get('/files', (req, res) => {
  gfs.files.find().toArray((err, files) => {
    // Check if files
    if (!files || files.length === 0) {
      return res.status(404).json({
        err: 'No files exist'
      });
    }

    // Files exist
    return res.json(files);
  });
});

// @route GET /files/:filename
// @desc  Display single file object
app.get('/files/:filename', (req, res) => {
  gfs.files.findOne({ filename: req.params.filename }, (err, file) => {
    if (!file || file.length === 0) {
      return res.status(404).json({
        err: 'No file exists'
      });
    }
    return res.json(file);
  });
});

// @route GET /image/:filename
// @desc Display Image
app.get('/image/:filename', (req, res) => {
  gfs.files.findOne({ filename: req.params.filename }, (err, file) => {
    if (!file || file.length === 0) {
      return res.status(404).json({
        err: 'No file exists'
      });
    }

    // Check if image
    if (file.contentType === 'image/jpeg' || file.contentType === 'image/png') {
      // Read output to browser
      const readstream = gfs.createReadStream(file.filename);
      readstream.pipe(res);
    } else {
      res.status(404).json({
        err: 'Not an image'
      });
    }
  });
});

// @route DELETE /files/:id
// @desc  Delete file
app.delete('/files/:id', (req, res) => {
  gfs.remove({ _id: req.params.id, root: 'uploads' }, (err, gridStore) => {
    if (err) {
      return res.status(404).json({ err: err });
    }
    res.redirect('/');
  });
});

const port = 5000;

app.listen(port, () => console.log(`Server started on port ${port}`)); 3. Testing the Upload Feature We can test our implementation using tools like Postman. Upload a File: Use a POST request to http://localhost:5000/upload with form data containing the file. Retrieve Files: Use a GET request to http://localhost:5000/files to get the list of all files. Retrieve a Single File: Use a GET request to http://localhost:5000/files/:filename to get a specific file’s details. Display Image: Use a GET request to http://localhost:5000/image/:filename to view the image in the browser. Delete a File: Use a DELETE request to http://localhost:5000/files/:id to delete a file. Upload a File : Use a POST request to http://localhost:5000/upload with form data containing the file. Upload a File http://localhost:5000/upload Retrieve Files : Use a GET request to http://localhost:5000/files to get the list of all files. Retrieve Files http://localhost:5000/files Retrieve a Single File : Use a GET request to http://localhost:5000/files/:filename to get a specific file’s details. Retrieve a Single File http://localhost:5000/files/:filename Display Image : Use a GET request to http://localhost:5000/image/:filename to view the image in the browser. Display Image http://localhost:5000/image/:filename Delete a File : Use a DELETE request to http://localhost:5000/files/:id to delete a file. Delete a File http://localhost:5000/files/:id Conclusion In this article, I demonstrated how to handle file uploads efficiently using Node.js, Express.js, and MongoDB. By leveraging GridFS, we can store and manage large files seamlessly, ensuring scalability, security, and performance. This approach can be further customized and expanded based on specific application requirements, making it a versatile solution for file management in modern web applications.

Walkthroughs, tutorials, guides, and tips. This story will teach you how to do something new or how to do something better.

Efficient File Uploads in Node.js: Using Express, MongoDB, and GridFS for Scalable Storage

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Why I Use RTK Query for API Calls in React

100 Days of AI Day 1: From Newsletter to Podcast, Leveraging AI for Audio Transformation

10 Threats to an Open API Ecosystem

10 Indications That You Should Invest in Automation Via APIs

10 Best Practices for Securing Your API

The Noonification: Getting Your API Into Production (10/28/2022)

Why I Use RTK Query for API Calls in React

100 Days of AI Day 1: From Newsletter to Podcast, Leveraging AI for Audio Transformation

10 Threats to an Open API Ecosystem

10 Indications That You Should Invest in Automation Via APIs

10 Best Practices for Securing Your API

The Noonification: Getting Your API Into Production (10/28/2022)

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps