Efficient File Uploads in Node.js: Using Express, MongoDB, and GridFS for Scalable Storage

File uploads are a common feature in many web applications, but handling them efficiently and securely can be challenging. In this article, we will explore how to implement efficient file uploads using Node.js, Express.js, and MongoDB. We will cover the entire process, from setting up the server to storing files in MongoDB GridFS, ensuring that your application can handle large files without sacrificing performance.

Problem Statement

Handling file uploads efficiently involves several challenges:

1. Scalability: Managing large files without overwhelming server memory.
2. Security: Preventing unauthorized access and ensuring data integrity.
3. Performance: Uploading and retrieving files quickly.
4. Storage Management: Efficiently storing and retrieving files from the database.

Solution

We will use the following stack to address these challenges:

• Node.js: For the server-side environment.
• Express.js: As the web framework.
• Multer: Middleware for handling multipart/form-data, which is used for uploading files.
• MongoDB: For storing metadata and files.
• GridFS: A specification for storing and retrieving large files in MongoDB.

Step-by-Step Implementation

1. Setting Up the Project

First, let’s create a new Node.js project and install the necessary dependencies.

mkdir file-upload-app
cd file-upload-app
npm init -y
npm install express multer mongoose gridfs-stream

2. Creating the Server

Next, we will set up a basic Express server.

// server.js
const express = require('express');
const mongoose = require('mongoose');
const multer = require('multer');
const { GridFsStorage } = require('multer-gridfs-storage');
const Grid = require('gridfs-stream');
const methodOverride = require('method-override');
const bodyParser = require('body-parser');
const crypto = require('crypto');
const path = require('path');

const app = express();

// Middleware
app.use(bodyParser.json());
app.use(methodOverride('_method'));
app.set('view engine', 'ejs');

// Mongo URI
const mongoURI = 'mongodb://localhost:27017/fileupload';

// Create mongo connection
const conn = mongoose.createConnection(mongoURI, { useNewUrlParser: true, useUnifiedTopology: true });

// Init gfs
let gfs;
conn.once('open', () => {
  // Init stream
  gfs = Grid(conn.db, mongoose.mongo);
  gfs.collection('uploads');
});

// Create storage engine
const storage = new GridFsStorage({
  url: mongoURI,
  file: (req, file) => {
    return new Promise((resolve, reject) => {
      crypto.randomBytes(16, (err, buf) => {
        if (err) {
          return reject(err);
        }
        const filename = buf.toString('hex') + path.extname(file.originalname);
        const fileInfo = {
          filename: filename,
          bucketName: 'uploads'
        };
        resolve(fileInfo);
      });
    });
  }
});
const upload = multer({ storage });

// @route POST /upload
// @desc  Uploads file to DB
app.post('/upload', upload.single('file'), (req, res) => {
  res.json({ file: req.file });
});

// @route GET /files
// @desc  Display all files in JSON
app.get('/files', (req, res) => {
  gfs.files.find().toArray((err, files) => {
    // Check if files
    if (!files || files.length === 0) {
      return res.status(404).json({
        err: 'No files exist'
      });
    }

    // Files exist
    return res.json(files);
  });
});

// @route GET /files/:filename
// @desc  Display single file object
app.get('/files/:filename', (req, res) => {
  gfs.files.findOne({ filename: req.params.filename }, (err, file) => {
    if (!file || file.length === 0) {
      return res.status(404).json({
        err: 'No file exists'
      });
    }
    return res.json(file);
  });
});

// @route GET /image/:filename
// @desc Display Image
app.get('/image/:filename', (req, res) => {
  gfs.files.findOne({ filename: req.params.filename }, (err, file) => {
    if (!file || file.length === 0) {
      return res.status(404).json({
        err: 'No file exists'
      });
    }

    // Check if image
    if (file.contentType === 'image/jpeg' || file.contentType === 'image/png') {
      // Read output to browser
      const readstream = gfs.createReadStream(file.filename);
      readstream.pipe(res);
    } else {
      res.status(404).json({
        err: 'Not an image'
      });
    }
  });
});

// @route DELETE /files/:id
// @desc  Delete file
app.delete('/files/:id', (req, res) => {
  gfs.remove({ _id: req.params.id, root: 'uploads' }, (err, gridStore) => {
    if (err) {
      return res.status(404).json({ err: err });
    }
    res.redirect('/');
  });
});

const port = 5000;

app.listen(port, () => console.log(`Server started on port ${port}`));

3. Testing the Upload Feature

We can test our implementation using tools like Postman.

1. Upload a File: Use a POST request to http://localhost:5000/upload with form data containing the file.
2. Retrieve Files: Use a GET request to http://localhost:5000/files to get the list of all files.
3. Retrieve a Single File: Use a GET request to http://localhost:5000/files/:filename to get a specific file’s details.
4. Display Image: Use a GET request to http://localhost:5000/image/:filename to view the image in the browser.
5. Delete a File: Use a DELETE request to http://localhost:5000/files/:id to delete a file.

Conclusion

In this article, I demonstrated how to handle file uploads efficiently using Node.js, Express.js, and MongoDB. By leveraging GridFS, we can store and manage large files seamlessly, ensuring scalability, security, and performance. This approach can be further customized and expanded based on specific application requirements, making it a versatile solution for file management in modern web applications.