We recently compiled a list of the Worst Cloud Stocks To Buy According to Short Sellers. In this article, we will look at where MongoDB, Inc. (NASDAQ:MDB) stands against the worst cloud stocks to buy according to short sellers. Cloud computing refers to the use of remote servers, typically ...

Is MongoDB, Inc. (MDB) the Worst Cloud Stock To Buy According to Short Sellers?

MongoDB occupies a unique niche in database market, offering document-oriented approach suited for handling vast amounts of data. Find out why MDB stock is a Buy.

MongoDB: This Growing Company Tends To Surprise The Market ...

MongoDB, Inc. is an American software company that develops and provides commercial support for the source-available database MongoDB, a NoSQL database that stores data in JSON-like documents with flexible schemas.

MongoDB Inc. - Wikipedia

The American software company 10gen began developing MongoDB in 2007 as a component of a planned platform-as-a-service product. In 2009, the company shifted to an open-source development model and began offering commercial support and other services. In 2013, 10gen changed its name to MongoDB Inc.

MongoDB - Wikipedia

Needham &amp; Company LLC reaffirmed a &quot;buy&quot; rating and set a $495.00 price objective on shares of MongoDB in a research report on Wednesday.

Needham & Company LLC Reaffirms "Buy" Rating for MongoDB (NASDAQ:MDB) ...

Anyone from MongoDB company here to refer me to a VP customer success role based in London or Germany ? ... Digite, Inc. I am Data Scientist with 5+ years of experience and I am from India. Currently I am working with the Equity research team of a global Bank. I am looking for remote opportunities from Europe, US, APAC and UK. Skills: Machine Learning, Problem formulation, Statical Analysis, NLP, Deep Learning Python, Pytorch, Pyspark, Keras, SQL, MongoDB ...

Working at MongoDB | Glassdoor

Fair Value is the appropriate price ... you might expect after purchasing shares in the company and holding them over the default time horizon of 5 years, based on the EPS growth rate that we have projected. ... MongoDB, Inc....

MongoDB, Inc. (MDB) Stock Price, News, Quote & History - Yahoo Finance

MongoDB | 703,916 followers on LinkedIn. A developer data platform for developers to do their best work. #LoveYourDevelopers | Headquartered in New York, MongoDB&#x27;s mission is to empower innovators to create, transform, and disrupt industries by unleashing the power of software and data.

MongoDB | LinkedIn

M&amp;A solutions provider Datasite leverages MongoDB Atlas to deliver a faster, more seamless experience to its users.Read customer story → · How Finnish delivery company uses MongoDB to scale and support new and unique customer requirements.Read customer story →

Our Customers | MongoDB

One of our core company values at MongoDB is “Embrace the Power of Differences” which means that we commit to creating a culture of inclusivity with employees from different backgrounds and experiences. We value a diverse workforce as a way to broaden our perspectives, foster innovation, ...

Life at MongoDB | MongoDB Blog

MongoDB

mongodb

We are back with another Company of the Week feature! Every week, we share an awesome tech brand from our tech company database, making their evergreen mark on the internet. This unique HackerNoon database ranks S&P 500 companies and top startups of the year alike. This week, we are proud to present MongoDB! You may have caught a mention of MongoDB during the pilot of the hit TV show “Silicon Valley,” but the NoSQL database product is not only popular with budding startups. In fact, you’d be amazed to learn that MongoDB powers some of the largest enterprises in the world, including prior HackerNoon company of the week Bosch which uses MongoDB Atlas when dealing with a vast amount of data in its workflows. Want to be featured on HackerNoon’s Company of the Week? Request Your Tech Company Page on HackerNoon! MongoDB <> HackerNoon Targeted Ads For all you programmers out there, you may have seen MongoDB appear across our website recently, especially if you were browsing any coding or programming-related tag on HackerNoon. Well, that’s because MongoDB and HackerNoon partnered together to highlight some amazing features from MongoDB to help you develop your own RAG or cloud document database for artificial intelligence. How’s that for cool!? In effect, you could use the same technology that is being utilized by the likes of Bosch, Novo Nordisk, Wells Fargo, and even Toyota, to create or innovate on your products! Learn How You Can Advertise to Your Specific Niche on HackerNoon Meet MongoDB: #FunFacts The founders of MongoDB — Dwight Merriman, Eliot Horowitz, and Kevin Ryan — previously worked with the online advertising company Doubleclick which was acquired by Google in the late 2000s for a whopping $3.1 billion. Using their knowledge of the limitations of relational databases, all three got together the same year Doubleclick was sold to found MongoDB whose mission is to empower innovators to create, transform, and disrupt industries by unleashing the power of software and data. Today, MongoDB’s Atlas is one of the most loved vector databases in the world, and the company, which is listed on Nasdaq, consistently ranks as one of the best places to work in. MongoDB is also a leader in cloud database management systems, as ranked by Gartner. How’s that for interesting 😏 Share Your Company's Story via HackerNoon That's all this week, folks! Stay Creative, Stay Iconic. The HackerNoon Team We are back with another Company of the Week feature! Every week, we share an awesome tech brand from our tech company database , making their evergreen mark on the internet. This unique HackerNoon database ranks S&P 500 companies and top startups of the year alike. Company of the Week Company of the Week tech company database tech company database This week, we are proud to present MongoDB ! You may have caught a mention of MongoDB during the pilot of the hit TV show “Silicon Valley,” but the NoSQL database product is not only popular with budding startups. In fact, you’d be amazed to learn that MongoDB powers some of the largest enterprises in the world, including prior HackerNoon company of the week Bosch which uses MongoDB Atlas when dealing with a vast amount of data in its workflows. MongoDB powers prior Bosch Want to be featured on HackerNoon’s Company of the Week? Request Your Tech Company Page on HackerNoon! Want to be featured on HackerNoon’s Company of the Week? Request Your Tech Company Page on HackerNoon ! Want to be featured on HackerNoon’s Company of the Week? Request Your Tech Company Page on HackerNoon MongoDB <> HackerNoon Targeted Ads For all you programmers out there, you may have seen MongoDB appear across our website recently, especially if you were browsing any coding or programming-related tag on HackerNoon. Well, that’s because MongoDB and HackerNoon partnered together to highlight some amazing features from MongoDB to help you develop your own RAG or cloud document database for artificial intelligence. How’s that for cool!? RAG cloud document database cool!? cool!? In effect, you could use the same technology that is being utilized by the likes of Bosch, Novo Nordisk, Wells Fargo, and even Toyota, to create or innovate on your products! Learn How You Can Advertise to Your Specific Niche on HackerNoon Learn How You Can Advertise to Your Specific Niche on HackerNoon Learn How You Can Advertise to Your Specific Niche on HackerNoon Learn How You Can Advertise to Your Specific Niche on HackerNoon Learn How You Can Advertise to Your Specific Niche on HackerNoon Meet MongoDB: #FunFacts The founders of MongoDB — Dwight Merriman, Eliot Horowitz, and Kevin Ryan — previously worked with the online advertising company Doubleclick which was acquired by Google in the late 2000s for a whopping $3.1 billion. Using their knowledge of the limitations of relational databases, all three got together the same year Doubleclick was sold to found MongoDB whose mission is to empower innovators to create, transform, and disrupt industries by unleashing the power of software and data. mission Today, MongoDB’s Atlas is one of the most loved vector databases in the world, and the company, which is listed on Nasdaq, consistently ranks as one of the best places to work in . MongoDB is also a leader in cloud database management systems, as ranked by Gartner. most loved vector databases best places to work in leader How’s that for interesting 😏 Share Your Company's Story via HackerNoon Share Your Company's Story via HackerNoon Share Your Company's Story via HackerNoon Share Your Company's Story via HackerNoon That's all this week, folks! Stay Creative, Stay Iconic. The HackerNoon Team

Meet MongoDB: HackerNoon Company of the Week

Ontmoet MongoDB: HackerNoon Company of the Week

Знаёмцеся, MongoDB: Кампанія тыдня HackerNoon

Conozca MongoDB: la empresa de la semana de HackerNoon

MongoDB से मिलिए: HackerNoon कंपनी ऑफ द वीक

Rankontre MongoDB: HackerNoon Konpayi Semèn nan

Ismerje meg a MongoDB-t: A hét HackerNoon cége

MongoDB のご紹介: HackerNoon の今週の企業

გაიცანით MongoDB: კვირის HackerNoon კომპანია

ជួបជាមួយ MongoDB: ក្រុមហ៊ុន HackerNoon ប្រចាំសប្តាហ៍

Susipažinkite su MongoDB: „HackerNoon“ savaitės kompanija

Hihaona amin'ny MongoDB: HackerNoon Company of the Week

Kopana le MongoDB: Khamphani ya HackerNoon ya Beke

Hello JavaScript Enthusiasts!Welcome to this week’s edition of "This Week in JavaScript"!This week, we have exciting updates that will make your development workflow smoother and faster. From VoidZero’s unified JavaScript toolchain to ESLint’s new features, and even MongoDB’s performance boost, there's a lot to unpack! VoidZero Inc. Raises $4.6M to Unify JavaScript Tooling VoidZero has raised $4.6M in seed funding, led by Accel, to develop a unified, high-performance toolchain for the JavaScript ecosystem. Why is this Important? JavaScript tooling is currently fragmented, with developers relying on various third-party dependencies. VoidZero aims to streamline the development process by creating a single toolchain that handles everything from bundling to testing. Key Highlights: Unified Framework: Same AST and resolver for all tasks, reducing redundant parsing.
Performance-Focused: Built in a compile-to-native language with support for parallelization and optimized JS plugins.
Modular Components: Easily customizable building blocks for advanced use cases.
Cross-Environment Support: Works across all JavaScript environments without runtime limitations. VoidZero’s new bundler, Rolldown, will be in alpha later this year and set to integrate with Vite soon. ESLint now Lints JSON and Markdown JSON and Markdown files can now be linted directly! This is part of ESLint’s push to become a more general-purpose linter, expanding beyond JavaScript. What’s New? JSON Support: With the new @eslint/json plugin, you can lint JSON, JSONC, and JSON5 files. To get started, just install the plugin and update your ESLint config to include the JSON rules. Markdown Support: The @eslint/markdown plugin introduces linting for CommonMark and GitHub-Flavored Markdown. Just install and configure it in your project to start catching issues in your markdown files. This update opens up more possibilities for linting different file types in web projects, bringing everything under the same ESLint umbrella. Tauri 2.0 is Here Tauri 2.0’s stable release is now live, bringing key improvements and mobile support for iOS and Android. This Rust-based framework lets you build lightweight, cross-platform desktop and mobile applications simply by using HTML, JavaScript, and CSS. It does all this and makes the file size smaller and makes performance faster. With Tauri 2.0, you get: Unified Codebase: One UI codebase for desktop (Windows, macOS, Linux) and mobile (iOS, Android).
Hot-Module Replacement (HMR): Seamlessly update your app's frontend without rebuilding.
Mobile Support: Use Swift or Kotlin to build mobile-specific functionalities while sharing Rust logic.
Advanced Plugins: Extend your app's capabilities with a more refined plugin system, simplifying development. Tauri continues to evolve, offering developers a highly efficient and flexible way to build for all platforms. New Tools and Releases Tabulator 6.3: Simplify managing complex data tables in web apps with powerful editing, filtering, and sorting features.
MongoDB 8.0: Enjoy a 30% performance boost and enhanced security with Queryable Encryption and better scaling.
Eleventy 3.0: The latest release of this static site generator makes creating fast, efficient websites easier than ever.
µExpress: Speed up your Node.js apps with this optimized version of Express, offering up to 4x faster requests. That’s a wrap for this week! If you found these JavaScript updates useful, give this post an upvote to help others discover it. Don’t forget to share it with your fellow developers and keep an eye out for more insights in the JavaScript world. Until next time, keep coding and stay curious! References VoidZero Raises $4.6M ESLint Now Lints JSON and Markdown Tauri 2.0 Release Tabulator 6.3 MongoDB 8.0Eleventy 3.0 µExpress Hello JavaScript Enthusiasts! Welcome to this week’s edition of " This Week in JavaScript "! This week, we have exciting updates that will make your development workflow smoother and faster. From VoidZero’s unified JavaScript toolchain to ESLint’s new features, and even MongoDB’s performance boost, there's a lot to unpack! This Week in JavaScript VoidZero Inc. Raises $4.6M to Unify JavaScript Tooling VoidZero Inc. Raises $4.6M to Unify JavaScript Tooling VoidZero has raised $4.6M in seed funding, led by Accel, to develop a unified, high-performance toolchain for the JavaScript ecosystem. VoidZero Why is this Important? Why is this Important? JavaScript tooling is currently fragmented, with developers relying on various third-party dependencies. VoidZero aims to streamline the development process by creating a single toolchain that handles everything from bundling to testing. Key Highlights: Key Highlights: Unified Framework: Same AST and resolver for all tasks, reducing redundant parsing. Performance-Focused: Built in a compile-to-native language with support for parallelization and optimized JS plugins. Modular Components: Easily customizable building blocks for advanced use cases. Cross-Environment Support: Works across all JavaScript environments without runtime limitations. Unified Framework: Same AST and resolver for all tasks, reducing redundant parsing. Unified Framework: Performance-Focused: Built in a compile-to-native language with support for parallelization and optimized JS plugins. Performance-Focused: Modular Components: Easily customizable building blocks for advanced use cases. Modular Components: Cross-Environment Support: Works across all JavaScript environments without runtime limitations. Cross-Environment Support: VoidZero’s new bundler, Rolldown , will be in alpha later this year and set to integrate with Vite soon. Rolldown ESLint now Lints JSON and Markdown ESLint now Lints JSON and Markdown JSON and Markdown files can now be linted directly! This is part of ESLint ’s push to become a more general-purpose linter, expanding beyond JavaScript. ESLint What’s New? JSON Support: With the new @eslint/json plugin, you can lint JSON, JSONC, and JSON5 files. To get started, just install the plugin and update your ESLint config to include the JSON rules. JSON Support: With the new @eslint/json plugin, you can lint JSON, JSONC, and JSON5 files. To get started, just install the plugin and update your ESLint config to include the JSON rules. JSON Support: Markdown Support: The @eslint/markdown plugin introduces linting for CommonMark and GitHub-Flavored Markdown. Just install and configure it in your project to start catching issues in your markdown files. Markdown Support: The @eslint/markdown plugin introduces linting for CommonMark and GitHub-Flavored Markdown. Just install and configure it in your project to start catching issues in your markdown files. Markdown Support: This update opens up more possibilities for linting different file types in web projects, bringing everything under the same ESLint umbrella. ESLint Tauri 2.0 is Here Tauri 2.0 is Here Tauri 2.0’s stable release is now live, bringing key improvements and mobile support for iOS and Android. This Rust-based framework lets you build lightweight, cross-platform desktop and mobile applications simply by using HTML, JavaScript, and CSS. It does all this and makes the file size smaller and makes performance faster. Tauri 2.0’s With Tauri 2.0, you get: With Tauri 2.0, you get: Unified Codebase: One UI codebase for desktop (Windows, macOS, Linux) and mobile (iOS, Android). Hot-Module Replacement (HMR): Seamlessly update your app's frontend without rebuilding. Mobile Support: Use Swift or Kotlin to build mobile-specific functionalities while sharing Rust logic. Advanced Plugins: Extend your app's capabilities with a more refined plugin system, simplifying development. Unified Codebase: One UI codebase for desktop (Windows, macOS, Linux) and mobile (iOS, Android). Unified Codebase: Hot-Module Replacement (HMR): Seamlessly update your app's frontend without rebuilding. Hot-Module Replacement (HMR): Mobile Support: Use Swift or Kotlin to build mobile-specific functionalities while sharing Rust logic. Mobile Support: Advanced Plugins: Extend your app's capabilities with a more refined plugin system, simplifying development. Advanced Plugins: Tauri continues to evolve, offering developers a highly efficient and flexible way to build for all platforms. Tauri New Tools and Releases New Tools and Releases Tabulator 6.3: Simplify managing complex data tables in web apps with powerful editing, filtering, and sorting features. MongoDB 8.0: Enjoy a 30% performance boost and enhanced security with Queryable Encryption and better scaling. Eleventy 3.0: The latest release of this static site generator makes creating fast, efficient websites easier than ever. µExpress: Speed up your Node.js apps with this optimized version of Express, offering up to 4x faster requests. Tabulator 6.3: Simplify managing complex data tables in web apps with powerful editing, filtering, and sorting features. Tabulator 6.3: MongoDB 8.0: Enjoy a 30% performance boost and enhanced security with Queryable Encryption and better scaling. MongoDB 8.0: Eleventy 3.0: The latest release of this static site generator makes creating fast, efficient websites easier than ever. Eleventy 3.0: µExpress: Speed up your Node.js apps with this optimized version of Express, offering up to 4x faster requests. µExpress: That’s a wrap for this week! If you found these JavaScript updates useful, give this post an upvote to help others discover it. Don’t forget to share it with your fellow developers and keep an eye out for more insights in the JavaScript world. Until next time, keep coding and stay curious! References VoidZero Raises $4.6M VoidZero Raises $4.6M ESLint Now Lints JSON and Markdown ESLint Now Lints JSON and Markdown Tauri 2.0 Release Tauri 2.0 Release Tabulator 6.3 Tabulator 6.3 MongoDB 8.0 Eleventy 3.0 MongoDB 8.0 Eleventy 3.0 µExpress µExpress

VoidZero, The New ESLint, MongoDB 8.0, and More - This Week in JS

VoidZero, The New ESLint, MongoDB 8.0 a další – tento týden v JS

VoidZero, The New ESLint, MongoDB 8.0 og mere - denne uge i JS

VoidZero, das neue ESLint, MongoDB 8.0 und mehr – Diese Woche in JS

VoidZero, el nuevo ESLint, MongoDB 8.0 y más: esta semana en JS

VoidZero、新しい ESLint、MongoDB 8.0 など - 今週の JS

VoidZero, The New ESLint, MongoDB 8.0, ແລະອື່ນໆອີກ - ໃນອາທິດນີ້ໃນ JS

„VoidZero“, „The New ESLint“, „MongoDB 8.0“ ir dar daugiau – šią savaitę JS

VoidZero, The New ESLint, MongoDB 8.0 болон бусад зүйлс - Энэ долоо хоногт JS дээр

VoidZero, ESLint Baharu, MongoDB 8.0 dan Banyak Lagi - Minggu Ini dalam JS

VoidZero, ESLint i ri, MongoDB 8.0 dhe më shumë - Këtë javë në JS

VoidZero, The New ESLint, MongoDB 8.0 va boshqalar - bu hafta JSda

VoidZero、新 ESLint、MongoDB 8.0 等 - 本周 JS 资讯

File uploads are a common feature in many web applications, but handling them efficiently and securely can be challenging. In this article, we will explore how to implement efficient file uploads using Node.js, Express.js, and MongoDB. We will cover the entire process, from setting up the server to storing files in MongoDB GridFS, ensuring that your application can handle large files without sacrificing performance. Problem Statement Handling file uploads efficiently involves several challenges: Scalability: Managing large files without overwhelming server memory.
Security: Preventing unauthorized access and ensuring data integrity.
Performance: Uploading and retrieving files quickly.
Storage Management: Efficiently storing and retrieving files from the database. Solution We will use the following stack to address these challenges: Node.js: For the server-side environment.
Express.js: As the web framework.
Multer: Middleware for handling multipart/form-data, which is used for uploading files.
MongoDB: For storing metadata and files.
GridFS: A specification for storing and retrieving large files in MongoDB. Step-by-Step Implementation 1. Setting Up the Project First, let’s create a new Node.js project and install the necessary dependencies. mkdir file-upload-app
cd file-upload-app
npm init -y
npm install express multer mongoose gridfs-stream 2. Creating the Server Next, we will set up a basic Express server. // server.js
const express = require('express');
const mongoose = require('mongoose');
const multer = require('multer');
const { GridFsStorage } = require('multer-gridfs-storage');
const Grid = require('gridfs-stream');
const methodOverride = require('method-override');
const bodyParser = require('body-parser');
const crypto = require('crypto');
const path = require('path');

const app = express();

// Middleware
app.use(bodyParser.json());
app.use(methodOverride('_method'));
app.set('view engine', 'ejs');

// Mongo URI
const mongoURI = 'mongodb://localhost:27017/fileupload';

// Create mongo connection
const conn = mongoose.createConnection(mongoURI, { useNewUrlParser: true, useUnifiedTopology: true });

// Init gfs
let gfs;
conn.once('open', () => {
  // Init stream
  gfs = Grid(conn.db, mongoose.mongo);
  gfs.collection('uploads');
});

// Create storage engine
const storage = new GridFsStorage({
  url: mongoURI,
  file: (req, file) => {
    return new Promise((resolve, reject) => {
      crypto.randomBytes(16, (err, buf) => {
        if (err) {
          return reject(err);
        }
        const filename = buf.toString('hex') + path.extname(file.originalname);
        const fileInfo = {
          filename: filename,
          bucketName: 'uploads'
        };
        resolve(fileInfo);
      });
    });
  }
});
const upload = multer({ storage });

// @route POST /upload
// @desc  Uploads file to DB
app.post('/upload', upload.single('file'), (req, res) => {
  res.json({ file: req.file });
});

// @route GET /files
// @desc  Display all files in JSON
app.get('/files', (req, res) => {
  gfs.files.find().toArray((err, files) => {
    // Check if files
    if (!files || files.length === 0) {
      return res.status(404).json({
        err: 'No files exist'
      });
    }

    // Files exist
    return res.json(files);
  });
});

// @route GET /files/:filename
// @desc  Display single file object
app.get('/files/:filename', (req, res) => {
  gfs.files.findOne({ filename: req.params.filename }, (err, file) => {
    if (!file || file.length === 0) {
      return res.status(404).json({
        err: 'No file exists'
      });
    }
    return res.json(file);
  });
});

// @route GET /image/:filename
// @desc Display Image
app.get('/image/:filename', (req, res) => {
  gfs.files.findOne({ filename: req.params.filename }, (err, file) => {
    if (!file || file.length === 0) {
      return res.status(404).json({
        err: 'No file exists'
      });
    }

    // Check if image
    if (file.contentType === 'image/jpeg' || file.contentType === 'image/png') {
      // Read output to browser
      const readstream = gfs.createReadStream(file.filename);
      readstream.pipe(res);
    } else {
      res.status(404).json({
        err: 'Not an image'
      });
    }
  });
});

// @route DELETE /files/:id
// @desc  Delete file
app.delete('/files/:id', (req, res) => {
  gfs.remove({ _id: req.params.id, root: 'uploads' }, (err, gridStore) => {
    if (err) {
      return res.status(404).json({ err: err });
    }
    res.redirect('/');
  });
});

const port = 5000;

app.listen(port, () => console.log(`Server started on port ${port}`)); 3. Testing the Upload Feature We can test our implementation using tools like Postman. Upload a File: Use a POST request to http://localhost:5000/upload with form data containing the file.
Retrieve Files: Use a GET request to http://localhost:5000/files to get the list of all files.
Retrieve a Single File: Use a GET request to http://localhost:5000/files/:filename to get a specific file’s details.
Display Image: Use a GET request to http://localhost:5000/image/:filename to view the image in the browser.
Delete a File: Use a DELETE request to http://localhost:5000/files/:id to delete a file. Conclusion In this article, I demonstrated how to handle file uploads efficiently using Node.js, Express.js, and MongoDB. By leveraging GridFS, we can store and manage large files seamlessly, ensuring scalability, security, and performance. This approach can be further customized and expanded based on specific application requirements, making it a versatile solution for file management in modern web applications. File uploads are a common feature in many web applications, but handling them efficiently and securely can be challenging. In this article, we will explore how to implement efficient file uploads using Node.js, Express.js, and MongoDB. We will cover the entire process, from setting up the server to storing files in MongoDB GridFS, ensuring that your application can handle large files without sacrificing performance. Problem Statement Handling file uploads efficiently involves several challenges: Scalability: Managing large files without overwhelming server memory. Security: Preventing unauthorized access and ensuring data integrity. Performance: Uploading and retrieving files quickly. Storage Management: Efficiently storing and retrieving files from the database. Scalability : Managing large files without overwhelming server memory. Scalability Security : Preventing unauthorized access and ensuring data integrity. Security Performance : Uploading and retrieving files quickly. Performance Storage Management : Efficiently storing and retrieving files from the database. Storage Management Solution We will use the following stack to address these challenges: Node.js: For the server-side environment. Express.js: As the web framework. Multer: Middleware for handling multipart/form-data, which is used for uploading files. MongoDB: For storing metadata and files. GridFS: A specification for storing and retrieving large files in MongoDB. Node.js : For the server-side environment. Node.js Express.js : As the web framework. Express.js Multer : Middleware for handling multipart/form-data , which is used for uploading files. Multer multipart/form-data MongoDB : For storing metadata and files. MongoDB GridFS : A specification for storing and retrieving large files in MongoDB. GridFS Step-by-Step Implementation 1. Setting Up the Project First, let’s create a new Node.js project and install the necessary dependencies. mkdir file-upload-app
cd file-upload-app
npm init -y
npm install express multer mongoose gridfs-stream mkdir file-upload-app
cd file-upload-app
npm init -y
npm install express multer mongoose gridfs-stream 2. Creating the Server Next, we will set up a basic Express server. // server.js
const express = require('express');
const mongoose = require('mongoose');
const multer = require('multer');
const { GridFsStorage } = require('multer-gridfs-storage');
const Grid = require('gridfs-stream');
const methodOverride = require('method-override');
const bodyParser = require('body-parser');
const crypto = require('crypto');
const path = require('path');

const app = express();

// Middleware
app.use(bodyParser.json());
app.use(methodOverride('_method'));
app.set('view engine', 'ejs');

// Mongo URI
const mongoURI = 'mongodb://localhost:27017/fileupload';

// Create mongo connection
const conn = mongoose.createConnection(mongoURI, { useNewUrlParser: true, useUnifiedTopology: true });

// Init gfs
let gfs;
conn.once('open', () => {
  // Init stream
  gfs = Grid(conn.db, mongoose.mongo);
  gfs.collection('uploads');
});

// Create storage engine
const storage = new GridFsStorage({
  url: mongoURI,
  file: (req, file) => {
    return new Promise((resolve, reject) => {
      crypto.randomBytes(16, (err, buf) => {
        if (err) {
          return reject(err);
        }
        const filename = buf.toString('hex') + path.extname(file.originalname);
        const fileInfo = {
          filename: filename,
          bucketName: 'uploads'
        };
        resolve(fileInfo);
      });
    });
  }
});
const upload = multer({ storage });

// @route POST /upload
// @desc  Uploads file to DB
app.post('/upload', upload.single('file'), (req, res) => {
  res.json({ file: req.file });
});

// @route GET /files
// @desc  Display all files in JSON
app.get('/files', (req, res) => {
  gfs.files.find().toArray((err, files) => {
    // Check if files
    if (!files || files.length === 0) {
      return res.status(404).json({
        err: 'No files exist'
      });
    }

    // Files exist
    return res.json(files);
  });
});

// @route GET /files/:filename
// @desc  Display single file object
app.get('/files/:filename', (req, res) => {
  gfs.files.findOne({ filename: req.params.filename }, (err, file) => {
    if (!file || file.length === 0) {
      return res.status(404).json({
        err: 'No file exists'
      });
    }
    return res.json(file);
  });
});

// @route GET /image/:filename
// @desc Display Image
app.get('/image/:filename', (req, res) => {
  gfs.files.findOne({ filename: req.params.filename }, (err, file) => {
    if (!file || file.length === 0) {
      return res.status(404).json({
        err: 'No file exists'
      });
    }

    // Check if image
    if (file.contentType === 'image/jpeg' || file.contentType === 'image/png') {
      // Read output to browser
      const readstream = gfs.createReadStream(file.filename);
      readstream.pipe(res);
    } else {
      res.status(404).json({
        err: 'Not an image'
      });
    }
  });
});

// @route DELETE /files/:id
// @desc  Delete file
app.delete('/files/:id', (req, res) => {
  gfs.remove({ _id: req.params.id, root: 'uploads' }, (err, gridStore) => {
    if (err) {
      return res.status(404).json({ err: err });
    }
    res.redirect('/');
  });
});

const port = 5000;

app.listen(port, () => console.log(`Server started on port ${port}`)); // server.js
const express = require('express');
const mongoose = require('mongoose');
const multer = require('multer');
const { GridFsStorage } = require('multer-gridfs-storage');
const Grid = require('gridfs-stream');
const methodOverride = require('method-override');
const bodyParser = require('body-parser');
const crypto = require('crypto');
const path = require('path');

const app = express();

// Middleware
app.use(bodyParser.json());
app.use(methodOverride('_method'));
app.set('view engine', 'ejs');

// Mongo URI
const mongoURI = 'mongodb://localhost:27017/fileupload';

// Create mongo connection
const conn = mongoose.createConnection(mongoURI, { useNewUrlParser: true, useUnifiedTopology: true });

// Init gfs
let gfs;
conn.once('open', () => {
  // Init stream
  gfs = Grid(conn.db, mongoose.mongo);
  gfs.collection('uploads');
});

// Create storage engine
const storage = new GridFsStorage({
  url: mongoURI,
  file: (req, file) => {
    return new Promise((resolve, reject) => {
      crypto.randomBytes(16, (err, buf) => {
        if (err) {
          return reject(err);
        }
        const filename = buf.toString('hex') + path.extname(file.originalname);
        const fileInfo = {
          filename: filename,
          bucketName: 'uploads'
        };
        resolve(fileInfo);
      });
    });
  }
});
const upload = multer({ storage });

// @route POST /upload
// @desc  Uploads file to DB
app.post('/upload', upload.single('file'), (req, res) => {
  res.json({ file: req.file });
});

// @route GET /files
// @desc  Display all files in JSON
app.get('/files', (req, res) => {
  gfs.files.find().toArray((err, files) => {
    // Check if files
    if (!files || files.length === 0) {
      return res.status(404).json({
        err: 'No files exist'
      });
    }

    // Files exist
    return res.json(files);
  });
});

// @route GET /files/:filename
// @desc  Display single file object
app.get('/files/:filename', (req, res) => {
  gfs.files.findOne({ filename: req.params.filename }, (err, file) => {
    if (!file || file.length === 0) {
      return res.status(404).json({
        err: 'No file exists'
      });
    }
    return res.json(file);
  });
});

// @route GET /image/:filename
// @desc Display Image
app.get('/image/:filename', (req, res) => {
  gfs.files.findOne({ filename: req.params.filename }, (err, file) => {
    if (!file || file.length === 0) {
      return res.status(404).json({
        err: 'No file exists'
      });
    }

    // Check if image
    if (file.contentType === 'image/jpeg' || file.contentType === 'image/png') {
      // Read output to browser
      const readstream = gfs.createReadStream(file.filename);
      readstream.pipe(res);
    } else {
      res.status(404).json({
        err: 'Not an image'
      });
    }
  });
});

// @route DELETE /files/:id
// @desc  Delete file
app.delete('/files/:id', (req, res) => {
  gfs.remove({ _id: req.params.id, root: 'uploads' }, (err, gridStore) => {
    if (err) {
      return res.status(404).json({ err: err });
    }
    res.redirect('/');
  });
});

const port = 5000;

app.listen(port, () => console.log(`Server started on port ${port}`)); 3. Testing the Upload Feature We can test our implementation using tools like Postman. Upload a File: Use a POST request to http://localhost:5000/upload with form data containing the file. Retrieve Files: Use a GET request to http://localhost:5000/files to get the list of all files. Retrieve a Single File: Use a GET request to http://localhost:5000/files/:filename to get a specific file’s details. Display Image: Use a GET request to http://localhost:5000/image/:filename to view the image in the browser. Delete a File: Use a DELETE request to http://localhost:5000/files/:id to delete a file. Upload a File : Use a POST request to http://localhost:5000/upload with form data containing the file. Upload a File http://localhost:5000/upload Retrieve Files : Use a GET request to http://localhost:5000/files to get the list of all files. Retrieve Files http://localhost:5000/files Retrieve a Single File : Use a GET request to http://localhost:5000/files/:filename to get a specific file’s details. Retrieve a Single File http://localhost:5000/files/:filename Display Image : Use a GET request to http://localhost:5000/image/:filename to view the image in the browser. Display Image http://localhost:5000/image/:filename Delete a File : Use a DELETE request to http://localhost:5000/files/:id to delete a file. Delete a File http://localhost:5000/files/:id Conclusion In this article, I demonstrated how to handle file uploads efficiently using Node.js, Express.js, and MongoDB. By leveraging GridFS, we can store and manage large files seamlessly, ensuring scalability, security, and performance. This approach can be further customized and expanded based on specific application requirements, making it a versatile solution for file management in modern web applications.

Efficient File Uploads in Node.js: Using Express, MongoDB, and GridFS for Scalable Storage

How to Build Your Own TODO-list Service With Golang and MongoDB

How to Integrate MongoDB in an AWS EC2 Instance

In this post, I will share real experience that I gained while working with hundreds of millions of pieces of data in MongoDB. Don't Store All Data in a Single Mongo Collection This was the worst mistake that we made, which became the root cause of all issues. We used to store 500 million complex structured data in a single collection, which caused the following side effects: Indexes creation takes a very long time.
If collection is dropped accidentally, all data is gone.
Querying data takes longer than usual, even though with indexes.
Counting with filters is timing out because of large document scanning. Probably there will be even more issues as the data increases on a daily basis. Bad Document Structure Invest some time to define the document structure precisely. This is a critical part that never gets attention when starting the project. Our mistake was excluding the field entirely from some of the documents instead of assigning a default value for the field. That ended up with slow query scanning because of existence checking. In MongoDB, it's possible to give partial indexes, but it's really not working that well where field existence is checked. The example below shows null value assigned for phone instead of excluding it from the document completely. {
  "_id": "507f191e810c19729de860ea",
  "name": "John Doe",
  "age": 30,
  "email": "john.doe@example.com",
  "phone": null, //assign default value instead of removing
  "isActive": true,
  "tags": ["user", "active", "new"]
  "registeredOn": "2024-04-19T15:00:00Z"
} {$exists: true} filter is slow so consider adding a default value and indexing the fields for fast query results. Use Bulk Operations Bulk operations are used to execute multiple write operations (inserts, updates, deletes) efficiently in a single command. This method can significantly reduce the number of round trips between your application and the MongoDB server, leading to performance improvements, especially when dealing with large volumes of data. Since we were processing millions of data from RabbitMQ queues, it required handling messages in batches and making bulk DB operations for each batch. Here's the documentation for pymongo on how to use bulk operations for better write performance. Use Indexes Indexing plays a crucial part in the DB performance. Create required indexes in the collection based on the filters you are using in the codebase. That also refers to the part about creating a valid document structure. Instead of removing the field completely, assign a default value to benefit from indexing power. Instead of Count, Use Aggregation Pipeline If you need to get count of data based on a specific filter, I wouldn't recommend using count() , instead, use a combination of $match and $count aggregation for fast performance. $match filters the documents first, which means only the relevant documents that meet the specified conditions are passed down to the $count stage. This is more efficient than counting all documents and then filtering them, as count() might initially do without any conditions. Wrap Up That's all that I have experienced so far with MongoDB. I will update the article once I find more hints and best practices to help optimize the performance. In this post, I will share real experience that I gained while working with hundreds of millions of pieces of data in MongoDB. Don't Store All Data in a Single Mongo Collection This was the worst mistake that we made, which became the root cause of all issues. We used to store 500 million complex structured data in a single collection, which caused the following side effects: Indexes creation takes a very long time. If collection is dropped accidentally, all data is gone. Querying data takes longer than usual, even though with indexes. Counting with filters is timing out because of large document scanning. Indexes creation takes a very long time. If collection is dropped accidentally, all data is gone. Querying data takes longer than usual, even though with indexes. Counting with filters is timing out because of large document scanning. Probably there will be even more issues as the data increases on a daily basis. Bad Document Structure Invest some time to define the document structure precisely. This is a critical part that never gets attention when starting the project. Our mistake was excluding the field entirely from some of the documents instead of assigning a default value for the field. That ended up with slow query scanning because of existence checking. In MongoDB, it's possible to give partial indexes, but it's really not working that well where field existence is checked. The example below shows null value assigned for phone instead of excluding it from the document completely. null phone {
  "_id": "507f191e810c19729de860ea",
  "name": "John Doe",
  "age": 30,
  "email": "john.doe@example.com",
  "phone": null, //assign default value instead of removing
  "isActive": true,
  "tags": ["user", "active", "new"]
  "registeredOn": "2024-04-19T15:00:00Z"
} {
  "_id": "507f191e810c19729de860ea",
  "name": "John Doe",
  "age": 30,
  "email": "john.doe@example.com",
  "phone": null, //assign default value instead of removing
  "isActive": true,
  "tags": ["user", "active", "new"]
  "registeredOn": "2024-04-19T15:00:00Z"
} {$exists: true} filter is slow so consider adding a default value and indexing the fields for fast query results. {$exists: true} Use Bulk Operations Bulk operations are used to execute multiple write operations (inserts, updates, deletes) efficiently in a single command. This method can significantly reduce the number of round trips between your application and the MongoDB server, leading to performance improvements, especially when dealing with large volumes of data. Since we were processing millions of data from RabbitMQ queues, it required handling messages in batches and making bulk DB operations for each batch. Here's the documentation for pymongo on how to use bulk operations for better write performance. documentation documentation pymongo Use Indexes Indexing plays a crucial part in the DB performance. Create required indexes in the collection based on the filters you are using in the codebase. That also refers to the part about creating a valid document structure. Instead of removing the field completely, assign a default value to benefit from indexing power. Instead of Count, Use Aggregation Pipeline If you need to get count of data based on a specific filter, I wouldn't recommend using count() , instead, use a combination of $match and $count aggregation for fast performance. count() $match $count $match filters the documents first, which means only the relevant documents that meet the specified conditions are passed down to the $count stage. This is more efficient than counting all documents and then filtering them, as count() might initially do without any conditions. $match $count count() Wrap Up That's all that I have experienced so far with MongoDB. I will update the article once I find more hints and best practices to help optimize the performance.

Lessons I Learned From Managing Hundreds of Millions of Data in MongoDB

Once, I was tasked with improving the database and general app operations of backend services which were using MongoDB as their main database. These services were part of a huge infrastructure where millions of messages came through the queues and needed to be processed based on the message actions. That means tons of DB ops each second and other additional checks while processing. Real Case Scenario The processing layer of the service was using Pymongo to interact with MongoDB and the service itself was running in a synchronous environment. Even though the database operations were handled in bulk, the performance was still not capable of handling incoming data. Synchronous code was making things even worse. The code execution waits for the result from the current operation to move forward. That's a serious bottleneck in scalable systems. This was causing queue overflows and potential data loss every time. Asynchronous Environment The solution I implemented was a combination of: Motor
Asyncio
Uvloop Let's quickly go through the definitions of these items. PyMongo is the official MongoDB driver for Python, providing a simple and intuitive way to interact with MongoDB databases. It's synchronous, meaning each database operation blocks the execution of your program until it completes it, which can be a bottleneck in I/O-bound tasks. Motor is the asynchronous driver for MongoDB, built on top of PyMongo and designed to take advantage of Python's asyncio library. Motor allows you to perform non-blocking database operations, making it suitable for high-performance applications that require concurrency. To illustrate the performance differences, I prepared a stress test using two scripts: one using Motor (asynchronous) and the other using PyMongo (synchronous). Both scripts performed the same task of reading and writing documents to MongoDB in batches. Both scripts read 300k documents from the source collection and migrated them to the new target collection. Asynchronous Script (Motor) import logging
import asyncio
import time
from bson import ObjectId
from motor.motor_asyncio import AsyncIOMotorClient

# Configure logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

# MongoDB setup
MONGO_URI = 'mongodb://root:root@localhost:27019'
DB_NAME = 'products'
COLLECTION_NAME = 'gmc_products'

client = AsyncIOMotorClient(MONGO_URI)
db = client[DB_NAME]
collection = db[COLLECTION_NAME]
target_collection = db["new_collection"]

async def fetch_products(batch_size, last_id=None):
    query = {'_id': {'$gt': last_id}} if last_id else {}
    cursor = collection.find(query).sort('_id').limit(batch_size)
    products = await cursor.to_list(length=batch_size)
    return products

async def bulk_write_to_mongo(products):
    for product in products:
        product['_id'] = ObjectId()  # Generate a new ObjectId for each product

    try:
        result = await target_collection.insert_many(products, ordered=False)
        logger.info(f'Inserted {len(result.inserted_ids)} products into MongoDB.')
    except Exception as e:
        logger.error(f'Error inserting products into MongoDB: {e}')

async def process_batches(batch_size, concurrency_limit):
    tasks = []
    last_id = None
    while True:
        products = await fetch_products(batch_size, last_id)
        if not products:
            break
        last_id = products[-1]['_id']
        tasks.append(bulk_write_to_mongo(products))
        if len(tasks) >= concurrency_limit:
            await asyncio.gather(*tasks)
            tasks = []
    # Process remaining tasks if any
    if tasks:
        await asyncio.gather(*tasks)

async def main():
    batch_size = 1000
    concurrency_limit = 10

    start_time = time.time()
    await process_batches(batch_size, concurrency_limit)
    end_time = time.time()
    logger.info(f'Total time: {end_time - start_time:.2f} seconds.')

if __name__ == '__main__':
    asyncio.run(main()) Synchronous Script (Pymongo) import logging
import time
from bson import ObjectId
from pymongo import MongoClient

# Configure logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

# MongoDB setup
MONGO_URI = 'mongodb://root:root@localhost:27019'
DB_NAME = 'products'
COLLECTION_NAME = 'gmc_products'
TARGET_COLLECTION_NAME = 'new_collection'

client = MongoClient(MONGO_URI)
db = client[DB_NAME]
collection = db[COLLECTION_NAME]
target_collection = db[TARGET_COLLECTION_NAME]

def fetch_products(batch_size, last_id=None):
    query = {'_id': {'$gt': last_id}} if last_id else {}
    cursor = collection.find(query).sort('_id').limit(batch_size)
    products = list(cursor)
    return products

def bulk_write_to_mongo(products):
    for product in products:
        product['_id'] = ObjectId()  # Generate a new ObjectId for each product

    try:
        result = target_collection.insert_many(products, ordered=False)
        logger.info(f'Inserted {len(result.inserted_ids)} products into MongoDB.')
    except Exception as e:
        logger.error(f'Error inserting products into MongoDB: {e}')

def process_batches(batch_size):
    last_id = None
    while True:
        products = fetch_products(batch_size, last_id)
        if not products:
            break
        last_id = products[-1]['_id']
        bulk_write_to_mongo(products)

def main():
    batch_size = 1000

    start_time = time.time()
    process_batches(batch_size)
    end_time = time.time()
    logger.info(f'Total time: {end_time - start_time:.2f} seconds.')

if __name__ == '__main__':
    main() Results and Analysis Execution Time of Migrating 300k documents: Asynchronous script: 17.15 seconds
Synchronous script: 23.26 seconds The asynchronous script completed the task 6.11 seconds faster than the synchronous script. While this might not seem like a significant difference for a single run, it becomes more pronounced in high-load scenarios or when processing large datasets continuously. Benefits of Using Motor and Asynchronous Environment Improved Throughput: Asynchronous operations can handle more tasks concurrently, increasing overall throughput. This is especially beneficial in applications with high I/O operations, such as web servers handling multiple database queries simultaneously. Non-Blocking I/O: Asynchronous operations do not block the main thread, allowing other tasks to continue running. This results in better CPU utilization and smoother application performance, particularly under load. Scalability: Asynchronous code scales better with the number of concurrent operations. For example, a web application using Motor can handle more simultaneous requests compared to one using PyMongo. Resource Efficiency: Asynchronous operations can lead to more efficient use of system resources. For instance, the event loop in asyncio allows the application to switch between tasks, reducing idle times and improving overall efficiency. You can find the source code on the GitHub repository below: https://github.com/PylotStuff/motor-asyncio-performance?embedable=true Conclusion The choice between Motor and PyMongo depends on the specific needs of your application. For applications that require high concurrency and efficient I/O handling, Motor and the asynchronous approach offer significant advantages. However, for simpler applications or scripts where ease of implementation is a priority, PyMongo's synchronous approach might be sufficient. By leveraging asynchronous operations with Motor, you can build more scalable and performant applications, making it a worthwhile consideration for modern web development. Once, I was tasked with improving the database and general app operations of backend services which were using MongoDB as their main database. These services were part of a huge infrastructure where millions of messages came through the queues and needed to be processed based on the message actions. That means tons of DB ops each second and other additional checks while processing. Real Case Scenario The processing layer of the service was using Pymongo to interact with MongoDB and the service itself was running in a synchronous environment. Even though the database operations were handled in bulk, the performance was still not capable of handling incoming data. Synchronous code was making things even worse. The code execution waits for the result from the current operation to move forward. That's a serious bottleneck in scalable systems. This was causing queue overflows and potential data loss every time. Asynchronous Environment The solution I implemented was a combination of: Motor Asyncio Uvloop Motor Asyncio Uvloop Let's quickly go through the definitions of these items. PyMongo is the official MongoDB driver for Python, providing a simple and intuitive way to interact with MongoDB databases. It's synchronous, meaning each database operation blocks the execution of your program until it completes it, which can be a bottleneck in I/O-bound tasks. PyMongo Motor is the asynchronous driver for MongoDB, built on top of PyMongo and designed to take advantage of Python's asyncio library. Motor allows you to perform non-blocking database operations, making it suitable for high-performance applications that require concurrency. Motor To illustrate the performance differences, I prepared a stress test using two scripts: one using Motor (asynchronous) and the other using PyMongo (synchronous). Both scripts performed the same task of reading and writing documents to MongoDB in batches. Both scripts read 300k documents from the source collection and migrated them to the new target collection. Asynchronous Script (Motor) import logging
import asyncio
import time
from bson import ObjectId
from motor.motor_asyncio import AsyncIOMotorClient

# Configure logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

# MongoDB setup
MONGO_URI = 'mongodb://root:root@localhost:27019'
DB_NAME = 'products'
COLLECTION_NAME = 'gmc_products'

client = AsyncIOMotorClient(MONGO_URI)
db = client[DB_NAME]
collection = db[COLLECTION_NAME]
target_collection = db["new_collection"]

async def fetch_products(batch_size, last_id=None):
    query = {'_id': {'$gt': last_id}} if last_id else {}
    cursor = collection.find(query).sort('_id').limit(batch_size)
    products = await cursor.to_list(length=batch_size)
    return products

async def bulk_write_to_mongo(products):
    for product in products:
        product['_id'] = ObjectId()  # Generate a new ObjectId for each product

    try:
        result = await target_collection.insert_many(products, ordered=False)
        logger.info(f'Inserted {len(result.inserted_ids)} products into MongoDB.')
    except Exception as e:
        logger.error(f'Error inserting products into MongoDB: {e}')

async def process_batches(batch_size, concurrency_limit):
    tasks = []
    last_id = None
    while True:
        products = await fetch_products(batch_size, last_id)
        if not products:
            break
        last_id = products[-1]['_id']
        tasks.append(bulk_write_to_mongo(products))
        if len(tasks) >= concurrency_limit:
            await asyncio.gather(*tasks)
            tasks = []
    # Process remaining tasks if any
    if tasks:
        await asyncio.gather(*tasks)

async def main():
    batch_size = 1000
    concurrency_limit = 10

    start_time = time.time()
    await process_batches(batch_size, concurrency_limit)
    end_time = time.time()
    logger.info(f'Total time: {end_time - start_time:.2f} seconds.')

if __name__ == '__main__':
    asyncio.run(main()) import logging
import asyncio
import time
from bson import ObjectId
from motor.motor_asyncio import AsyncIOMotorClient

# Configure logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

# MongoDB setup
MONGO_URI = 'mongodb://root:root@localhost:27019'
DB_NAME = 'products'
COLLECTION_NAME = 'gmc_products'

client = AsyncIOMotorClient(MONGO_URI)
db = client[DB_NAME]
collection = db[COLLECTION_NAME]
target_collection = db["new_collection"]

async def fetch_products(batch_size, last_id=None):
    query = {'_id': {'$gt': last_id}} if last_id else {}
    cursor = collection.find(query).sort('_id').limit(batch_size)
    products = await cursor.to_list(length=batch_size)
    return products

async def bulk_write_to_mongo(products):
    for product in products:
        product['_id'] = ObjectId()  # Generate a new ObjectId for each product

    try:
        result = await target_collection.insert_many(products, ordered=False)
        logger.info(f'Inserted {len(result.inserted_ids)} products into MongoDB.')
    except Exception as e:
        logger.error(f'Error inserting products into MongoDB: {e}')

async def process_batches(batch_size, concurrency_limit):
    tasks = []
    last_id = None
    while True:
        products = await fetch_products(batch_size, last_id)
        if not products:
            break
        last_id = products[-1]['_id']
        tasks.append(bulk_write_to_mongo(products))
        if len(tasks) >= concurrency_limit:
            await asyncio.gather(*tasks)
            tasks = []
    # Process remaining tasks if any
    if tasks:
        await asyncio.gather(*tasks)

async def main():
    batch_size = 1000
    concurrency_limit = 10

    start_time = time.time()
    await process_batches(batch_size, concurrency_limit)
    end_time = time.time()
    logger.info(f'Total time: {end_time - start_time:.2f} seconds.')

if __name__ == '__main__':
    asyncio.run(main()) Synchronous Script (Pymongo) import logging
import time
from bson import ObjectId
from pymongo import MongoClient

# Configure logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

# MongoDB setup
MONGO_URI = 'mongodb://root:root@localhost:27019'
DB_NAME = 'products'
COLLECTION_NAME = 'gmc_products'
TARGET_COLLECTION_NAME = 'new_collection'

client = MongoClient(MONGO_URI)
db = client[DB_NAME]
collection = db[COLLECTION_NAME]
target_collection = db[TARGET_COLLECTION_NAME]

def fetch_products(batch_size, last_id=None):
    query = {'_id': {'$gt': last_id}} if last_id else {}
    cursor = collection.find(query).sort('_id').limit(batch_size)
    products = list(cursor)
    return products

def bulk_write_to_mongo(products):
    for product in products:
        product['_id'] = ObjectId()  # Generate a new ObjectId for each product

    try:
        result = target_collection.insert_many(products, ordered=False)
        logger.info(f'Inserted {len(result.inserted_ids)} products into MongoDB.')
    except Exception as e:
        logger.error(f'Error inserting products into MongoDB: {e}')

def process_batches(batch_size):
    last_id = None
    while True:
        products = fetch_products(batch_size, last_id)
        if not products:
            break
        last_id = products[-1]['_id']
        bulk_write_to_mongo(products)

def main():
    batch_size = 1000

    start_time = time.time()
    process_batches(batch_size)
    end_time = time.time()
    logger.info(f'Total time: {end_time - start_time:.2f} seconds.')

if __name__ == '__main__':
    main() import logging
import time
from bson import ObjectId
from pymongo import MongoClient

# Configure logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

# MongoDB setup
MONGO_URI = 'mongodb://root:root@localhost:27019'
DB_NAME = 'products'
COLLECTION_NAME = 'gmc_products'
TARGET_COLLECTION_NAME = 'new_collection'

client = MongoClient(MONGO_URI)
db = client[DB_NAME]
collection = db[COLLECTION_NAME]
target_collection = db[TARGET_COLLECTION_NAME]

def fetch_products(batch_size, last_id=None):
    query = {'_id': {'$gt': last_id}} if last_id else {}
    cursor = collection.find(query).sort('_id').limit(batch_size)
    products = list(cursor)
    return products

def bulk_write_to_mongo(products):
    for product in products:
        product['_id'] = ObjectId()  # Generate a new ObjectId for each product

    try:
        result = target_collection.insert_many(products, ordered=False)
        logger.info(f'Inserted {len(result.inserted_ids)} products into MongoDB.')
    except Exception as e:
        logger.error(f'Error inserting products into MongoDB: {e}')

def process_batches(batch_size):
    last_id = None
    while True:
        products = fetch_products(batch_size, last_id)
        if not products:
            break
        last_id = products[-1]['_id']
        bulk_write_to_mongo(products)

def main():
    batch_size = 1000

    start_time = time.time()
    process_batches(batch_size)
    end_time = time.time()
    logger.info(f'Total time: {end_time - start_time:.2f} seconds.')

if __name__ == '__main__':
    main() Results and Analysis Execution Time of Migrating 300k documents: Asynchronous script: 17.15 seconds Synchronous script: 23.26 seconds Asynchronous script: 17.15 seconds Synchronous script: 23.26 seconds The asynchronous script completed the task 6.11 seconds faster than the synchronous script. 6.11 seconds While this might not seem like a significant difference for a single run, it becomes more pronounced in high-load scenarios or when processing large datasets continuously. Benefits of Using Motor and Asynchronous Environment Improved Throughput : Asynchronous operations can handle more tasks concurrently, increasing overall throughput. This is especially beneficial in applications with high I/O operations, such as web servers handling multiple database queries simultaneously. Improved Throughput Non-Blocking I/O : Asynchronous operations do not block the main thread, allowing other tasks to continue running. This results in better CPU utilization and smoother application performance, particularly under load. Non-Blocking I/O Scalability : Asynchronous code scales better with the number of concurrent operations. For example, a web application using Motor can handle more simultaneous requests compared to one using PyMongo. Scalability Resource Efficiency : Asynchronous operations can lead to more efficient use of system resources. For instance, the event loop in asyncio allows the application to switch between tasks, reducing idle times and improving overall efficiency. Resource Efficiency You can find the source code on the GitHub repository below: https://github.com/PylotStuff/motor-asyncio-performance?embedable=true https://github.com/PylotStuff/motor-asyncio-performance?embedable=true Conclusion The choice between Motor and PyMongo depends on the specific needs of your application. For applications that require high concurrency and efficient I/O handling, Motor and the asynchronous approach offer significant advantages. However, for simpler applications or scripts where ease of implementation is a priority, PyMongo's synchronous approach might be sufficient. By leveraging asynchronous operations with Motor, you can build more scalable and performant applications, making it a worthwhile consideration for modern web development.

Boost MongoDB Performance: Motor Client vs PyMongo - Which is Faster?

Working with Mongoose models can sometimes feel like keeping track of a complex web. Unlike many SQL databases, finding free and easy tools to visualize your Mongoose schemas can be tough. This can make it difficult to understand how your data is connected and collaborate with other developers. To solve this problem, I built an open-source tool that visualizes your Mongoose schemas. Now you can easily see how your data fits together and keep your projects organized! URL: http://mongoose-schema-visualize.s3-website.ap-south-1.amazonaws.com/ Repo: https://github.com/iam-abdul/Mongoose-Schema-Visualization-Tool NPM Package: https://www.npmjs.com/package/mongoose-parser My Approach While it might be tempting to directly query the database using the connection URI to extract model relationships, this approach has a few drawbacks: Security Concerns: Sharing connection URIs can be a security risk, as it exposes sensitive database credentials.
Unnecessary Complexity: For a simple task like schema visualization, querying the database adds unnecessary complexity to the tool. To address these concerns, I opted to obtain the schema by parsing the JavaScript files. This approach has some key advantages: Security: By parsing the code, there’s no need to handle connection URIs, keeping those credentials safe.


Efficiency: Parsing JavaScript files is generally faster and lighter on resources compared to querying the database. Parsing the JavaScript files does require some additional logic, but the benefits in terms of security and efficiency make it a worthwhile trade-off. It’s important to note that this entire operation happens within your browser. Since it’s a static site, there are no API calls involved. You simply upload your JavaScript files, and the tool does its magic to visualize your Mongoose schema relationships, all without ever touching your database directly. Parsing the Schema Files Extracting model relationships from JavaScript files can be done using regular expressions. However, this approach has some limitations: Complexity: Mongoose schema definitions can involve complex nesting and property structures. Regular expressions can quickly become cumbersome and error-prone when dealing with such complexity.


Maintainability: As your schemas evolve, regex patterns might need frequent updates to keep up, making the code harder to maintain in the long run. To address these limitations, I opted for a more robust approach: parsing the Abstract Syntax Tree of the JavaScript files. What’s an AST? An AST, or Abstract Syntax Tree, is a tree-like representation of the code’s structure. Imagine it like a family tree but for your code! It breaks down the code into its core elements, like variables, functions, and expressions, making it easier to understand the relationships between different parts of the code. Where are ASTs Used? ASTs are a powerful tool used in various programming contexts beyond this schema visualization tool. Here are a few examples: Compilers: Compilers, which translate code from one language to another, often use ASTs as an intermediate representation to analyze and optimize the code before generating the final output.


Linters and Static Code Analysers: These tools use ASTs to identify potential errors, stylistic inconsistencies, or inefficiencies in the code without actually running it. Benefits of Using an AST for Schema Visualisation By converting the JavaScript files to ASTs, the tool gains several advantages: Accuracy: ASTs provide a more accurate and reliable way to identify schema definitions within your code.


Flexibility: ASTs can handle complex code structures more effectively than regular expressions, making the tool adaptable to different schema definition styles.


Maintainability: Using ASTs makes the parsing logic cleaner and easier to maintain as your schemas evolve. While parsing with an AST requires a bit more initial setup compared to regular expressions, the benefits in terms of accuracy, flexibility, and maintainability make it a worthwhile investment for this tool. It allows for robust and reliable extraction of model relationships from your JavaScript schema files. Since mongoose schemas are javascript files, I made use of acorn (https://www.npmjs.com/package/acorn) a javascript parser written in javascript 🙃 to build the AST. Acron creates the AST for the given javascript files and from that i look for mongoose methods and extract the associated model definitions. I converted this set of functions into an NPM package of its own (https://www.npmjs.com/package/mongoose-parser). The tool uses this package to extract the schemas. Once the schema is extracted, the frontend react app converts this schema into a set of nodes and edges and creates the visual representation with the help of react-flow (https://www.npmjs.com/package/reactflow). Usage Just drag and drop the JavaScript files containing your Mongoose schema definitions here, and watch your schemas come to life! http://mongoose-schema-visualize.s3-website.ap-south-1.amazonaws.com/ Thanks for reading :) Socials Linkedin: https://www.linkedin.com/in/abdul-mohammad-a567b2183/ Github: https://github.com/iam-abdul Twitter: https://x.com/iam_abd038 Working with Mongoose models can sometimes feel like keeping track of a complex web. Unlike many SQL databases, finding free and easy tools to visualize your Mongoose schemas can be tough. This can make it difficult to understand how your data is connected and collaborate with other developers. To solve this problem, I built an open-source tool that visualizes your Mongoose schemas. Now you can easily see how your data fits together and keep your projects organized! URL: http://mongoose-schema-visualize.s3-website.ap-south-1.amazonaws.com/ http://mongoose-schema-visualize.s3-website.ap-south-1.amazonaws.com/ Repo: https://github.com/iam-abdul/Mongoose-Schema-Visualization-Tool https://github.com/iam-abdul/Mongoose-Schema-Visualization-Tool NPM Package: https://www.npmjs.com/package/mongoose-parser https://www.npmjs.com/package/mongoose-parser My Approach My Approach While it might be tempting to directly query the database using the connection URI to extract model relationships, this approach has a few drawbacks: Security Concerns: Sharing connection URIs can be a security risk, as it exposes sensitive database credentials. Unnecessary Complexity: For a simple task like schema visualization, querying the database adds unnecessary complexity to the tool. Security Concerns: Sharing connection URIs can be a security risk, as it exposes sensitive database credentials. Security Concerns: Unnecessary Complexity: For a simple task like schema visualization, querying the database adds unnecessary complexity to the tool. Unnecessary Complexity: To address these concerns, I opted to obtain the schema by parsing the JavaScript files. This approach has some key advantages: Security: By parsing the code, there’s no need to handle connection URIs, keeping those credentials safe. Efficiency: Parsing JavaScript files is generally faster and lighter on resources compared to querying the database. Security: By parsing the code, there’s no need to handle connection URIs, keeping those credentials safe. Security: By parsing the code, there’s no need to handle connection URIs, keeping those credentials safe. Security: Efficiency: Parsing JavaScript files is generally faster and lighter on resources compared to querying the database. Efficiency: Parsing JavaScript files is generally faster and lighter on resources compared to querying the database. Efficiency: Parsing the JavaScript files does require some additional logic, but the benefits in terms of security and efficiency make it a worthwhile trade-off. It’s important to note that this entire operation happens within your browser. Since it’s a static site, there are no API calls involved. You simply upload your JavaScript files, and the tool does its magic to visualize your Mongoose schema relationships, all without ever touching your database directly. Parsing the Schema Files Extracting model relationships from JavaScript files can be done using regular expressions. However, this approach has some limitations: Complexity: Mongoose schema definitions can involve complex nesting and property structures. Regular expressions can quickly become cumbersome and error-prone when dealing with such complexity. Maintainability: As your schemas evolve, regex patterns might need frequent updates to keep up, making the code harder to maintain in the long run. Complexity: Mongoose schema definitions can involve complex nesting and property structures. Regular expressions can quickly become cumbersome and error-prone when dealing with such complexity. Complexity: Mongoose schema definitions can involve complex nesting and property structures. Regular expressions can quickly become cumbersome and error-prone when dealing with such complexity. Complexity: Maintainability: As your schemas evolve, regex patterns might need frequent updates to keep up, making the code harder to maintain in the long run. Maintainability: As your schemas evolve, regex patterns might need frequent updates to keep up, making the code harder to maintain in the long run. Maintainability: To address these limitations, I opted for a more robust approach: parsing the Abstract Syntax Tree of the JavaScript files. What’s an AST? What’s an AST? An AST, or Abstract Syntax Tree, is a tree-like representation of the code’s structure. Imagine it like a family tree but for your code! It breaks down the code into its core elements, like variables, functions, and expressions, making it easier to understand the relationships between different parts of the code. Where are ASTs Used? Where are ASTs Used? ASTs are a powerful tool used in various programming contexts beyond this schema visualization tool. Here are a few examples: Compilers: Compilers, which translate code from one language to another, often use ASTs as an intermediate representation to analyze and optimize the code before generating the final output. Linters and Static Code Analysers: These tools use ASTs to identify potential errors, stylistic inconsistencies, or inefficiencies in the code without actually running it. Compilers: Compilers, which translate code from one language to another, often use ASTs as an intermediate representation to analyze and optimize the code before generating the final output. Compilers: Compilers, which translate code from one language to another, often use ASTs as an intermediate representation to analyze and optimize the code before generating the final output. Compilers: Linters and Static Code Analysers: These tools use ASTs to identify potential errors, stylistic inconsistencies, or inefficiencies in the code without actually running it. Linters and Static Code Analysers: These tools use ASTs to identify potential errors, stylistic inconsistencies, or inefficiencies in the code without actually running it. Linters and Static Code Analysers: Benefits of Using an AST for Schema Visualisation Benefits of Using an AST for Schema Visualisation By converting the JavaScript files to ASTs, the tool gains several advantages: Accuracy: ASTs provide a more accurate and reliable way to identify schema definitions within your code. Flexibility: ASTs can handle complex code structures more effectively than regular expressions, making the tool adaptable to different schema definition styles. Maintainability: Using ASTs makes the parsing logic cleaner and easier to maintain as your schemas evolve. Accuracy: ASTs provide a more accurate and reliable way to identify schema definitions within your code. Accuracy: ASTs provide a more accurate and reliable way to identify schema definitions within your code. Accuracy: Flexibility: ASTs can handle complex code structures more effectively than regular expressions, making the tool adaptable to different schema definition styles. Flexibility: ASTs can handle complex code structures more effectively than regular expressions, making the tool adaptable to different schema definition styles. Flexibility: Maintainability: Using ASTs makes the parsing logic cleaner and easier to maintain as your schemas evolve. Maintainability: Using ASTs makes the parsing logic cleaner and easier to maintain as your schemas evolve. Maintainability: While parsing with an AST requires a bit more initial setup compared to regular expressions, the benefits in terms of accuracy, flexibility, and maintainability make it a worthwhile investment for this tool. It allows for robust and reliable extraction of model relationships from your JavaScript schema files. Since mongoose schemas are javascript files, I made use of acorn ( https://www.npmjs.com/package/acorn ) a javascript parser written in javascript 🙃 to build the AST. https://www.npmjs.com/package/acorn Acron creates the AST for the given javascript files and from that i look for mongoose methods and extract the associated model definitions. I converted this set of functions into an NPM package of its own ( https://www.npmjs.com/package/mongoose-parser ). The tool uses this package to extract the schemas. https://www.npmjs.com/package/mongoose-parser Once the schema is extracted, the frontend react app converts this schema into a set of nodes and edges and creates the visual representation with the help of react-flow ( https://www.npmjs.com/package/reactflow ). https://www.npmjs.com/package/reactflow Usage Just drag and drop the JavaScript files containing your Mongoose schema definitions here, and watch your schemas come to life! http://mongoose-schema-visualize.s3-website.ap-south-1.amazonaws.com/ http://mongoose-schema-visualize.s3-website.ap-south-1.amazonaws.com/ Thanks for reading :) Socials Socials Linkedin: https://www.linkedin.com/in/abdul-mohammad-a567b2183/ https://www.linkedin.com/in/abdul-mohammad-a567b2183/ Github: https://github.com/iam-abdul https://github.com/iam-abdul Twitter: https://x.com/iam_abd038

I Built an Open-Source Schema Visualization Tool for Mongoose/MongoDB Under 1000 Lines of Code

In previous articles, I explained how we could . But we started this exploration into MongoDB, covering a lot of the basics, and I wanted to start looking into more interesting aspects of how we use these things together. As I was putting video content together about all of these topics, one thing that caught my attention was the generic methods vs. methods that operated on BsonDocument, and I was curious if the performance was any different. So, to start, I figured we’d look at C# MongoDB insert benchmarks and see if any interesting patterns stand out. use C# and MongoDB to get records inserted Considerations For These C# MongoDB Benchmarks I was chatting with on Twitter the other day about benchmarks that he often posts on social media. It turns out there was another discussion floating around the Twitter-verse, where sync vs. async calls to databases were being debated. The proposal was that for very fast DB queries, the async variations are indeed slower. David Callan Of course, this got the gears turning. I have been writing about and creating videos for MongoDB in C# all week and hinting at performance benchmarks coming. This coupled with my conversation with Dave AND the conversation I saw on Twitter made me think that I had to give this more attention. I speculated it might have something to do with Task vs ValueTask, but I didn’t have a lot of grounds for that. However, I originally wanted to look into these benchmarks because I was curious if using BsonDocument compared to a dedicated DTO (whether it was a struct, class, or record variation) would have different performance characteristics. That meant that I would want to ensure I covered a matrix across: Synchronous Asynchronous Asynchronous with ValueTask Struct Class Record Struct Record Class BsonDocument And with that, I set off to go write some super simple benchmarks! Let’s check them out. The C# MongoDB Insert Benchmarking Code As with all benchmarking we do in C#, is our go-to tool! Make sure you start by installing the BenchmarkDotNet NuGet package. We’ll use this to ensure consistent setup, warmup, runs, and reporting for our MongoDB benchmarks. BenchmarkDotNet Since we’re trying to reduce as many possible external factors as we can for these benchmarks, we’re going to use Testcontainers to run an instance of MongoDB in a Docker container. Of course, interacting with anything outside of our code directly can lead to inconsistencies and more room for error to show up in our results. However, this should help minimize things. You’ll want to get the Testcontainers.MongoDB NuGet package for this as well. You can , but we’ll start with what our entry point looks like: find all the relevant code on GitHub using BenchmarkDotNet.Running;

using System.Reflection;

BenchmarkRunner.Run(
 Assembly.GetExecutingAssembly(),
 args: args); Nice and simple just to kick off the benchmarks. And the benchmarks are the most important part here: using BenchmarkDotNet.Attributes;

using MongoDB.Bson;
using MongoDB.Driver;

using Testcontainers.MongoDb;

[MemoryDiagnoser]
//[ShortRunJob]
[MediumRunJob]
public class InsertBenchmarks
{
 private MongoDbContainer? _container;
 private MongoClient? _mongoClient;
 private IMongoCollection<BsonDocument>? _collection;
 private IMongoCollection<RecordStructDto>? _collectionRecordStruct;
 private IMongoCollection<RecordClassDto>? _collectionRecordClass;
 private IMongoCollection<StructDto>? _collectionStruct;
 private IMongoCollection<ClassDto>? _collectionClass;

 [GlobalSetup]
 public async Task SetupAsync()
 {
 _container = new MongoDbBuilder()
 .WithImage("mongo:latest")
 .Build();
 await _container.StartAsync();
 
 _mongoClient = new MongoClient(_container.GetConnectionString());
 var database = _mongoClient.GetDatabase("test");
 _collection = database.GetCollection<BsonDocument>("test");
 _collectionRecordStruct = database.GetCollection<RecordStructDto>("test");
 _collectionRecordClass = database.GetCollection<RecordClassDto>("test");
 _collectionStruct = database.GetCollection<StructDto>("test");
 _collectionClass = database.GetCollection<ClassDto>("test");
 }

 [GlobalCleanup]
 public async Task CleanupAsync()
 {
 await _container!.StopAsync();
 }

 [Benchmark]
 public async Task InsertOneAsync_BsonDocument()
 {
 await _collection!.InsertOneAsync(new BsonDocument()
 {
 ["Name"] = "Nick Cosentino",
 });
 }

 [Benchmark]
 public async ValueTask InsertOneAsyncValueTask_BsonDocument()
 {
 await _collection!.InsertOneAsync(new BsonDocument()
 {
 ["Name"] = "Nick Cosentino",
 });
 }

 [Benchmark]
 public void InsertOne_BsonDocument()
 {
 _collection!.InsertOne(new BsonDocument()
 {
 ["Name"] = "Nick Cosentino",
 });
 }

 [Benchmark]
 public async Task InsertOneAsync_RecordStruct()
 {
 await _collectionRecordStruct!.InsertOneAsync(new RecordStructDto("Nick Cosentino"));
 }

 [Benchmark]
 public async ValueTask InsertOneAsyncValueTask_RecordStruct()
 {
 await _collectionRecordStruct!.InsertOneAsync(new RecordStructDto("Nick Cosentino"));
 }

 [Benchmark]
 public void InsertOne_RecordStruct()
 {
 _collectionRecordStruct!.InsertOne(new RecordStructDto("Nick Cosentino"));
 }

 [Benchmark]
 public async Task InsertOneAsync_RecordClass()
 {
 await _collectionRecordClass!.InsertOneAsync(new RecordClassDto("Nick Cosentino"));
 }

 [Benchmark]
 public async ValueTask InsertOneAsyncValueTask_RecordClass()
 {
 await _collectionRecordClass!.InsertOneAsync(new RecordClassDto("Nick Cosentino"));
 }

 [Benchmark]
 public void InsertOne_RecordClass()
 {
 _collectionRecordClass!.InsertOne(new RecordClassDto("Nick Cosentino"));
 }

 [Benchmark]
 public async Task InsertOneAsync_Struct()
 {
 await _collectionStruct!.InsertOneAsync(new StructDto() { Name = "Nick Cosentino" });
 }

 [Benchmark]
 public async ValueTask InsertOneAsyncValueTask_Struct()
 {
 await _collectionStruct!.InsertOneAsync(new StructDto() { Name = "Nick Cosentino" });
 }

 [Benchmark]
 public void InsertOne_Struct()
 {
 _collectionStruct!.InsertOne(new StructDto() { Name = "Nick Cosentino" });
 }

 [Benchmark]
 public async Task InsertOneAsync_Class()
 {
 await _collectionClass!.InsertOneAsync(new ClassDto() { Name = "Nick Cosentino" });
 }

 [Benchmark]
 public async ValueTask InsertOneAsyncValueTask_Class()
 {
 await _collectionClass!.InsertOneAsync(new ClassDto() { Name = "Nick Cosentino" });
 }

 [Benchmark]
 public void InsertOne_Class()
 {
 _collectionClass!.InsertOne(new ClassDto() { Name = "Nick Cosentino" });
 }

 private record struct RecordStructDto(string Name);

 private record class RecordClassDto(string Name);

 private struct StructDto
 {
 public string Name { get; set; }
 }

 private class ClassDto
 {
 public string Name { get; set; }
 }
} The benchmark code has as much as possible that we’re not interested in exercising pulled out into the GlobalSetup and GlobalCleanup marked methods. Results For C# MongoDB Insert Benchmarks Cue the drumroll! It’s time to look at our MongoDB benchmark results and do a little bit of an analysis: Here are my takeaways from the benchmark data above: All async variations used ~5KB more than the normal versions of the methods that we had to use. There didn’t seem to be any difference in allocated memory for async vs async value task BUT Gen0 and Gen1 didn’t have any value for *some* of the ValueTask benchmarks — However, not for all of them. It almost looks like ValueTask combined with a struct data type for the insert results in Gen0 and Gen1 with no value, but plain old BsonDocument is an exception to this. The fastest and lowest memory footprint seems to be InsertOne_RecordClass, although the InsertOne_BsonDocument is only a few microseconds off from this. Async versions of the benchmarks seem slower than their normal versions across the board, as well. This seems to be very much aligned with some of Twitter's opening thoughts on Async operations! So, some hypotheses proved/disproved: Async *is* overall worse for very fast DB operations ValueTask doesn’t stand out as a consistent performance optimization in these situations For single items, there’s no big difference in the memory footprint we’re seeing between any of these variations of the data types It’ll be a good exercise to follow up with benchmarking, inserting many items into MongoDB from C#. I think we may start to see some of these variations stand out in different ways once we’re working with collections of items — But this is still a hypothesis that needs to be proven! Wrapping Up C# MongoDB Insert Benchmarks This was a simple investigation of insert benchmarks for MongoDB using C#. Overall, there were some surprises for me, but I still think there’s more investigation to be done when we’re working with multiple records at a time. I truly was a little bit surprised to see async be worse across the board since I figured that perhaps any type of IO would mask the performance impact of async overhead. But this was a fun experiment to try out, and there is more to come! If you found this useful and you’re looking for more learning opportunities, consider and check out my ! subscribing to my free weekly software engineering newsletter free videos on YouTube Also published . here

Benchmarks for Inserting Documents Using C# for MongoDB

In this article, I’ll guide you through C# Testcontainers for MongoDB! I love using Testcontainers for local database work when I don’t want to think about managing the overhead of spinning up and tearing down a database, or managing something in the cloud. MongoDB is no exception here as the C# Testcontainers NuGet package that we have for MongoDB is a dream to work with. In this article, I’ll introduce you to the basics of Testcontainers. From there, I’ll demonstrate how simple it is to interact with your newly created database via the C# MongoDB.Driver NuGet package — almost like you wouldn’t even know there was a container spun up! Let’s dive into it. Overview of C# Testcontainers Testcontainers is an awesome NuGet package that can greatly enhance your testing and local development workflow when building .NET applications. It provides a simple and effective way to manage Docker containers, allowing you to spin up lightweight and isolated instances and have them tear down like nothing ever happened! That means with Testcontainers, you no longer have to worry about setting up and tearing down test databases or dealing with complex configuration files. It handles all the details of container management, making this process feel trivial — I sure wouldn’t want to handle this manually. Some of the key benefits of Testcontainers include: Its ability to provide a consistent and reproducible test environment. You can define the exact service version and configuration that you require, ensuring that your test results are reliable and consistent across different environments. The speed and flexibility it brings to your testing and development workflow. By using lightweight Docker containers, you can quickly spin up and tear down test environments on demand. No configuring cloud or shared resources ahead of time… Just on-the-fly awesomeness. Seamless integration with existing testing frameworks and tools in the DotNet ecosystem, such as Xunit, NUnit, and BenchmarkDotNet. This allows you to easily incorporate containerized service instances into your existing test suites without any major modifications. For this article, we’ll be focusing on how we can use C# Testcontainers for MongoDB in particular. Much of the advice covered here will be applicable to other services, especially databases, but we’ll keep our sights set on MongoDB. Setting Up Testcontainers in C# To get started with Testcontainers in C#, you’ll need to install and configure the necessary dependencies in your project. Follow the steps below to set up Testcontainers for MongoDB in C#. 1. Installing Testcontainers Generally, the first step would be to install the Testcontainers NuGet package in your C# project. You would open the NuGet Package Manager Console, and run the following command: Install-Package Testcontainers However, we need the MongoDB flavor of this package (I am using version 3.8.0 at the time of writing) which includes the base Testcontainers package. You can install it via the package manager user interface or with this command: Install-Package Testcontainers.MonogoDB 2. Configuring Testcontainers for MongoDB Once you have installed Testcontainers, you need to configure it for MongoDB. This involves setting up a MongoDB container with the required settings. Here is an example of how you can configure Testcontainers for MongoDB in C#: using Testcontainers.MongoDb;

MongoDbContainer container = new MongoDbBuilder()
 .WithImage("mongo:latest")
 .Build(); In the above example, we create a new instance of the , and specify the MongoDB image and port binding. The method sets the image for the MongoDB container, so using “mongo:latest” will always pull the latest image. MongoDbBuilder WithImage Optionally, we could use port binding to configure which ports we want to be using explicitly: using Testcontainers.MongoDb;

MongoDbContainer container = new MongoDbBuilder()
 .WithImage("mongo:latest")
 .WithPortBinding(1337, 27017)
 .Build(); The code above uses method binds the container’s port 27017 to the host machine’s port 1337. We’ll see in the upcoming sections that unless you have a specific need to bind ports, you can potentially just leave this be. We can get a dynamic connection string, which is very convenient! WithPortBinding 3. Starting and Stopping the MongoDB Container After configuring Testcontainers for MongoDB, you can start and stop the container as needed. Here are code examples for starting and stopping the container: To start the MongoDB container: await container.StartAsync(); To stop the MongoDB container: await _container.StopAsync(); Remember the MongoDbContainer type can be disposed of, so call to clean up when you’re ready. DisposeAsync Using C# Testcontainers for MongoDB in Practice Now that we’ve seen how to spin up and tear down containers, what are we supposed to do with them? We have plenty of options, and truly, the limit is your own imagination (and time, money, and other resources…)! Most people leverage C# Testcontainers for writing integration or . You can match up Testcontainers with xUnit or NUnit, for example, and write tests that interact with a real database. functional tests For me, I needed to go wire C# Testcontainers for MongoDB up to BenchmarkDotNet so I could ! A similar use case where I needed transient local databases, but not necessarily for a test framework. benchmark inserting records into MongoDB Regardless, we need to be able to connect to these MongoDB database containers from C#, and these next subsections will cover that. Connecting to C# Testcontainers for MongoDB with MongoDB.Driver Let’s assume you read the earlier sections and you have the correct MongoDB package installed for Testcontainers. If you skimmed ahead to this section, go back, read it, and get the right package. Otherwise, you’ll be scratching your head about why you can’t find the right dependencies! You’ll also need to get the MongoDB.Driver NuGet package installed. This is what we’re going to be using to establish a connection to the database container that we spun up. I have other articles you can read if you want more explanation on how MongoDB.Driver works in general: MongoDB in C#: Simplified Guide For Inserting Data MongoDB Filtering in C# – Beginner’s Guide For Easy Filters How To Delete Documents From MongoDB In C# – What You Need To Know How To Update MongoDB Documents in C# – What You Need To Know With the right packages ready to go, we can combine the code we saw earlier with some MongoDB driver code to get things glued together: using MongoDB.Bson;
using MongoDB.Driver;

using Testcontainers.MongoDb;

MongoDbContainer container = new MongoDbBuilder()
 .WithImage("mongo:latest")
 .Build();
await container.StartAsync();
string connectionString = container.GetConnectionString();

MongoClient mongoClient = new MongoClient(connectionString);
IMongoDatabase database = mongoClient.GetDatabase("test");
IMongoCollection<BsonDocument> collection = database.GetCollection<BsonDocument>("test"); In the example above, we can call on the newly spun-up MongoDB container from Testcontainers. The awesome part about this is that no matter how you configure your Testcontainer for MongoDB using the builder pattern we see in this code and earlier in the article, will get you what you need to connect. GetConnectionString() GetConnectionString() Because takes in a connection string as a single argument, it’s trivial to get access to the MongoDB database to start working with it right away! MongoClient Performing CRUD Operations on the MongoDB Testcontainer Now that we have our MongoDB Testcontainer set up and we have a connected to it, we can start performing CRUD operations on it. The connection details provided by the container instance made it a breeze to combine these two things, and truly, we can focus on just the MongoDB.Driver method calls for CRUD. MongoClient Here’s an example of performing a simple CRUD operation: // using code from earlier examples...

// Create
await collection.InsertOneAsync(new BsonDocument()
{
 ["Name"] = "Nick Cosentino",
});

// Read
var filterBuilder = Builders<BsonDocument>.Filter;
var filter = filterBuilder.Eq("Name", "Nick Cosentino");
var results = collection.Find(filter);

// Update
var updateBuilder = Builders<BsonDocument>.Update;
var update = updateBuilder.Set("Name", "Dev Leader");
collection.UpdateOne(filter, update);

// Delete
filter = filterBuilder.Eq("Name", "Dev Leader");
collection.DeleteOne(filter); Using the previous code snippets, we were already up and running with our MongoDB container instance and a . As a result, you can see in the code snippet above that we can directly work with our that we obtained from the prior setup. MongoClient IMongoCollection<BsonDocument> This helps illustrate that once you have the MongoDB docker container up and running via Testcontainers, you don’t have to treat things any special way once you connect to it! Wrapping Up C# Testcontainers for MongoDB To conclude, C# Testcontainers for MongoDB is an incredibly simple way to get a transient MongoDB data store up and running for your testing and development. While this article didn’t focus on specific use cases, hopefully, you have a better understanding of how transparent it feels to layer a MongoDB driver on top of Testcontainers to interact with MongoDB. Given that C# Testcontainers takes the headache out of the setup, teardown, and even connection string management, it’s an easy recommendation from me if you want to get up and running with MongoDB for local usage. If you found this useful and you’re looking for more learning opportunities, consider , and check out my ! subscribing to my free weekly software engineering newsletter free videos on YouTube Also published here

Stories, News & Mentions about MONGODB

Mongodb’s Stories on HackerNoon

Mongodb's News & Mentions