Onсe upon a time, we had a complex project enough (ride-sharing and taxi application) with stack Node.js and MongoDB. We have chosen this stack because it was preferable by the customer, good known by our team, and at the same time looks like good a suite for project tasks. Everything was great, the number of users became more than twelve thousand, the number of active drivers was close to three hundred drivers. In one year, the number of rides becomes more than two million. But once we need to create an admin panel to control and monitor all processes (from the business point of view) in the main application. The huge percent of requirements was to have advanced lists of a variety of entities, with bind statistics over them. Because we use , as ODM, first of all, we take a look at its plugins. The most popular of them, that related to pagination are: mongoose mongoose-paginate mongoose-paginate-v2 mongoose-aggregate-paginate mongoose-aggregate-paginate-v2 Another requirement was to have a possibility to choose a specific page on-demand, so the approach to use “ ”-like pagination, that based on cursors was prohibited immediately — its and libraries. previous-next mongoose-paginate-v2 mongoose-aggregate-paginate-v2 The oldest, and probably the simplest in usage is — it uses simple search queries, limit, sort, and the skip operations. I guess it’s a good variant for simple pagination — just install a plugin, add few lines of code to your endpoint, and that’s all — work is done. It even can use “ ” of mongoose, — something that emulates joins from SQL world. Technically it just makes additional queries to the database, that probably not the way you want. Even more, when you just have a little bit more complicated query, with any data transformation, it will be totally unusable. mongoose-paginate populate I know just one way to normally use it in such cases — first create — technically its pre-saved queries, that MongoDB represents as read-only collections. And just then run pagination using mongoose-paginate over this view. Not bad — you will hide complicated queries under view, but we have a better idea of how to solve this problem. MongoDB View MongoDB Aggregation Framework is here! You know, I guess, it was really day for the MongoDB community when Aggregation Framework was released. Probably it allows for most of the queries that you can imagine. So, we think about taking into use mongoose-aggregate-paginate . But the next two things that disappointed us: I mean — what task does it help to solve, that cannot be solved without this plugin, with the same effort. Looks like it just one more additional dependency in your project, because it doesn’t bring any profit, even don’t save your time… What does this plugin need? to making queries. This library makes calls into a database and waits for responses via . First — to fetch query result and second — to calculate the count of total records that query returns, without and stages. It needs this to calculate the number of total pages. Internal codebase, and general approach TWO Promise.all $filter $limit How we can avoid additional queries into the database? The worst thing here that we need to run all aggregation pipeline twice, that can be costly enough in terms of memory and CPU usage. Even more, if collection huge, and documents tend to be few megabytes, it can impact Disc I/O usage, that also a big problem. The good news — Aggregation Framework has a specific stage in its arsenal, that can solve this problem. It’s $facet: Processes multiple aggregation pipelines within a single stage on the same set of input documents. Each sub-pipeline has its field in the output document where its results are stored as an array of documents. . MongoDB documentation about $facet stage Aggregation Pipeline for pagination will have the next shape: { : {
      <outputField1>: [ <stage2>, ... ],
      <outputField2>: [ <stage1>, <stage2>, ... ],
      ...

   }
} $facet , < > stage1 Run all operations, that don’t directly affect on final pagination result, after all possible filters ( stages). There are stages like $project or — that don’t change the number or order of result documents. Try to cut off as many documents as you can at once. $match $lookup Try to make your models as more self-sufficient, to avoid additional . It’s normal to duplicate some data or make pre-computing fields. $lookups If you have a really huge pipeline, that processes many data, your query will probably . In this case, you need to use flag. use more than 100MB allowDiskUse Follow Aggregation Pipelines performance . This advice helps you to make your queries more efficient. optimization guide And technically — you can make dynamic queries on your application code side — depends on conditions you can add, remove or modify specific stages. It can speed up your queries, and moreover, make your code more eloquent. Coz NDA, I cannot show you real database schema and real queries. But let me show you a small example of such pagination. Imagine that you have two collections — and . collection is static enough in thinking of types and amount of fields in different documents.  But is polymorphic, can be changed during time, as a result of business requirements updates. Statistic Drivers Drivers Statistic Also, some drivers could have big statistic documents and history in general. So you cannot make Statistic as subdocument of Driver. So code and MongoDB query will have the next shape: ridesInfoPaginationPipeline = [{ : {
      ...filter, : ,
    }
  },
  { : {
      ...sort, : ,
    }
  },
  { : { : , : , : , : ,
    },
  },
  { : { : , : ,
    },
  },
  { : { : { : [{ : [ , , ]
        }, ]
      }, : , : , : , : ,
    },
  },
  { : { : [{ : }], : [{ : { : }
      }],
    },
  },
  { : },
  { : { : { : [ , skip, { : [limit, ]
        }]
      }, : { : , : { : limit
        }, : { : ((skip / limit) + )
        }, : { : { : [ , limit]
          }
        },
      },
    },
  },
]; executePagination = () => { Statistic.aggregate(ridesInfoPaginationPipeline());
} const ( ) => filter = {}, skip = , limit = , sort = {} 0 10 $match active true $sort createdAt -1 $lookup from 'statistic' localField '_id' foreignField 'driverId' as 'driver' $unwind path '$driver' preserveNullAndEmptyArrays true $project driver $ifNull $concat '$driver.firstName' ' ' '$driver.lastName' 'Technical' entityId 1 message 1 meta 1 createdAt 1 $facet total $count 'createdAt' data $addFields _id '$_id' $unwind '$total' $project data $slice '$data' $ifNull '$total.createdAt' meta total '$total.createdAt' limit $literal page $literal 1 pages $ceil $divide '$total.createdAt' const async return As you see, using and stage we can: Aggregation Framework $facet make data transformation and complex queries; fetch data from multiple collections; get metadata of pagination (total, page, pages)in the one query without additional query execution. Regarding the of such an approach, I guess just one is major — . It includes performance troubleshooting, knowledge of a variety of stages, and data modeling approaches. main drawbacks higher complicity of development and debug process, along with higher entry threshold So, pagination, that based on MongoDB Aggregation Framework, is not pretending to be a silver bullet. But after many attempts and pitfalls — we found that this solution is covered all our cases, with no effects and no high coupling to a specific library.

Fetch

maxvynohradov.github.io

Read My Stories

Too Long; Didn't Read

In your car, at home, or at work — Bosch technology shapes many areas of life.

Advanced and Efficient MongoDB Pagination

Too Long; Didn't Read

Companies Mentioned

Max Vynohradov

About Author

TOPICS

THIS ARTICLE WAS FEATURED IN...

Advanced and Efficient MongoDB Pagination

Too Long; Didn't Read

Companies Mentioned

Max Vynohradov

About Author

TOPICS

THIS ARTICLE WAS FEATURED IN...

RELATED STORIES