paint-brush
How to Erase Expired Docs Automatically with MongoDB (TTL index)by@lironer
23,049 reads
23,049 reads

How to Erase Expired Docs Automatically with MongoDB (TTL index)

by lironOctober 28th, 2020
Read on Terminal Reader
Read this story w/o Javascript

Too Long; Didn't Read

How to Erase Expired Docs Automatically with MongoDB (TTL index) I needed to delete some documents that I saved in MongoDB after some time. My wife told me that MongoDB already has that mechanism built-in. A background thread reads the values in the index and removes expired documents from the collection (usually every minute) The indexed field must be BSON date type or an array of BSON dates. If a document does not contain the indexed field, the document will not expire.

Company Mentioned

Mention Thumbnail
featured image - How to Erase Expired Docs Automatically with MongoDB (TTL index)
liron HackerNoon profile picture

Recently I needed to delete some documents that I saved in MongoDB after some time. I can think of a few examples of why we would want to delete data after some time:

  • logs / events
  • api keys / access tokens
  • non-active users
  • etc

We could do it by running a cronjob that deletes the data, delete the data every time we insert new data or any other solution.

Luckily for me, my wife told me that MongoDB already has that mechanism built-in.

TTL index

TTL (Time-To-Live) indexes are special single-field indexes that MongoDB can use to automatically remove documents from a collection after a certain amount of time.

A background thread in MongoDB reads the values in the index and removes expired documents from the collection (usually every minute).

For example, to create a TTL index on the

lastModifiedDate
field of the
eventlog
collection, use the following operation in the mongo shell:

db.eventlog.createIndex( { "lastModifiedDate": 1 }, { expireAfterSeconds: 3600 } )
  • The indexed field must be BSON date type or an array of BSON dates
  • If the indexed field in a document is not a date or an array that holds a date value(s), the document will not expire.
  • If a document does not contain the indexed field, the document will not expire.

Conditional delete

As of MongoDB 3.2, a collection can be partially indexed using a specified filter expression,

partialFilterExpression
. TTL index can also be used with partial indexes.

For example:

Delete documents that created 1 hour ago if

state
equals to
TMP

db.eventlog.createIndex(
  { created_at: 1 },
  { expireAfterSeconds: 3600, partialFilterExpression: { state: 'TMP' } }
);

Delete documents that created 1 day ago if

count
is lower than
5

db.eventlog.createIndex(
  { created_at: 1 },
  { expireAfterSeconds: 86400, partialFilterExpression: { count: { $lt: 5 } } }
);

Additional info on partial indexes

Real-world example

Recently I finished developing BundleMon, which is a free tool that helps you monitor your apps bundle size.

One of the components in BundleMon is a service that saves historic reports in order to compare bundle size between branches. So when you open a PR, BundleMon saves a record with the current bundle size report.

No need to save the report for more than 30 days, so I just added a TTL index:

db.reports.createIndex(
  { creationDate: 1 },
  { expireAfterSeconds: 2592000, partialFilterExpression: { prNumber: { $exists: true } } }
);

Also published at https://dev.to/lironer/delete-expired-documents-automatically-with-mongodb-ttl-index-l44