Combining MongoDB Collections

Recently, I had been working with a database I had created several months before. For that particular project, I had created a separate database collection for each group of items. At the time, I had a collection of books and articles, sorted by topic, and for some reason, had put all books and articles in separate collections based on topic. Then, I created endpoints for each topic. That way, I can switch out the url collection for the topic the user searched for, like this:

In this case, the user searched for “how to put on pants”. But having a collection with the topic of “how to put on pants”, I now can fetch just those 4 results, instead of fetching the entire library. Another component I used to make this work was a word map object. I won’t get into that here, but if you’re interested, here’s where you can check that out: https://hackernoon.com/creating-more-accurate-search-results-for-small-sites-436e64da79b6

Although this works great, I recently wanted to be able to search all of the collections at once, which wasn’t easy with so many separate collections. Basically I wanted to combine all the collections into one single collection.

I started off by doing a fetch which listed all of the collections in the database. I’m using mlab, and the way to do that if you’re using mlab is as follows:

To get the collections in the specified database:

GET /databases/{database}/collectionsExample:https://api.mlab.com/api/1/databases/my-db/collections?apiKey=myAPIKey

So your code in implementation would look something like this:

fetch('https://api.mlab.com/api/1/databases/my-db/collections?apiKey=myAPIKey').then(function(response) {if (response.status != 200) {window.alert("Oopsie daisy");return ;}response.json().then(function(data) {let api = data;

If you console.log your api at this point, you’ll have an array of each and every one of your collection names as strings. Perfect. Now I’m going to do a for loop, and loop through those collection names, and do basically the same thing I did way up at the top, fetching in each loop iteration, and replacing the fetch collection name with the current api array item.

let arr = [];for (var i = 0; i < api.length; i++) {fetch('https://api.mlab.com/api/1/databases/my-db/collections/api[i]?apiKey=myAPIKey').then(function(response) {if (response.status != 200) {window.alert("dag nabbit");return ;}response.json().then(function(data) {let test = data;for (var k = 0; k < test.length; k++) {arr.push(test[k])}});});}

Here, we have our databases name in the fetch url (my-db) and in each loop, our collections name changes (api[i]). We fetch the data there and save the data as “test” . then, we have a nested loop, where we loop through test, and push each array item into our array called “arr”.

Now, if we console.log “arr” we should have a consolidated array of every single one of our collections in one place, instead of in multiple collections.

Next, we’ll need to reformat all the data into a new array of JSON objects.

setTimeout(function(){for (var i = 0; i < arr.length; i++) {let text = arr[i].article_text;let text2 = text.replace(/"/g, "");let text3 = text2.replace(/'/g, "");arr[i] = `<p>{"articleAuthor": "${arr[i].article_author}","articleAuthorTitle": "${arr[i].article_author_title}","articleName": "${arr[i].article_name}","articleDate": "${arr[i].article_date}","articleText": "${text3}"}</p>`;document.getElementById('verseText').innerHTML += arr[i];}}, 50000);

I used a set timeout function here, to allow all of the fetches in the previous step to complete. Otherwise, I may begin building the JSON data before the fetches complete. I loop through the data contained in the array “arr” and slap it in a paragraph with key:value pairs inside curly braces, then render each object in the DOM. Then I just copy and pasted the output into a json file. As long as there are no errors, then I should be good to go. With mlabs at this point I can just save the file and then export it to an mlabs collection using the command line prompt:

mongoexport -h ds#####.mlab.com:#### -d <database name> -c <collection name> -u <user> -p <password> -o filename.json

And we’re done. Now, I can access all of my data from a separate database, but I haven’t altered the original structure with separate collections, in case I still want to use that format later.

Now, before you comment and say how dumb this method is, don’t worry, I know. But, if you’re like me, and are more comfortable with the front end, this is one possible way to do this. I’m not saying it’s the best, but it’s one way, and it works.