Switch from Meteor to GraphQL: Secrets of Hassle-Free Migration Process

The reason I have decided to write this article is that at the moment our team is migrating a complex app from Meteor to GraphQL. Before describing why we are doing this, I want to briefly tell about the concept of this project.

The project we are working on is a work order management app — a complex system that connects on-demand service providers (that offer such services as dumpster rental, sewage treatment, landscape and equipment management, etc.) with companies of different sizes.

The platform helps companies to manage work orders, keep track of all documentation, automate invoice processing and eliminate paperwork. The app consists of 3 products that can be customized to the company needs

• Work order website for companies, where they can create and track work requests, assign suppliers, create reports, manage invoices, etc.
• Work order mobile app for technicians that allows leaving reports about the work done, track their working time, set work availability, etc.
• Chatbot system that automatically processes the requests from companies and creates work orders.

As you can see, this project consists of different applications that deal with a large number of users, requests, and data. One more important aspect of this product is that it has to process all the data in real time. That’s why we have chosen Meteor for building the MVP.

Meteor for MVP. Why we have chosen Meteor for this project

Meteor is a full-stack JavaScript framework that covers all aspects of writing web or mobile applications. It provides ready-to-use instruments that simplify the development process. Another key feature of Meteor is a reactive data flow by default. All the data exists on the client in real time, so changes made in the database immediately populate to all connected clients.

Meteor is exceptionally useful when it comes to MVP development — you can start in minutes and postpone making super complex decisions regarding app architecture until you really need them. As Donald Knuth has said: “Premature optimization is the root of all evil,” so you know what I mean.

Meteor also has great integration with the most popular front-end libraries (React, Angular, Vue), hosting as a service, integration with OAuth (talking about social networks login here) and established zero configuration build system (tell me about web pack config, am I right?).

You can also easily build and publish Android and iOS apps with Meteor (using Cordova), which are not that good as native apps but suit great for MVPs or even production-ready apps for simple products.

Meteor comes handy when you want to build an MVP — you just concentrate on the look, feel and functionality. Moreover, Meteor allows building real-time web apps (like chat, games, collaboration tools, etc.) easily. This is convenient for any application — users quickly become dependant on real-time data in their apps and you can provide it to them in no time.

When have we made a decision to migrate from Meteor to GraphQL?

As our app became larger and more needs for various clients (few web and mobile apps, microservices and third-party applications) appeared on the horizon, we have realized that we need more universal and scalable solution for the backend.

The obvious option is to build REST server using any language. But since we live in 2018, there is something better. Ladies and gentlemen, I present to you GraphQL (well not really I, but mostly Facebook. You can dive into GraphQL particularities in our articles).

In short, GraphQL is a query language that allows you to work with your application’s data-layer in very declarative and rather crisp way. It focuses on the connection between entities (thus ‘graph’). It is implemented in most of the popular languages and has various client-side frameworks. We’ve chosen Apollo GraphQL family of libraries (not framework or library, Carl!)

Here are the main advantages of GraphQL (and Apollo in particular).

Main GraphQL advantages

• Need to provide API endpoint for multiple apps

GraphQL is perfect for being an intermediary for multiple clients as it has libraries for most of the platforms. Even if there is no possibility or desire to use a client library you can just POST some JSON and it works!

• GraphQL encourages incremental adoption

You can keep all your previous investments intact and add new functionality using GraphQL server. That was perfect for our case!

• GraphQL is almost technology agnostic

As there are GraphQL implementations in most of the languages, you can easily switch to another technological stack. The same goes for data sources. You can work with any database or external resource for managing your data. This also facilitates integration with third-party services, which was a requirement we had in mind.

• GraphQL is easy to scale and split into microservices

As each of the resolvers is independent, you can easily host some of your apps as a serverless function or even separate service with different stack and hardware resources.

• Such libraries as Apollo cover a lot of the complicated decisions regarding building API

When writing REST server from the ground you can easily mess up with the architecture and have troubles in making it fly in the future. This particularly relates to the controller level of your application.

• GraphQL takes care of connection between entities in your data

You can have your database normalized at last as all the joins will be covered for you. This has given us some perks that we will cover later. But in general, this is one of the main ideas behind GraphQL and it really rules! Want a more high-level description? Check out this article.

• It has an awesome client-side infrastructure

You can really enjoy using GraphQL on the client. The client app can easily define data it needs in a human-readable way. There is no redundant information transferring, which means less traffic use and faster load time. Moreover, with query hashing, you basically only send data through the wire. Just look into hosting queries and responses on CDN — it rocks!

As an app grows, auto-tests are essential for facilitating changes and improving the codebase. Lack of test coverage will most likely result in longer development time, higher error-proneness and overall frustration. Since there are resolvers in the form of pure functions at the core of GraphQL architecture, testing them is a pure delight! Meteor provides a sophisticated testing framework as well, but it is overcomplicated and obfuscates testing because of a number of technical details.

• Awesome features for developers

They include client dev tools, support of optimistic rendering and subscriptions, automatic documentation with GraphiQL, great support for FP paradigm and types, tools for performance monitoring and optimization.

The process of switching to GraphQL

As GraphQL supports incremental adoption, we began with replacing old functionality and building a new one with GraphQL endpoints. It turned out to be a rather smooth transition with a few exceptions. Let’s see a brief description of a transition for some particular entity collection from Meteor to GraphQL:

• Create a schema and resolvers for queries and mutations in your GraphQL server.
• Cover it all with tests. Optional (no, it’s not). We prefer TDD but it’s up to you.
• Replace fetching data through Meteor with Apollo client.
• Replace Meteor methods with mutations.
• Create and add subscriptions to have Meteor-like reactivity for your data if needed.
• Replace references to the removed Сollection that remained with querying your GraphQL endpoint.
• Remove the code related to the entity from Meteor.

This way you can replace your existing Meteor stack with more flexible GraphQL.

Main challenges and difficulties while migrating from Meteor to GraphQL and how you can solve them

Now let’s talk about serious business. Do you know that feeling when you read some article with ponies and rainbows everywhere, then you start to work on something similar and it turns into vampires and cemeteries? I would like to avoid such an experience.

It’s no secret that we have faced some pretty serious problems while making transitions and it took some time to figure out the best solutions. Let’s dive in more details here.

Problem one. Meteor-like for fun. As I’ve mentioned before, users prone to expect the data to update in real-time at any moment. What was more or less trivial with Meteor, requires a pretty robust setup with Apollo GraphQL. To make it even bitter, the API for subscriptions is not fully developed and stable.

You can choose one of the following solutions:

• Have your subscription to notify the client that data have changed and make it refetch all current views
• Notify client about the changes to the certain data and make it refetch some particular queries
• Implement a full-scale subscription, pass all the necessary data with subscription and update the view.

First two options are probably better than polling but still pretty ugly. Let’s briefly describe the third solution using the following example:

• Imagine you have simple TODO app with TODO itself and a task that relates to this TODO.
• Create a subscription for the TODOs on your server using methods described in the documentation.
• Add simple publication on change of your TODO item. You can put some handler in your mutation resolvers, use some kind of hooks or try new Mongo Change Streams, if you are using MongoDB).
• Few points to note here: sometimes you’ll need to publish changes to related entity on change of other entity. For example, suppose you can update tasks independently of its TODO. In this case, you’ll need to publish this TODO to have related subscription update views regardless of that TODO itself didn’t change.

The trickiest part is client implementation.

For this, you need to:

1. Query for data using ordinary query
2. Subscribe this data changes using exact same field set
3. Close an old subscription and initiate a new one if parameters of the subscription have changed (e.g. you have a table with filters. When the filter is changed, close current subscription and start a new one.)
4. If all goes well Apollo will automatically update stored data and views.
5. You can create a pretty straightforward wrapper around default Apollo helper to perform the operation mentioned above.

As you can see, subscriptions with Apollo are pretty complex compared to Meteor. But GraphQL is in process of constant development and they might be much easier soon. In particular, there is a concept of ‘live queries’ that will make everything above much more convenient. (check out this article for more)

Problem two. ’N’ want’s a friend too. There is a common known problem in computer science known as ‘n + 1’. It occurs when you try to load a list of some items with a link to other items.

For example, you query for a list of movies with their directors. In GraphQL, director for each movie will be loaded separately, increasing load on your database. Fortunately, there is an awesome tool by Facebook called data loader.

While it looks like magic, it is pretty simple and I urge you to dive into its implementation (just around 250 lines of code). I won’t go into details as there are a lot of articles on the web or in the docs. I’m just happy that now you are informed and safe.

Problem three. Find this for me. You’re probably familiar with this common scenario: the user should be able to use full-text search on every field they can see in the table. Now imagine a table where you have data from multiple entities, e.g.: rating, box-office, country of origin.

As far as this is a table of ‘movies’, it consists of data from 4 different entities — Movie, Actor, Rating, Country. If a user wants to search across all of your databases for any of these entities using single field — you are in trouble. The thing is when you search for a Movie by an Actor you’ll need to find the actor, then find the movie with this actor, and return Movie with all related data.

Moreover, you should do this for every possible scenario and do this quickly. If you were in some more traditional environment, you could make some joins with ‘like’ search and return the result. That is when special third-party services come to the rescue. We’ve chosen Elasticsearch.

All you need to do is configure Elastic, load your data to the service and resolve your GraphQL search query using Elastic API. Don’t forget to keep your data updated. And I hope you won’t need real-time data there, will you? Well yeah, it is not that simple. We will cover this in more details in the following articles.

What is nice about this is that with GraphQL you will still have the awesome power of relations between entities. When you have your Elastic movie search result, GraphQL will automatically stitch it together with Actors, Ratings, and Countries!

I want to mention a few more important things to keep in mind when you start your Meteor to GraphQL migration:

• Authentication and Authorization — you’ll need to figure out Meteor auth. mechanisms and replace it with a few lines of code and raw node libraries. You’ll also need to login a user both on the GraphQL and Meteor servers.
• Dig into GraphQL way of thinking before making serious moves. It may take time to do things in optimal way, as it is not common to think in graphs in software development. You usually try to fetch all possible data for a particular request and stitch it according to the user needs. But with GraphQL you need to concentrate on getting only the root a user asked for and let the magic happen.
• Try to model you schema according to existing relations. Do not create queries like movies (actor) {}. Just go with Actor { movies }.
• Do not skip Interfaces and Unions while reading the docs. Those are really powerful instruments which can come in handy on multiple occasions.
• Bear in mind that GraphQL implementations develop really quickly and API you used yesterday may become obsolete tomorrow. There are much better ways of doing stuff and awesome useful instruments emerge every week. So you have to live with it.
• Documentation for some parts of Apollo stack might be incomplete, obsolete or confusing. You’ll need to find some articles or even dive into source code and come up with your own article. Or a couple of articles.

Migrate or not to migrate?

To summarize — GraphQL and Meteor are awesome tools for web app development. Meteor lets you start easily and is perfect for small or medium scale real-time apps. But if you require a lot of flexibility, need to support multiple clients and have a complexly distributed data layer — GraphQL is a way to go.

Originally published at apiko.com on July 10, 2018.