How To Orchestrate Event-Driven Microservices

In the previous post in this series, we learned about the pitfalls of tightly coupled direct call microservices and how to avoid them by implementing the choreographed event-driven microservices pattern. This article will expand on a different event-driven architecture pattern. But first, a quick recap on some key concepts before we get cracking on achieving loosely coupled microservices utopia!

We established that the direct call microservice pattern leads to a tight web of coupled services which in reality leads to a distributed monolith. One way to avoid building a distributed monolith is to instead implement event-driven microservices. The golden rule of Event-Driven microservices is that all communication is asynchronous. Instead of API calls, microservices publish records of their doings, also known as events. An Event is a record of business action and must contain all information relevant to that action. Events are published to messaging infrastructure (think Kafka, RabbitMQ), and it is left to consuming microservices to figure out how to operate on them. By removing this tight coupling between services, it's possible to truly reap the benefits offered by the microservices architecture pattern.

The choreography pattern that we explored earlier offers many of the benefits of this style of architecture, but it has its downsides too. It is difficult to add intermediary steps in a business flow, and the monitoring and observability can get complicated (although not more so than in the direct call pattern). The choreography pattern is also not suitable for business workflows that are complex and that involve a large number of services. If you have many microservices that implement complex, often changing business flows, you may be better off implementing the orchestration pattern.

What is the Orchestration Pattern?

As opposed to the choreography pattern where services emit events to a stream; and other services consume from the stream, the orchestration pattern involves a central orchestration service (or services). This orchestration service contains the entire business workflow logic, and issues commands to and awaits responses from worker microservices. Think of this as an orchestra where a central Conductor is responsible for keeping the orchestra in time and coordinating all the various players to produce one cohesive musical piece. Netflix liked this analogy so much that they even named their orchestration engine "Conductor".

Let's look at the same simplified business logic flow that we modeled with the choreography pattern

• An order needs to be created.
• An email with the details of the order needs to be sent to the customer.
• Inventory needs to be decreased.
• A hold needs to be placed on the customer's credit card.

The business flow looks something like this:

The actions related to certain domains may still be implemented by the respective microservices, i.e., the Orders Service handles orders-related actions, Inventory Service does the same for inventory, Comms service handles the email communications, and Payment service handles payment transactions. However, there are two key differences:

1. The business workflow is defined by implicit relationships between the services in the choreography pattern, while the workflow is explicitly defined on the orchestrator service in the orchestrator pattern.
2. Instead of services triggering actions on each other via events, the orchestration service triggers actions on the worker microservices via events and awaits their response.

How do you model the above flow with an Orchestrated Event-Driven Architecture?

As mentioned above, the business workflow logic is defined on the Orchestration Service, which then sends commands to the worker microservices for the specific actions to be taken in each step of the workflow. The Orchestration Service also stores some materialized state which gets updated based on the response from the worker services. Each worker microservice is, however, responsible for implementing its own error handling, retries, and failure management. The Orchestration Service is agnostic to the inner workings of the worker microservices.

In the above example,

1. The Orchestration Service receives some "trigger" that starts the workflow.
2. As part of the first step in the workflow, the Orchestration Service produces an event on Stream 1, which is consumed by the Orders Service. The Orders Service creates the order, updates its internal database, and then writes an event to the Response stream. This event is then read by the Orchestration Service, which updates its materialized state and determines the next step in the workflow to execute based on the state.
3. The next steps in the workflow follow a similar pattern - the Orchestration Service executes a command on a stream which is then read by the worker service in question. The worker services execute some logic and return a response on a specific response stream. Once the Orchestration Service receives the response, it updates some internal state and determines the next step in the workflow to execute.

What are the benefits of Orchestration?

Like the choreography pattern, the orchestration pattern offers many of the benefits of loosely coupled architecture. Microservices can be scaled up and down independently, cascading failures are avoided, and isolated development is possible. In addition, this style also offers the following advantages over choreography:

Better visibility and monitoring. Since the entire business workflow is defined in the orchestration layer, this pattern offers great visibility into the progression of the flow. By visualizing the workflow, one can easily tell how far the workflow has progressed or where an error has occurred. Similarly, monitoring and observability for a workflow can also be consolidated to the orchestration layer.

Modifying the business workflow is easier as it is explicitly defined in one place as opposed to the choreography pattern where adding to or modifying the workflow means changes to multiple microservices. The business workflow is modified within the orchestration layer, and new steps can be added fairly easily.

For example, we want to add a new step to update accounts before we decrease the inventory. In the orchestration process, this change is made directly to the business workflow defined in the orchestration layer, and a new stream is created for the Orchestration Service to communicate with the Accounts Service. The Accounts Service then responds on a new response stream created for this purpose.

Transactional processing and rollbacks are possible with this model. Since the workflow is defined in one place, this model can be extended to add logic to revert the transaction from any point in the workflow. Transactions can be rolled back by reversing the workflow logic and providing each workflow logic with a compensating or reversing action. Again, the Orchestration Service should only be responsible for issuing the rollback command. The worker microservice is fully responsible for ensuring that its state is consistent after a rollback.

What are some downsides of orchestration?

One of the main downsides of the orchestration pattern is that the Orchestration Service becomes a single point of failure for the business logic. If the Orchestration Service is down, any business flow defined on it will come to a standstill until the issue is resolved.

Another downside is that these systems can be fairly complex and require additional effort to build and maintain. They also add overhead to the workflows themselves as there is additional i/o and the Orchestration Service maintains and updates the internal state which could increase the end-to-end processing time of a business workflow.

In many cases, though, the overhead cost of maintaining an additional orchestration layer is offset by the ease of defining and modifying business workflows as well as the clarity and structure offered by the orchestration pattern.

In short...

The orchestration pattern is a great choice over the choreography pattern when there are a large number of microservices involved. It is also much more suited to complex business workflows as adding, removing, and modifying steps is much easier. The downside of the orchestration pattern is that it requires the maintenance of an orchestration platform that could be complex and could become a single point of failure. However, the clarity, structure, good visibility as well as ease of operations and troubleshooting often offset the additional complexity of the orchestration layer and make it a great choice for complex and changing business flows executed across a large number of microservices.

Thank you to the good folks at excalidraw.com and the libraries by @Youri Tjang and @Kaligule for the drawing tools!