How to Combine a Workflow Engine with a Low-code Platform for Automated Task Execution

Introduction

I recently discovered the open-source workflow engine Camunda after one of our customers mentioned using it with our low-code product, Linx. After looking through Camunda's documentation, I realized that it could be a perfect partner for Linx.

Workflow (or Business Process Management) software generally covers three areas:

• Workflow engine. The engine keeps state and decides what must happen next.
• User task management. The user interface and logic that governs task allocation to users and how they interact with those tasks.
• Automated task execution. The mechanism to interact with other technologies. These may be connectors to APIs or hooks into one or more programming languages.

Camunda covers all these areas and has a comprehensive REST API. You can run it as a standalone service and hook up a custom user task interface and automated task execution to the API.

Automated tasks are usually programmed in a language the workflow engine supports. In Camunda's case, it's Java or the language of your choice if you use the REST API. Another way to develop automated tasks is to use a low-code tool - this is where I thought the two would work well together.

Hypotheses

In theory, using a workflow engine to orchestrate business processes built with a low-code tool should give us the best of both worlds.

We design the workflow with a BPMN modeling tool and create the task automation using low-code. When doing the modeling, we concentrate on the business problem, and when implementing the automated tasks, we focus on the technical aspects but without having to code.

The separation of the orchestration (workflow engine) and task automation (low-code) has some other advantages:

• Maintenance. We can make changes on one without impacting the other one.

• Flexibility. We can mix execution technologies. For example, use low-code where we can and code where we have to. Or have multiple, even different, workflow engines. Whatever works for the scenario.

• Performance. We can scale the area that requires it. For example, if the workflow engine struggles, we scale that, and if any of the execution tasks need resources, we scale that.

• Testing. We can test process logic and task execution separately.

  
  

And some downsides:

• More than one technology to maintain.
• Complexity. It might not always be obvious how the different pieces fit together, making it more challenging to maintain.

Test

Lots of our customers still use files as part of their system integration flows. Files from System A are used to update System B. There are usually tens to hundreds of formats, all with different rules to apply. A common process is to import the file into a staging database, enrich the data, validate the data, and then update the target.

So to test the idea of using Linx with Camunda, I decided to build a generic file import process. We add new file imports by adding the task automation in Linx. The workflow itself stays the same.

The generic file import process looks like this:

All the Service Tasks (those with the gears) are also External Tasks. They are executed outside of Camunda, in this case by Linx.

My initial architecture had Camunda calling a Microservice hosted by Linx to execute the tasks.

But after figuring out that I would have to write code and do some configuration of my Camunda instance to be able to call external APIs, I decided to try a different tact and ended up with this:

Linx polls Camunda for work and executes it instead of Camunda calling into Linx.

The process of building it was easier than expected.

• Design the workflow with Camunda Modeler.
• Create a function in Linx for each Service Task in Camunda. These functions should be idempotent as they might be called more than once for the same workflow instance if something goes wrong between Camunda and Linx.
• Create a Timer Service in Linx to poll Camunda for tasks.
• Add a function in Linx to match the Task Id with its related Linx function.

Result

I quite like how this turned out. Combining a proper workflow engine with a developer-friendly low-code tool gives us the best of both worlds. We get an easy way to build the business process, bullet-proof state management, and a quick way to implement all the automation and integration pieces.

Resources

The project on Github

Linx documentation

Camunda documentation