How to Make QA Transparent: Allure TestOps for Quality Assistance

We have already introduced our QA practices and how much we care about the transparency of all processes. But how to ensure this transparency on all development levels?

Today, we have new insights from Mikhail Sidelnikov, the Lead QA in PropellerAds, part of the AdTech Holding. Mikhail shared the best practices for working with the Allure TestOps tool.

Why did the team choose it, what tools did it use before, and when does Allure TestOps become the best solution for quality assistance workflows?

Why is Transparency So Important?

As a business based on top-notch technologies, PropellerAds uses CI/CD approach. It implies constant releases several times a day, so the process just mustn’t be interrupted due to any misunderstandings or mistakes.

How does it work?

• A developer creates a pull request
• The QA team immediately starts testing
• The test results instantly appear in Github, Slack, and Allure TestOps.

It’s a continuous process — everything is automated, and there’s almost no manual work. The result of this process must be a high-quality product — and this is why every step must be transparent.

But how to ensure it?

Mikhail:

— ‘Test is the basis of quality. But it doesn’t like solitude — the test is not an introvert! It needs a family and friends of its kind to bring real value. And, like everyone in real life, a test and its family need a comfortable lodging with a convenient infrastructure and a space for improvement.’

So how did the PropellerAds QA team solve the ‘housing question’ for its tests?

First QA Practices in PropellerADs

At first, PropellerAds didn’t even have a full-fledged QA department: it was organized only in 2015. It took time to automate workflows and build an efficient, inspired team.

As the team grew, the more its demands grew, too! The number of tests and microservices was rapidly increasing — and it was essential to keep them all organized. Otherwise, it would become impossible to work with them, integrate with various CI/CDs — and keep transparent, after all.

Implementing TestRail

The founders of the PropellerAds QA team had an age-long experience of working with TestRail. So, it became the most obvious solution. Besides, it seemed to meet all the needs of the team as follows:

• Required features: case keeping, historicity, test launching
• Pretty acceptable integration with Jira
• Team’s experience of integrating TestRail with CI/CD
• Overall team’s experience of working with TestRail

So, the decision was made.

Issues With TestRail

It all went pretty smoothly at first. Still, after some time, the team felt that TestRail is not the best tool that meets all its needs. What was the problem? The main point was that the team was still growing, and all the processes became much more demanding.

In other words, everything was rapidly scaling:

• The team;
• The number of manual cases
• The number of manual test runs
• The number of releases

Mikhail:

—It became extremely difficult and inconvenient to create cases in TestRail first, and then do it in the code. We wanted to see the whole transparent image in our Test Management System.

Finally, the team started starving for some alternative workflow — and tools. The list of the issues became bigger and bigger, and the most painful points included the following:

• The new projects were poorly integrated
• There was no support for the new cases or steps: if you fixed something in the code, you needed to do the same manually in TestRail
• The transparency and relevance became tending to zero

Overall, there was a threat that the product quality would begin to suffer.

New Demands

Finally, it became pretty obvious that the team needed a new solution. But first, it was necessary to make things clear: what does the team lack and what does it need at this stage? After a discussion, the QA team made up a straightforward list of requirements.

Here it is:

• Automated downloading of the test results. The team wanted to see the tests in the TMS immediately after their launch, without using API
• Integration with Jira. It was about transparency again: having a test status right in the task manager is the best way to make workflows as transparent as possible.
• Simple integration with test codes: any alternations should appear in the TMS without the need of changing anything manually
• Test launches in CI/CD. It would be helpful to do it much quicker — and, in a perfect scenario, to launch them right from the TMS. TestRail has the UI customization feature, but this solution required constant support.

How Allure TestOps Changed the Process

The demands became more than apparent, but they didn’t help to find the right solution. And here the PropellerAds QA team was lucky to meet a reputed developer - Artem Eroshenko, who suddenly gave simple and straightforward advice:

Why use TestRail if there is Allure TestOps?

The first thought was: and what is Allure TestOps?

Allure TestOps is a testing platform that allows you to store and manage test cases. It is based on Allure Reports, very familiar to most team members.

Mikhail:
— So we thought: it’s a fresh and trendy tool — why not try it?

And, a spoiler: it worked out perfectly. The process began to change in the best possible way — and in the shortest time.

So what were the most helpful features of Allure TestOps that boosted the workflow so much?

Test Results

If you have ever tried TestRail, you know that downloading test results is complicated there: it involves much fuss with API.

Allure TestOPS solves the same problem in a very elegant and simple way.

If your test results imply generating an Allure Report, you already have your tests in TestOps. The team has been using Allure for quite a while — and it meant that the TestOps platform kept their tests.

A brief guide: to add any integration, you just need to enter the settings of TestOps, select any product you want, and then - set up it in several clicks! Quick and painless, like here:

Mikhail:

—No hidden rocks: three fields, and you’re done with your integration. It works the same with the other tools: minimum settings. And, our so beloved Dark Mode is here, too!

If you need to integrate Teamcity or other tools of this kind, you can do it with special plugins. It takes two mouse clicks to set the results downloading to TestOps. What is even more helpful, it happens in the runtime. Here is how it looks: elegant and simple again:

After you are ready with the settings, you can launch jobs in your CI/CD. And the process becomes multiple times more transparent immediately after. Allure TestOps allows you to check the existing test coverage, understand how often you launch particular tests, realize which projects are stable and which are not, and even see certain tests.

The overall testing flow in Allure TestOps looks like this:

• You set the TeamCity build configuration once: indicate Allure Project ID and a path to Allure Results in the Build Feature
• You launch the Team City Build Configuration
• You wait for your results in Allure and that’s it!
  

So, after you are done with all settings, you only do two simple steps — no JS, web servers, and API. Everything is set via the UI of the tools you use.

Quality and Status

It turned out that Allure TestOps has a Jira plugin that allows you to see all tests for a particular task. The plugin includes pagination, search filters, and the latest results of these tests.

The settings are also quick and simple — as well as the interface:

Mikhail:

All the results are updated automatically, and you can see all the cases that were created for this particular task. It was exactly what we needed to understand what and how we check. We significantly boost our good old transparency — and ensure a better understanding of the quality of the features we release.

Autotests Code

The PropellerAds QA team has loads of autotests: thanks to them, it can ensure the top quality of the product. Allure TestOps greatly simplifies this essential point. It allows you to create necessary annotations, and each annotation sets a label for a test. In the end, the label will appear in TestOps.

There is a large list of already existing annotations — and not only within Java. Here is an example:

If you lack ready annotations, you can easily add your own one:  @LabelAnnotation(name = “any name”). Almost all TestOps data work via the labels, and these labels are customizable. Thanks to it, you can see various tree maps and layers in the list of all existing tests, filter, and group your cases.

Other Features

Thus, all the issues were solved thanks to TestOps. Besides these solved issues, we also got access to more new features. Here they are:

• Allure allows you to check the test launch history. Mikhail says it was a killer feature: you can quickly check when and where the test was launched and when it started to fail.
• Test plans. Now the team could not just create, but also launch these test plans right from TestOps.
• Convenient dashboards with statistical data. You can create them yourself, or use the ready dashboards.
• Various settings for projects. They include the deletion of old test results you don’t need anymore, setting different test fields for manual and automated cases, and launching tests right from Allure TestOps with various options.
• And many many others.

Mikhail:

— It’s only a small part of everything you get with Allure TestOps. Want something new or found a bug? Create a bug ticket or ask the developers directly in the Telegram channel. It’s really convenient! Or they helpdesk - these guys try to answer as soon as possible - Good service.

To Sum Up: Allure TestOps vs. TestRail

So, was the ‘housing problem’ for our tests solved? Yes, definitely. Constantly developing, intuitive, with instant access to all necessary tools — Allure TestOps was the most proper solution.

But is TestRail so bad, after all?

Mikhail:

TestRail is a good tool — but it better suits different needs than we have.

When it’s better to use TestRail, and when Allure TestOps is your savior?

Overall, Allure TestOps has proved to be much more efficient for ensuring QA transparency —  at every step and level. And that is what we need in PropellerAds.

The lead image for this article was generated by HackerNoon's AI Image Generator via the prompt "an engineer looking at an engine".