How to Solve Allure Report Security Issues

Detailed and visually appealing test reports are crucial for software testers to understand test results and make informed decisions. With meticulous attention to detail, the Social Discovery Group's team crafts visually captivating test reports with the help of Allure Reports - the open-source powerhouse that seemed to have all the answers. Yet, we uncovered a flaw: security. Anyone with a link could peek inside, and its adaptability across subsystems was lacking. In this article, the SDG testing team shares how they managed to resolve Allure report security issues by creating a proper Allure Docker and modifying the storage scheme.

SDG development environment is based on Microsoft products, with Azure DevOps utilized for executing CI/CD processes. Through the CI pipeline, the repository of automated tests is built, followed by categorization via several CD pipelines based on the preferences of testers for test runs.

Let's consider the scheme as it was originally set up:

In this setup, the CD pipeline also serves as a generator for two Allure reports: one for Slack notifications, providing testers with a conveniently readable link specifying stages and test categories, and another for exporting the Allure report to Azure DevOps.

This is achieved through the installation of an extension in Azure DevOps, which enables report generation and subsequent uploading to the $web container for static websites in the Azure storage account. The enabled configuration of anonymous access allows the report to be displayed via a link.

For added convenience, an additional plugin was also used to display the Allure report in the build pipeline.

While this scheme is functional, it has certain drawbacks:

Firstly, this method lacks security as Allure reports are accessible to anyone with the link. Depending on the test, the report may contain confidential information about services, their vulnerabilities, and any other sensitive data that should be restricted from public access. General access is granted due to the enabled "anonymous access" flag in the Azure storage account, which provides anyone with access to the address. Disabling this function would cease external access to the files for unauthorized individuals, but it would also render the report inaccessible on the Azure DevOps page.

Secondly, the method is not universally applicable across subsystems. If we examine the list of jobs in the CD pipeline, we'll see that the native job "azcopy" is used to copy the report to the storage account. This job is only available with agents on Windows, and when using Linux, it returns an error.

Thirdly, there's the inconvenience of viewing test history and trends. The only available history of reports is accessible through Slack notifications, which isn't a convenient method for finding specific tests.

All of the abovementioned factors led to the conclusion that the existing system needs to be modified. Therefore, several solutions to this problem were considered:

1. Creating a method to generate a unified .html file that would combine the entire Allure report project into one document.

2. Utilizing third-party services that offer authentication for viewing the report.

3. Searching for a pre-made image with its own authentication for viewing reports.

Now, let's delve into each of these points in more detail.

As mentioned earlier, disabling the "anonymous access" function in the storage account restricts access to files via the link for unauthorized users. To provide access to specific individuals from outside, Azure itself offers several possible options - these include granting SAS tokens and configuring Access policies. However, due to the nature of generating the Allure report, specifically the index.html file, which references existing JavaScript scripts in the project structure:

When opening the report, the page turns out to be empty because access between files within the blob storage itself is restricted.

To create a single .html file, we discovered an image that, when deployed, facilitated the generation of the file from an entire Allure project using the embedded command "allure-combine ./path/to/allure/generated/report/folder." However, this process requires the agent to have Python installed. Unfortunately, this approach proved ineffective due to the absence of files attached to the results of API tests, which are crucial components for ensuring the proper functioning of testers' work.

It's also worth noting that Azure offers IP whitelisting, which allows the filtering of individuals who require access. However, since the team in need of access does not have a static IP address, this method was not suitable.

Considering that Allure is a popular practice for collecting tests in the development environment, there are paid services available online. For instance, the service https://qameta.io/ provides storage and generation of Allure reports in a user-friendly web interface. However, such services are typically paid, and the required functionality is minimal. Therefore, we considered this implementation option as a last resort.

The choice we settled on was the allure-docker service Docker image (https://github.com/fescobar/allure-docker-service) with the integration capability of the user interface panel (https://github.com/fescobar/allure-docker-service-ui) with authentication. Communication with the service is configured via HTTP/HTTPS, and the image itself supports functionality through API requests.

 Let's highlight some advantages of this solution:

1. Security: Data transmission is encrypted using SSL and TLS mechanisms, and the image provides built-in users "admin" and "viewer" with login credentials that can be modified and provided to users based on their needs.

2. Convenience: The service comes with its own visual interface offering various functionalities for working with Allure reports.

3. Functionality: Thanks to API requests, interesting combinations can be devised with release pipelines on the Azure DevOps side. For instance, generating a report, uploading it, and then clearing the cache within a single stage.

In conclusion, our release pipeline looks like this:

It's worth noting that we didn't move away from the Azure DevOps extension, which allows the uploading of Allure reports, as by that time, we had already transitioned to our own agents and adhered to the idea of having everything work "on our own" machines.

Regarding storage, we experimented with writing to Azure storage accounts, specifically uploading to File Share, Blob, and Blob using the azcopy utility.

The results varied significantly. When using the command for File Share az storage file upload-batch, the write speed for our test was over an hour:

When using Blob Storage with the az storage blob upload-batch command, the performance improved sixfold, taking approximately 11 minutes:

The fastest result was by azcopy tool:

While using the built-in integration of azcopy in the release pipeline, it is incompatible with Linux machines. However, by using the Azure CLI job and installing this utility on the agent, the command can be confidently employed:

azcopy copy 'SOURCE'  'DESTINATION' --recursive=true

Thanks to the discovery of the Allure Docker image from the depths of the net, we were able to modify the storage and display scheme of reports. Data access is now password-protected, and report creation and processing remain fast. Moreover, from the users' (testers') perspective, the changes are minimal. The solution operates independently from other programs, thanks to a separate agent in the cluster, and cannot affect other development processes. The cost-effectiveness of this solution is comparable to that of a separate self-hosted agent ($15, provided it is isolated), which is much cheaper than existing alternatives. For example, the previously considered qameta.io, where the price for a single user is $30/month.

Written by the DevOps team of Social Discovery Group.