How to Build Your First Allure Report From Scratch

is an open-source multi-language test reporting tool. It builds a detailed representation of what has been tested end extracts maximum of from the everyday tests execution. Allure Report In this guide, we’ll take a journey through the main steps to creating your first Allure report and discover all the fancy features that it brings to routine automated testing reports. As far as Allure Report has various integrations with various testing frameworks on different programming languages, there is a chance that some steps will vary for each reader, so feel free to jump into the for details. official documentation page Installing Allure Report As always, the first step is to install Allure library. The exact steps vary depending on your OS: Linux For debian-based repositories, a PPA is provided: sudo apt-add-repository ppa:qameta/allure sudo apt-get update sudo apt-get install allure Mac OS X For Maс OS, automated installation is available via Homebrew brew install allure Windows For Windows, Allure is available from the commandline-installer. Scoop To install Allure, download and install Scoop, and then execute in the Powershell: scoop install allure Manual Installation Download the latest version as a zip archive from . GitHub releases Unpack the archive to the allure-commandline directory. Navigate to directory. bin Use for Windows or for Unix platforms. allure.bat allure Add to system PATH. allure Allure CLI requires Java Runtime Environment to be installed. Plugging Allure Into The Code Next step is to provide all the necessary dependencies within the configuration file, so the build tool would be able to use Allure. Each framework and build tool has its own configuration settings, so the best way to get a clue on adding dependencies is to look for an example at or get an . the documentation page example at GitHub Good news, everyone! Qameta Software, the Allure Report maintainer, is developing a GUI-based template projects generator. The templates will be standalone examples with a number of mock tests and all the infrastructure preset! Stay tuned and subscribe to our blog to get your hands on the tool among the first. Adding Annotations to The Allure Report After plugging the Allure Report into the codebase, we can run it. However, the report will be empty. In order to provide all the necessary data to Allure, we need to annotate the tests. There are several types of annotations: Descriptive annotations that provide as much information about the test case and its context Step annotation, the one that allows Allure to build nice and detailed test scenarios Parameterizing annotations that allow you to specify Allure Report Descriptive Annotations @Epic, @Feature, @Story A set of annotations designed to make test-case tree grouping more flexible and informative. The annotations follow Agile approach for tasks definition. These annotations may be implemented on the class or on the method level. Epic defines the highest-level task that will be decomposed into features. Features will group specific stories, providing an easily-readable structure. As story is the lowest part of epic-feature-story hierarchy, the class-level story adds data to all methods of the class. @Description An annotation that provides a detailed description of a test method/class to be displayed in the Allure Report. @Owner Simple annotation to highlight a person behind the exact test case, so everyone would know whom to ask for a fix in case of a broken/failed test. Quite useful for large teams. @Severity In Allure, any @Test can be defined with a @Severity annotation with any of these values like BLOCKER, CRITICAL, NORMAL, MINOR, TRIVIAL. The severity level will be displayed in the Report, so the tester understands the severity level of a test if Failed. Sample Tests You may find the example of all these annotation usage in Java code, annotations for any other technology will look quite similar: public class AllureExampleTest { @Test @Epic("Sign In flow") @Feature("Login form") @Story("User enters wrong password") @Owner("Nicola Tesla") @Severity(SeverityLevel.BLOCKER) @Description("Test that verifies a user cannot enter the page without logging in") public void annotationDescriptionTest() { } /** * JavaDoc description */ @Test @Description(useJavaDoc = true) public void javadocDescriptionTest() { } } Allure Report Step Annotation Detailed reporting with steps is one of the features people love about Allure Report. It is the @Step annotation that makes it possible by providing a human-readable description of any action within a test. Steps can be used in various testing scenarios. They can: be parametrized, make checks, have nested steps, and create attachments. Each step has a name. In order to define steps in code, each method should have a annotation with String description; otherwise the step name is equal to the annotated method name. @Step Note that steps’ mechanics were revised and now it supports smart fields analysis. In Allure 1, users had to specify indexes to refer to which args’ values they wanted to inject into a step. Allure 2 uses a reflection-based approach, which provides deep fields’ extraction by their names. In code, the Step annotation will look like this: package io.qameta.allure.examples.junit5; import io.qameta.allure.Allure; import io.qameta.allure.Step; import org.junit.jupiter.api.Test; public class AllureStepTest { private static final String GLOBAL_PARAMETER = "global value"; @Test public void annotatedStepTest() { annotatedStep("local value"); } @Test public void lambdaStepTest() { final String localParameter = "parameter value"; Allure.step(String.format("Parent lambda step with parameter [%s]", localParameter), (step) -> { step.parameter("parameter", localParameter); Allure.step(String.format("Nested lambda step with global parameter [%s]", GLOBAL_PARAMETER)); }); } @Step("Parent annotated step with parameter [{parameter}]") public void annotatedStep(final String parameter) { nestedAnnotatedStep(); } @Step("Nested annotated step with global parameter [{this.GLOBAL_PARAMETER}]") public void nestedAnnotatedStep() { } Allure Report Parameters Annotations Attachment The annotation allows attaching a String or Byte array to the report. The annotation is used to attach a screenshot or a failure stack trace to the result. Link / Links Well, that’s simple. If you need to add a link to the test, be it a reference or It takes a number of parameters: : link text name : an actual link url : type of link type : similar to name value Muted An annotation that excludes a test from a report. TmsLink A way to link a result with a TMS object, if you use any. Allows entering of just test case id that will be added to the pre-configured (via ) URL. The annotation takes a String value, which is the link to the management system. For example, if our test case on TMS link is , then we can use as value. allure.link.tms.pattern https://tms.yourcompany.com/browse/tc-12 tc-12 Allure Report Launch Local A great way to get started with Allure Report is to run it locally. Local execution does not provide execution and results history, as well as the trend graphs. The best way to give Allure Report a try is to open with an IDE. Check all the dependencies and build the project with Gradle. an example After that, we need to run the tests with the command. When the tests are executed, Gradle will store the results in the target directory. ./gradlew test Let’s take the data and build a report! With the command, we start the Allure Report instance and build a local web-report which automatically opens as a page. allure serve /path/to/the/project/allure-example/build/allure-results CI (Jenkins, TeamCity and Bamboo) Allure Report has several neat integrations with different CI-systems. Each system setup has its specificities, so we won’t cover them in this post. Follow the Documentation page for steps to create a report with a CI-System. Allure Report Features Parameterized tests Allure knows how to work with parameterized automated tests. Let’s take a JUnit test as an example. First, let’s create a test class with parameterized tests of the following kind: @Layer("rest") @Owner("baev") @Feature("Issues") public class IssuesRestTest { private static final String OWNER = "allure-framework"; private static final String REPO = "allure2"; private final RestSteps steps = new RestSteps(); @TM4J("AE-T1") @Story("Create new issue") @Microservice("Billing") @Tags({@Tag("api"), @Tag("smoke")}) **@ParameterizedTest(name = "Create issue via api")** @ValueSource(strings = {"First Note", "Second Note"}) public void shouldCreateUserNote(String title) { parameter("owner", OWNER); parameter("repo", REPO); parameter("title", title); steps.createIssueWithTitle(OWNER, REPO, title); steps.shouldSeeIssueWithTitle(OWNER, REPO, title); } As you can see, after execution, Allure provides the parameterized test run results as a set of tests. If any of the tests fails, Allure provides detailed information about that particular case. Categories Categories is one of the most time-saving features of Allure Report. It provides simple automation for fail resolution. There are two categories of defects by default: Product defects (failed tests) Test defects (broken tests) It is fully customizable via simple json configuration. To create custom defects classification add file to directory before report generation. categories.json allure-results Open json template [ { "name": "Ignored tests", "matchedStatuses": ["skipped"] }, { "name": "Infrastructure problems", "matchedStatuses": ["broken", "failed"], "messageRegex": ".*bye-bye.*" }, { "name": "Outdated tests", "matchedStatuses": ["broken"], "traceRegex": ".*FileNotFoundException.*" }, { "name": "Product defects", "matchedStatuses": ["failed"] }, { "name": "Test defects", "matchedStatuses": ["broken"] } ] The json includes the following data: (mandatory) category name (optional) list of suitable test statuses. Default [“failed”, “broken”, “passed”, “skipped”, “unknown”] (optional) regex pattern to check test error message. Default “. “ (optional) regex pattern to check stack trace. Default “.” Test result falls into the category if its status is in the list and both error message and stack trace match the pattern. In case of using or plugins, file can be stored in test resources directory. allure-maven allure-gradle categories.json Retries Retires are executions of the same test cases (signature is calculated based on test method name and parameters as well) within one test suite execution, e.g. when we are using TestNG IRetryAnalyzer or JUnit retry Rules. Not supported for local runs. Test History Allure 2 supports history for tests in the report. At each report generation during the build, Jenkins, Allure Plugin will try to access working directory of the previous build and copy contents of folder to the current report. allure-report\\history At the moment history entry for test case stores information about up to 5 previous results. Not supported for local runs. Dashboards Report Structure and Overview The default page would be the ‘Overview’ page with dashboards and widgets. The page has several default widgets representing basic characteristics of your project and test environment: Statistics – overall report statistics. Launches – if this report represents several test launches, statistics per launch will be shown here. Behaviors – information on results aggregated according to stories and features. Executors – information on test executors that were used to run the tests. History Trend – if tests accumulate some historical data, it’s trend will be calculated and shown on the graph. Environment – information on test environment. Home page widgets are draggable and configurable. Also, Allure supports it’s own plugin system, so quite different widget layouts are possible. Categories The page shows total defects. If the test is an assertion failure, it reports as ‘Product defect’ and if a test has failed because of some exception, it shows the report as ‘Test defect’. Suites, Behaviors and Packages Three tabs that show the test case tree built on various Suites display test cases based on the suite executed. Behaviors tab builds the tree based on stories and features annotations. Packages tab shows the test cases grouped by package names. Graphs The tab for testing results visualization with charts. The default configurations provide: General execution results pie chart. Test Case execution duration and duration trend. Nice feature to investigate which tests take more time and need optimization. Retries and Categories trends show Timeline The view displays timeline of executed tests, quite a self-explaining graph. Learn more about Allure tools Qameta Software focuses on developing amazing tools that help software testers. Learn more about , lightweight automation reporting tool, and , the all-in-one DevOps-ready testing platform. Allure Framework Allure TestOps Also published . here