How to Use BitBucket Pipelines and Xray Test Management to Test Automation of Jira Cloud Workflows

In this post I want to show you how you can use your Bitbucket pipelines together with Jira Cloud. Furthermore I want to also introduce you into Xray Test Management for Jira Cloud a really awesome, cutting edge Test Management Suite for Jira which I use a lot.

You might have read my last post about Deploy a React App to Heroku with Bitbucket pipelines — if not you’ll have the chance later on, because I do refer to some configuration.

Initial Situation

In most enterprises we don’t want to change workflows in production
environment directly. Instead we have a test instance or project where
we build stuff first and then present it to the business.

In this article we’ll assume we have the following simple workflow and a rule for transition issues from “Backlog” to “In Progress”. This rule checks if the user is in the Project Role of Administrator (we’ll assume the user is 😉🙄).

Now that we have implemented the Jira workflow the way the business needs it, we need to test the workflow.

Sure, you could say, let’s do it manually because it’s a quick test. But in
most cases we do have more complex scenarios, don’t we?

I want to show you, how you can test your Workflow automatically with a small Pytest app (Python) running on Bitbucket Pipelines.

In addition to that I want to introduce the app Xray Test Management for Jira Cloud — a cutting edge Test Management Suite for Jira. There we will write a Test Plan and trigger the Bitbucket Pipeline.

Disclaimer

For simplicity’s sake, we will omit appropriate authentication or code formatting. What we will do here will work, but it should not be used in any production environment.

Prerequisites

To get started we need to prepare some stuff first to make sure that
everything is working fine. What we need for this tutorial is:

1. Python3 installed on your local machine – see documentation
2. Python IDE installed – in my case this is PyCharm from Jetbrains
3. Jira API token generated – see documentation
4. Xray API token generated – see documentation
5. Bitbucket App Password generated – see documentation

Let's Get Started!

Step 1 — Create a Python app:

Yes, this time round we’ll build a small Python app which does nothing more than transition an Jira issue from one status to another. Therefore, you need to:

• Open up PyCharm
• Create a new Python Project
• Make sure you use Pipenv as your Python environment

After we’ve created our Python app we need to install at least two python packages. Head over to:

• Settings → Project:<your-app-name> → Project Interpreter
• Install jira-3.0a2 and pytest via the little + sign on the right side

In the project root create a folder with the name tests and within the folder create a file called test_transition.py. The file will look pretty simple and we do manage all stuff e.g. authentication, within this file.

from jira import JIRA, JIRAError

jira = JIRA(
    options={"server": "https://<domain>.atlassian.net"},
    basic_auth=("<E-MAILADDRESS>", "<API-TOKEN>"),
)

At first we need to load the Jira library for authentication and using all
the Jira methods we need to update and transition issues.

After we’ve imported all we need, we can authenticate to our Jira Cloud instance.

It is important that you use the API token mentioned above in the prerequisites. Jira Cloud doesn’t allow you to use basic authentication with username and password anymore.

Next we add the test itself directly below the authentication. The steps are:

• Get the issue(s)
• Execute the transition — at least try
• If correct assert the new issue status (In Progress)
• If not correct assert the current issue status (Backlog)

def test_in_progress():
    issue = jira.issue("<ISSUE-KEY>")

    try:
        jira.transition_issue(issue, "In Progress")
    except JIRAError:
        assert issue.fields.status.name == "Backlog"
        return
    assert issue.fields.status.name == "In Progress"

At this point we already can test our Python app. To do that we need to create a Run Configuration:

1. Go to Run → Edit configurations
2. Click on + symbol → choose pytest (absolute path to your test file)
3. Add the Script Path and Working Directory (absolute path of tests folder)

After that we can right-click our tests folder and choose pytest in tests. Our result should look something like this:

Step 2 — Upload the app to Bitbucket:

For this please read my previous post where I explain in detail how you can setup your code repository and how to enable pipelines for it.

After you’ve setup your Bitbucket repository, open a terminal, make sure you are in the root directory of your Python app, and type the following
lines:

git init
git add .
git commit -m "initial commit"
git remote add origin https://[email protected]/username/name-of-your-repository.git
git push -u origin master

When you refresh the browser page, you should see all the files and folders of the app you’ve just commited to the git repository.

Now that we have everything up and running, add the following bitbucket-pipeline.yml:

image: python:3.8.3
pipelines:
  default:
    - step:
        caches:
          - pip
        name: Run tests
        script:
          -  |
             pip install pipenv
             pip --version
             pipenv install
             echo "Test my amazing script"
             pipenv run python3 -m pytest --junitxml=test-result.xml test_transition.py
             export token=$(curl -H "Content-Type: application/json" -X POST --data "{ \"client_id\": \"$client_id\",\"client_secret\": \"$client_secret\" }" https://xray.cloud.xpand-it.com/api/v1/authenticate| tr -d '"')
             curl -H "Content-Type: text/xml" -H "Authorization: Bearer $token" --data @test-result.xml  "https://xray.cloud.xpand-it.com/api/v1/import/execution/junit?projectKey=TP&testPlanKey=<ISSUE-KEY>"
             echo "done"

Puuhh! 😵 Let's go through this step by step:

• We install pipenv because it takes care of everything you need to work with your app and it is just awesome!
• We install all libraries added to our Project and therefore to the Pipfile with pipenv install
• We run our test(s) with the special parameter — junitxml=<filename>.xml (important use double dashes infront of junitxml!)
• We get our Xray Cloud App token with the credentials mentioned above
• We push the test-result.xml to the Xray Cloud App
• Finally we specify the Project Key and our Xray Test Plan which we will create in a second.

Step 3 — Install and configure Xray:

Installing Xray is really easy and straight forward.

1. Go to Apps in your Settings menu on the top right
2. Go to Find new Apps and search for Xray
3. Click on the Card and install the App via Try Free
4. In app marketplace side in Jira Cloud

Step 4 — Create a Test Plan and a Task:

First we create a new Project. You can choose any type of Project Type, all will work with Xray. In our case we create a Classic — Kanban Project with the name Test project and the key TP.

If you’ve never used Xray before and need help setting up your project, they got awesome documentation here:

Quick Setup - Xray Cloud Documentation - Xray (getxray.app)

After configuring our Project we need to create two issues.

1. Test Plan - This one will receive all Test Execution and Test issues which will automatically be created when triggering the pipeline (see our pipeline.yml)
2. Task - This one is our Task with the Workflow we want to Test.

Step 5 — Trigger Bitbucket Pipelines via Jira Automation:

So let’s do the following:

• Go to Project Settings
• Choose Automation
• Create a New rule

For the trigger we will use Issue transitioned:

Next, we add a New Action. Under the section Notification we pick Send web request.

Webhook URL:

https://api.bitbucket.org/2.0/repositories/{workspace}/{repo_slug}/pipelines/

In the text area just add the following JSON:

{
 "target": {
 "ref_type": "branch",
 "type": "pipeline_ref_target",
 "ref_name": "master"
 }
}

For more details take a look at the REST API Endpoint in the official documentation.

Now that we have everything setup, let’s grab some popcorn and let’s enjoy what we’ve just build.

Our Jira Board should look something like this

Let’s move the issue TP-1 in our Test Plan from Backlog to In Progress. This should trigger our pipeline! Go to your Bitpucket pipeline — you should see that the build just has started.

After the pipeline tasks are successfully done let’s head back to Jira.

Our Task was transitioned to In Progress thanks to our Python app. Now we know that the business team will be happy and we can prove to them even before the presentation that everythings works as requested.

Xray automatically created a Test Execution and a Test for us. Even more we’ve pushed the results to the Test Plan.

Test Plan — Result after running the Bitbucket pipeline

Test Execution — Created after running the Bitbucket pipeline

About Jodocus

The Jodocus team brings its expertise in business processes, ALM, DevOps and Cloud to you! We make our clients’ lives easier by fitting the tools to your needs.

Jodocus is a Cloud First Solution Partner with a focus on Business Processes. We help teams of all kind to digitize their Workflows and use the power of collaboration to elevate their daily work. Our expertise is to smoothly integrate Atlassian tools into your companies‘ toolset and get your teams ready for teamwork!

About Xray

Xray is the leading Quality Assurance and Test Management app for Jira. Improve the quality of your systems through effective and efficient testing that runs through the entire software development lifecycle. Xray supports both manual and automated tests and provides powerful reports to ensure full requirements coverage. More than 4.5 million testers, developers and QA managers trust Xray to manage 100+ million test cases each month. Xray is a mission-critical tool at over 5,000 companies in 70+ countries, including 137 of the Global 500.

Hope you liked it, even more I hope it helps you! Please let me know!

Also published at https://medium.com/jodocus-blog/test-automation-of-jira-cloud-workflows-using-bitbucket-pipelines-and-xray-test-management-3ff316b03e25