The Four Horsemen of Flaky Tests

Flaky automation tests can be a bane for QA engineers, introducing uncertainty and undermining the reliability of test suites. Drawing from my experience as an SDET and QA Automation mentor, this article offers practical advice to conquer flakiness. While I’ll provide JavaScript and Playwright examples, these universal tips are applicable in any language and framework, helping you write robust and dependable automation tests. Let’s dive in and ensure your tests stand firm.

1. Avoid the implicit wait methods

Occasionally, you will encounter situations where you must wait for elements to appear in the DOM or for your application to reach a specific state to proceed with your test scenario. Even with modern, intelligent automation frameworks like Playwright’s auto-wait features, there will be instances where you need to implement custom wait methods. For instance, consider a scenario with text fields and a confirmation button. Due to specific server-side behavior, the confirmation button becomes visible only after approximately five seconds of completing the form. In such cases, resisting the temptation of inserting a five-second implicit wait between two test steps is essential.

textField.fill('Some text')
waitForTime(5000)
confirmationButton.click()

Use a smart approach instead and wait for the exact state. It can be the text element appearing after some loading or a loader spinning element disappearing after the current step is successfully done. You can check if you are ready for the next test step.

button.click()
waitFor(textField.isVisible())
textField.fill()

Of course, there are cases where you need to wait for something you cannot check smartly. I suggest adding comments in such places or even adding the reason explanation as a parameter in your wait methods or something like that:

waitForTime(5000, {reason: "For unstable server processed something..."}

An additional profit of using it is the test reports where you can store such explanations so it will be obvious for someone else or even for future you what and why such five-second waiting here.

waitForTime(5000, {reason: "For unstable server processed something..."}
button.click()
waitFor(textField.isVisible())
textField.fill()

2. Use robust and reliable locators to select elements

Locators are a crucial part of automation testing, and everyone knows it. However, despite this simple fact, many automation engineers are bailing on the stability and using something like this in their tests.

//*[@id="editor_7"]/section/div[2]/div/h3[2]

That is a silly approach because the DOM structure is not so static; some different teams can sometimes change it, and e this after your tests fail. So, use the robust locators. You’re welcome to my XPath-related story reading, by the way.

3. Make tests independent from each other

When running the test suite with multiple tests in a single thread, it’s crucial to ensure each test is independent. This means that the first test should return the system to its original state so the next test doesn’t fail due to unexpected conditions. For example, if one of your tests modifies user settings stored in LocalStorage, clearing the LocalStorage entry for User Settings after the first test runs is a good approach. This ensures that the second test will be executed with default user settings. You can also consider actions like resetting the page to the default entry page and clearing cookies and database entries so that each new test begins with identical initial conditions.

For example, in Playwright, you can use the beforeAll(), beforeEach(), afterAll(), and afterEach() annotations to achieve it.

Check the next test suite with a couple of tests. The first is changing the user's appearance settings, and the second is checking the default appearance.

test.describe('Test suite', () => {
  test('TC101 - update appearance settings to dark', async ({page}) => {
    await user.goTo(views.settings);
    await user.setAppearance(scheme.dark);
  });

  test('TC102 - check if default appearance is light', async ({page}) => {
    await user.goTo(views.settings);
    await user.checkAppearance(scheme.light);
  });
});

If such a test suite runs in parallel, everything will go smoothly. However, if these tests are run one by one, you could face a failed test because one of them is interfering with another. The first test changed the appearance from light to dark. The second test checks if the current appearance is light. But it will fail since the first test changed the default appearance already.

To solve such a problem, I’ve added the afterEach() hook executed after each test. In this case, it will navigate to the default view and clear the user's appearance by removing it from the Local Storage.

test.afterEach(async ({ page }) => {
  await user.localStorage(appearanceSettings).clear()
  await user.goTo(views.home)
});

test.describe('Test suite', () => {
  test('TC101 - update appearance settings to dark', async ({page}) => {
    await user.goto(views.settings);
    await user.setAppearance(scheme.dark);
  });

  test('TC102 - check if default appearance is light', async ({page}) => {
    await user.goTo(views.settings);
    await user.checkAppearance(scheme.light);
  });
});

Using this approach, each test in this suite will be independent.

4. Use automatic retries wisely

No one is immune to encountering flaky tests, and a popular remedy for this issue involves using retries. You can configure retries to occur automatically, even at the CI/CD configuration level. For instance, you can set up automatic retries for each failed test, specifying a maximum retry count of three times. Any test that initially fails will be retried up to three times until the test execution cycle is complete. The advantage is that if your test is only slightly flaky, it may pass after a couple of retries.

However, the downside is that it could fail, resulting in extra runtime consumption. Moreover, this practice can inadvertently lead to an accumulation of flaky tests, as many tests may appear to “pass” on the second or third attempt, and you might incorrectly label them as stable. Therefore, I do not recommend setting auto-retries globally for your entire test project. Instead, use retries selectively in cases where you cannot promptly resolve the underlying issues in the test.

For instance, pretend you have a test case where you must upload some files using the ‘filechooser’ event. You get the <Timeout exceeded while waiting for event ‘filechooser’> error, which indicates that the Playwright test is waiting for a ‘filechooser’ event to occur but is taking longer ied timeout.

async function uploadFile(page: Page, file) {
  const fileChooserPromise = page.waitForEvent('filechooser');
  await clickOnElement(page, uploadButton);
  const fileChooser = await fileChooserPromise;
  await fileChooser.setFiles(file);
}

This can be due to various reasons, such as unexpected behavior in the application or a slow environment. Because it is random behavior, the first thing you could think about is to use automatic retry for this test. However, if you retry the entire test, you risk wasting extra time and getting the same behavior you got with the first error in the uploadFile() method itself, so if the error occurs, you don’t need to retry the entire test.

To do this, you can use the standard try…catch statement.

async function uploadFile(page: Page, file) {
  const maxRetries = 3;
  let retryCount = 0;

  while (retryCount < maxRetries) {
    try {
      const fileChooserPromise = page.waitForEvent('filechooser');
      await clickOnElement(page, uploadButton);
      const fileChooser = await fileChooserPromise;
      await fileChooser.setFiles(file);
      break; // Success, exit the loop
    } catch (error) {
      console.error(`Attempt ${retryCount + 1} failed: ${error}`);
      retryCount++;
    }
  }
}

Such an approach will save extra time and make your test less flaky.

In closing, the path to reliable automation tests is straightforward. You can minimize or actively address common challenges and implement intelligent strategies. Remember, this journey is ongoing, and your tests will become more dependable with persistence. Say goodbye to flakiness and welcome stability. Your commitment to quality ensures project success. Happy testing!