Enhancing Test Reliability: Exploring Playwright's Retry APIs for Robust Testing

Global Retries

Configuring Global Retries

Playwright provides a built-in global retry mechanism for test cases. This means that when a test fails, Playwright automatically retries the test up to the configured number of times before marking it as a failure. To set the global retry ability, you can use the retries option in the Playwright config file (playwright.config.ts):

import { defineConfig } from '@playwright/test';

export default defineConfig({
  retries: process.env.CI ? 2 : 0,
});

This code snippet configures retries only when running tests in a continuous integration (CI) environment. You can override this setting by using the --retries flag when running tests from the command line:

npx playwright test --retries=1

Configuring Retries per Test Block

If you need more granular control over retries, you can configure them for individual test blocks or groups of tests. To do this, use the test.describe.configure() function:

import { test } from '@playwright/test';

test.describe('Playwright Test', () => {
  test.describe.configure({ retries: 5 });

  test('should work', async ({ page }) => {
    // Your test code here
  });
});

This configuration allows the specified test block to be retried up to 5 times before being marked as a failure.

Auto-waiting and Retrying

Built-in Auto-waiting Mechanism

Playwright has a built-in auto-waiting and retry mechanism for locators (e.g., page.getByRole()) and matchers (e.g., toBeVisible()). This mechanism continuously runs the specified logic until the condition is met or the timeout limit is reached, helping to reduce or eliminate flakiness in your tests. For instance, you don't need to manually specify a wait time before running some code, such as waiting for a network request to complete.

To learn more about the specific timeout limits, refer to the Playwright timeout documentation.

Custom Conditions with Retrying and Polling APIs

Sometimes, you might need to wait for a condition unrelated to the UI, such as asynchronous processes or browser storage updates. In these cases, you can use Playwright's Retrying and Polling APIs to explicitly specify a condition that is awaited until it is met.

Using the Retry API

The Retry API uses a standard expect method along with the toPass(options) method to retry an assertion within the expect block. If the assertion fails, the expect block is retried until the timeout limit is reached or the condition passes.

The example below demonstrates waiting for a value to be written to local storage:

import { test } from '@playwright/test';

test('runs toPass() until the condition is met or the timeout is reached', async ({ page }) => {
  await expect(async () => {
    const localStorage = await page.evaluate(() => JSON.stringify(window.localStorage.getItem('user')));
    expect(localStorage).toContain('Tim Deschryver');
  }).toPass();
});

Using the Poll API

The Poll API is similar to the Retry API, but it uses the expect.poll() method instead of a standard expect block. The expect.poll() method also returns a result, which is used to invoke the matcher.

The example below demonstrates waiting for a process state to be completed:

import { test } from '@playwright/test';

test('runs expect.poll() until the condition is met or the timeout is reached', async ({ page }) => {
  await expect
    .poll(async () => {
      const response = await page.request.get('https://my.api.com/process-state');
      const json = await response.json();
      return json.state;
    })
    .toBe('completed');
});

Both the Retry and Poll APIs can be configured with custom timeout and interval durations:

import { test } from '@playwright/test';

test('runs toPass() until the condition is met or the timeout is reached', async ({ page }) => {
  await expect(async () => {
    // Your test code here
  }).toPass({ intervals: [1000, 1500, 2500], timeout: 5000 });
});

test('runs expect.poll() until the condition is met or the timeout is reached', async ({ page }) => {
  await expect
    .poll(async () => {
      // Your test code here
    }, { intervals: [1000, 1500, 2500], timeout: 5000 })
    .toBe('completed');
});

Test Retries in Worker Processes

How Worker Processes Work

Playwright Test runs tests in worker processes, which are independent OS processes orchestrated by the test runner. These workers have identical environments and start their own browsers. When all tests pass, they run in order in the same worker process. However, if any test fails, Playwright Test discards the entire worker process along with the browser and starts a new one. Testing continues in the new worker process, beginning with the next test.

Enabling Retries in Worker Processes

When you enable retries, the second worker process starts by retrying the failed test and continues from there. This approach works well for independent tests and guarantees that failing tests can't affect healthy ones.

To enable retries, you can use the --retries flag or configure them in the configuration file:

npx playwright test --retries=3

import { defineConfig } from '@playwright/test';

export default defineConfig({
  retries: 3,
});

Playwright Test categorizes tests as follows:

• "passed" - tests that passed on the first run;
• "flaky" - tests that failed on the first run but passed when retried;
• "failed" - tests that failed on the first run and all retries.

Detecting Retries at Runtime

You can detect retries at runtime using the testInfo.retry property, which is accessible to any test, hook, or fixture.

The example below demonstrates clearing the server-side state before retrying a test:

import { test, expect } from '@playwright/test';

test('my test', async ({ page }, testInfo) => {
  if (testInfo.retry) {
    await cleanSomeCachesOnTheServer();
  }
  // Your test code here
});

Configuring Retries for Specific Groups or Files

You can specify retries for a specific group of tests or a single file using the test.describe.configure() function:

import { test, expect } from '@playwright/test';

test.describe(() => {
  test.describe.configure({ retries: 2 });

  test('test 1', async ({ page }) => {
    // Your test code here
  });

  test('test 2', async ({ page }) => {
    // Your test code here
  });
});

Grouping Dependent Tests with test.describe.serial()

For dependent tests, you can use test.describe.serial() to group them together, ensuring they always run together and in order. If one test fails, all subsequent tests are skipped. All tests in the group are retried together. While it's usually better to make your tests isolated, this technique can be useful when you need to run tests in a specific order.

import { test } from '@playwright/test';

test.describe.serial.configure({ mode: 'serial' });

test('first good', async ({ page }) => {
  // Your test code here
});

test('second flaky', async ({ page }) => {
  // Your test code here
});

test('third good', async ({ page }) => {
  // Your test code here
});

Reusing a Single-Page Object Between Tests

By default, Playwright Test creates an isolated Page object for each test. However, if you'd like to reuse a single Page object between multiple tests, you can create your own in the test.beforeAll() hook and close it in the test.afterAll() hook:

import { test, Page } from '@playwright/test';

test.describe.configure({ mode: 'serial' });

let page: Page;
test.beforeAll(async ({ browser }) => {
  page = await browser.newPage();
});

test.afterAll(async () => {
  await page.close();
});

test('runs first', async () => {
  await page.goto('https://playwright.dev/');
});

test('runs second', async () => {
  await page.getByText('Get Started').click();
});

Conclusion

In summary, Playwright offers various retry APIs to make your tests more resilient and less flaky. The built-in retry mechanism for locators and matchers covers most daily use cases. However, for assertions that need to wait for external conditions, you can use the explicit retry and polling APIs. Additionally, you can utilize the global retry mechanism for test cases to handle inconsistencies caused by conditions beyond your control.

By incorporating these retry strategies into your testing workflow, you can ensure a more robust and reliable testing experience, leading to higher-quality software and happier end-users.