Global Retries Configuring Global Retries provides a built-in global retry mechanism for test cases. This means that when a test fails, Playwright automatically retries the test up to the configured number of times before marking it as a failure. To set the global retry ability, you can use the option in the Playwright config file ( ): Playwright retries playwright.config.ts import { defineConfig } from '@playwright/test';

export default defineConfig({
  retries: process.env.CI ? 2 : 0,
}); This code snippet configures retries only when running tests in a ) environment. You can override this setting by using the flag when running tests from the command line: continuous integration (CI --retries npx playwright test --retries=1 Configuring Retries per Test Block If you need more granular control over retries, you can configure them for individual test blocks or groups of tests. To do this, use the function: test.describe.configure() import { test } from '@playwright/test';

test.describe('Playwright Test', () => {
  test.describe.configure({ retries: 5 });

  test('should work', async ({ page }) => {
    // Your test code here
  });
}); This configuration allows the specified test block to be retried up to 5 times before being marked as a failure. Auto-waiting and Retrying Built-in Auto-waiting Mechanism Playwright has a built-in auto-waiting and retry mechanism for locators (e.g., ) and matchers (e.g., ). This mechanism continuously runs the specified logic until the condition is met or the timeout limit is reached, helping to reduce or eliminate flakiness in your tests. For instance, you don't need to manually specify a wait time before running some code, such as waiting for a network request to complete. page.getByRole() toBeVisible() To learn more about the specific timeout limits, refer to the . Playwright timeout documentation Custom Conditions with Retrying and Polling APIs Sometimes, you might need to wait for a condition unrelated to the UI, such as asynchronous processes or browser storage updates. In these cases, you can use Playwright's Retrying and Polling APIs to explicitly specify a condition that is awaited until it is met. Using the Retry API The API uses a standard method along with the method to retry an assertion within the block. If the assertion fails, the block is retried until the timeout limit is reached or the condition passes. Retry expect toPass(options) expect expect The example below demonstrates waiting for a value to be written to local storage: import { test } from '@playwright/test';

test('runs toPass() until the condition is met or the timeout is reached', async ({ page }) => {
  await expect(async () => {
    const localStorage = await page.evaluate(() => JSON.stringify(window.localStorage.getItem('user')));
    expect(localStorage).toContain('Tim Deschryver');
  }).toPass();
}); Using the Poll API The API is similar to the API, but it uses the method instead of a standard block. The method also returns a result, which is used to invoke the matcher. Poll Retry expect.poll() expect expect.poll() The example below demonstrates waiting for a process state to be completed: import { test } from '@playwright/test';

test('runs expect.poll() until the condition is met or the timeout is reached', async ({ page }) => {
  await expect
    .poll(async () => {
      const response = await page.request.get('https://my.api.com/process-state');
      const json = await response.json();
      return json.state;
    })
    .toBe('completed');
}); Both the and APIs can be configured with custom timeout and interval durations: Retry Poll import { test } from '@playwright/test';

test('runs toPass() until the condition is met or the timeout is reached', async ({ page }) => {
  await expect(async () => {
    // Your test code here
  }).toPass({ intervals: [1000, 1500, 2500], timeout: 5000 });
});

test('runs expect.poll() until the condition is met or the timeout is reached', async ({ page }) => {
  await expect
    .poll(async () => {
      // Your test code here
    }, { intervals: [1000, 1500, 2500], timeout: 5000 })
    .toBe('completed');
}); Test Retries in Worker Processes How Worker Processes Work Playwright Test runs tests in worker processes, which are independent OS processes orchestrated by the test runner. These workers have identical environments and start their own browsers. When all tests pass, they run in order in the same worker process. However, if any test fails, Playwright Test discards the entire worker process along with the browser and starts a new one. Testing continues in the new worker process, beginning with the next test. Enabling Retries in Worker Processes When you enable retries, the second worker process starts by retrying the failed test and continues from there. This approach works well for independent tests and guarantees that failing tests can't affect healthy ones. To enable retries, you can use the flag or configure them in the configuration file: --retries npx playwright test --retries=3 import { defineConfig } from '@playwright/test';

export default defineConfig({
  retries: 3,
}); Playwright Test categorizes tests as follows: "passed" - tests that passed on the first run; "flaky" - tests that failed on the first run but passed when retried; "failed" - tests that failed on the first run and all retries. Detecting Retries at Runtime You can detect retries at runtime using the property, which is accessible to any test, hook, or fixture. testInfo.retry The example below demonstrates clearing the server-side state before retrying a test: import { test, expect } from '@playwright/test';

test('my test', async ({ page }, testInfo) => {
  if (testInfo.retry) {
    await cleanSomeCachesOnTheServer();
  }
  // Your test code here
}); Configuring Retries for Specific Groups or Files You can specify retries for a specific group of tests or a single file using the function: test.describe.configure() import { test, expect } from '@playwright/test';

test.describe(() => {
  test.describe.configure({ retries: 2 });

  test('test 1', async ({ page }) => {
    // Your test code here
  });

  test('test 2', async ({ page }) => {
    // Your test code here
  });
}); Grouping Dependent Tests with test.describe.serial() For dependent tests, you can use to group them together, ensuring they always run together and in order. If one test fails, all subsequent tests are skipped. All tests in the group are retried together. While it's usually better to make your tests isolated, this technique can be useful when you need to run tests in a specific order. test.describe.serial() import { test } from '@playwright/test';

test.describe.serial.configure({ mode: 'serial' });

test('first good', async ({ page }) => {
  // Your test code here
});

test('second flaky', async ({ page }) => {
  // Your test code here
});

test('third good', async ({ page }) => {
  // Your test code here
}); Reusing a Single-Page Object Between Tests By default, Playwright Test creates an isolated object for each test. However, if you'd like to reuse a single object between multiple tests, you can create your own in the hook and close it in the hook: Page Page test.beforeAll() test.afterAll() import { test, Page } from '@playwright/test';

test.describe.configure({ mode: 'serial' });

let page: Page;
test.beforeAll(async ({ browser }) => {
  page = await browser.newPage();
});

test.afterAll(async () => {
  await page.close();
});

test('runs first', async () => {
  await page.goto('https://playwright.dev/');
});

test('runs second', async () => {
  await page.getByText('Get Started').click();
}); Conclusion In summary, Playwright offers various retry APIs to make your tests more resilient and less flaky. The built-in retry mechanism for locators and matchers covers most daily use cases. However, for assertions that need to wait for external conditions, you can use the explicit retry and polling APIs. Additionally, you can utilize the global retry mechanism for test cases to handle inconsistencies caused by conditions beyond your control. By incorporating these retry strategies into your testing workflow, you can ensure a more robust and reliable testing experience, leading to higher-quality software and happier end-users. Also published here. The lead image for this article was generated by HackerNoon's AI Image Generator via the prompt "A computer screen with a real bug on it."

Enhancing Test Reliability: Exploring Playwright's Retry APIs for Robust Testing

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Revolutionizing Playwright Learning: My Journey with ChatGPT

100 Days of AI Day 1: From Newsletter to Podcast, Leveraging AI for Audio Transformation

10 Threats to an Open API Ecosystem

10 Indications That You Should Invest in Automation Via APIs

10 Best Practices for Securing Your API

The Noonification: Getting Your API Into Production (10/28/2022)

Revolutionizing Playwright Learning: My Journey with ChatGPT

100 Days of AI Day 1: From Newsletter to Podcast, Leveraging AI for Audio Transformation

10 Threats to an Open API Ecosystem

10 Indications That You Should Invest in Automation Via APIs

10 Best Practices for Securing Your API

The Noonification: Getting Your API Into Production (10/28/2022)

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps