How to Automate Web Accessibility Testing

...and make it part of your continuous integration pipeline

Making web sites and apps available to everyone is an essential part of design and development. Enforcing digital accessibility, however, can quickly become a difficult and overwhelming task.

It takes time and commitment to understand and adopt the Web Content Accessibility Guidelines (WCAG), and the barrier becomes even greater when working on team projects. It requires every member to be informed about and committed to web accessibility (a11y) implementation, which is not always obvious to the presentation of the site or app. When working against a deadline, regression testing every page simply isn’t feasible.

In this post, I share how to automate web accessibility testing, using a library known as Pa11y CI, and make it part of the continuous integration pipeline for you and your team.

About Pa11y

Pa11y (pronounced “pally”) is a set of free and open source tools that aim to help designers and developers make accessible websites. They offer a range of utilities, including a dashboard interface, web service, and desktop application. To learn more about Pa11y, visit the homepage here.

The key to adopting a11y standards as an organization is to make catching accessibility errors a priority. This can be done using Pa11y CI, a command line version of the tool, as a gatekeeper to prevent inaccessible code from being checked into the code base.

Behind the scenes, Pa11y CI uses HTML CodeSniffer in Headless Chrome to evaluate web pages based on the WCAG. The HTML CodeSniffer is a client-side script that analyzes HTML source code and detects violations of defined standards, in this case, the WCAG.

If the number of errors exceeds the custom threshold, Pa11y CI fails with a non-zero error code and aborts the build process. Additionally, the tool generates a report with hints on how to fix the website to comply with a11y standards.

Five steps to integrate Pa11y CI

I made a sample app to try out and demonstrate the functionalities of Pa11y CI. You can find it on my Github page and follow along. While this demo is built with the Angular 7 framework, Pa11y CI can be used with any websites.

1. Install the necessary npm packages

To start, we need to install five node modules:

1. pa11y-ci
2. http-server —serves static files from the command line; we use this to serve the sitemap
3. wait-on —waits for files, ports, sockets, and http(s) resources to become available; we use this to hold off on running pa11y against our website until our website finishes building and the sitemap is served
4. npm-run-all — runs multiple npm scripts in a parallel or sequential fashion
5. husky — sets up git hooks

In your preferred console, run the following commands:

$ npm install -g pa11y-ci
$ npm install http-server -g
$ npm install wait-on
$ npm install npm-run-all --save-dev
$ npm install husky --save-dev

2. Set up Pa11y CI configuration

By default, Pa11y CI looks for a JSON config file named .pa11yci in the current working directory. There are a number of options for customizing the tests, including:

• standard

  : The accessibility standard to use: Section508, WCAG2A, WCAG2AA (default), WCAG2AAA.

• level

  : The level of issue to fail on (exit with code 2); options include error, warning, and notice.

• timeout

  : Time in milliseconds that a test should be allowed to run before calling back with a timeout error; defaults to 30,000. Note that this is the timeout for the entire test run, including the time to initialize Chrome, load the page, and run the tests.

• threshold

  : The number of errors, warnings, or notices permitted before failing the test with exit code 2.

• ignore

  : Array of codes and types to ignore; defaults to an empty array. Use the HTML Code Sniffer guidelines to determine the exact names of which regulations to ignore

• viewport

  : This can be used to check different window sizes, mobile views, etc.

• action

  : User interactions simulated before Pa11y tests the page; it defaults to an empty array. There are a number of options for this configuration, including the ability to click on an element, set a field value, check or uncheck a field, take a screen capture, wait for an element to load, wait for a fragment, path, or URL to load, as well as navigate to a URL. Take a look at the official documentation to see the options and how to use them.

• includeNotice

   and 

  includeWarnings

  : Both of these are boolean values. Issues with a type of notice or warning are not directly actionable, so they are excluded by default, but Pally provides the option to include them via the 

  .pa11yci

   file.

• concurency

  : The number of tests that should be run in parallel; defaults to 2.

• useIncognitoBrowserContext

  : Run test with an isolated incognito browser context, stops cookies being shared and modified between tests; defaults to false.

For more defaults on the available Pa11y configurations, take a look at the official documentation on Github.

As reference, below is a sample 

.pa11yci

Each test run uses the default set of configurations with a timeout limit of 5 seconds and an error threshold of 2. It ignores the “WCAG2AA.Principle1.Guideline1_4.1_4_3.G18,” which requires a color contrast ratio of at least 4.5:1.

This configuration file lists the individual URLs for testing, captures a screenshot for each, and includes specific configurations for each URL, such as a longer timeout period or an action to wait for a certain element to be visible.

// .pa11yci
{
  "standard": "WCAG2AAA",
  "level": "error",
  "defaults": {
    "timeout": 5000,
    "threshold": 2
  },
  "ignore": [
    "     WCAG2AA.Principle1.Guideline1_4.1_4_3.G18.Fail"
  ],
  "urls": [
    {
      "url": "http://localhost:4200",
      "screenCapture": "pa11y-screenshots/home.png"
      "timeout": 8000,
    },
    {
      "url": "http://localhost:4200/login",
      "screenCapture": "pa11y-screenshots/login.png",
      "actions": [
        "wait for element #login-form to be visible"
      ]
    },
  ]
}

3. Create sitemap

Rather than defining individual URLs in the config file, which can quickly become cumbersome for larger websites, you can refer to a locally hosted XML sitemap using the sitemap option.

Use botmap.io to generate a sitemap, and saved it to the project root directory. Then, use the http-server npm package installed earlier to serve the xml file, which will be available on the default port number 8080 at http://localhost:8080/[name-of-your-sitemap-file].xml.

4. Add custom scripts

In your package.json file, add the Pa11y CI tests as custom scripts.

// package.json
"scripts": {
  ...
  "sitemap": "http-server src/assets/nuts-n-bolts_sitemap.xml",
  "pa11y-ci:sitemap": "wait-on http-get://localhost:4200 && wait-on http-get://localhost:8080 && pa11y-ci --sitemap http://localhost:8080",
  "a11y-check": "npm-run-all -p start sitemap pa11y-ci:sitemap"
}

Let’s break down what each of the scripts is doing:

• sitemap

   : hosts the sitemap on the default port of 8080.

• pa11y-ci:sitemap

   : runs Pa11y CI tests after localhost:4200 is done loading.

• a11y-check

   : runs the above two scripts in parallel.

5. Add custom git hooks

As a final step, add the above scripts as custom git hooks, using the husky npm package.

// package.json
"husky": {
  "hooks": {
    "pre-push": "yarn a11y-check"
  }
}

With this, Pa11y tests will be run before every push command is executed, and only code that has passed the accessibility tests can be checked into the code base. And voilà, you’ve now successfully integrated automated accessibility tests into your development pipeline!

Summary

The internet is arguably the most powerful tool in the modern world, offering unprecedented access to information and thereby opportunities. Bob Kahn, co-inventor of the internet protocols (IP), once said:

What the internet is going to be in the future is what society makes it.

I hope to be part of an open and inviting web community, and I believe it starts with us — web designers and developers — making sure our content is accessible to everyone. The tools covered above are a solid start for preventing a11y errors early in the development process and making applications more inclusive. But adopting web accessibility is a continuous, group effort.

Beyond learning the technical skills to meet a11y requirements, we need to internalize the accessibility mindset and build a culture around it. Take the time to train your teams and make sure everyone is on the same page, from the project managers to the designers, developers, and QA teams. Hold regular discussions and workshops focused on understanding the latest guidelines.

Add accessibility requirements to your user stories and test for it in code reviews. Start documenting and sharing best practices in and around your community. Together, we can ensure this powerful platform is open to and accessible to everybody.