pa11y is an Open Source, continuous-integration accessibility testing tool that Phase2 has been rolling out into its work-streams and development process(es).
Pa11y efforts began in Spring 2018 when Jonathan Adams and Tanner Heffner installed it on Outrigger to be available out-of-the-box for developers to use on their local environments.
In February - March 2019, Northwell Health engaged Phase2 in both accessibility remediation and pa11y setup, so this has been a great opportunity for us to continue developing the product to better serve our purposes. We need a mechanism to both find and fix the accessibility issues. Pa11y automated testing has satisfied the ability to find issues, but the ‘fix’ step has not been as clear, due to vague context and organization of failures.
For instance, the scans often find thousands of failures, with each criteria reporting repetitive failures, but what’s not immediately apparent is which errors resolve many. Or which error negatively impacts a core business requirement. Currently, an AirTable database has been set-up with context and information to communicate, triage, and prioritize issues. This is currently a manual step to match up pa11y failures to the full context and remediation support information, and there is an opportunity for this step to be automated utilizing the AirTable API. This can then inform and improve a pa11y dashboard.
There is opportunity for the integration that Phase2 builds with pa11y + AirTable to replace SiteImprove. The main advantage of this is that pa11y can point to any URL, of which we can use for prospective clients. (On the contrary, due to contractual limitations we cannot use SiteImprove to point to production URLs)
pa11y on Outrigger
Complete - Spring 2018
pa11y basic dashboard
Complete - Fall 2018
pa11y test context & remediation into AirTable
Complete - Spring 2019
pa11y hosted report (Northwell)
Complete - Spring 2019
pa11y on Docksal (Northwell)
In Progress - Spring 2019
pa11y on Drupal (Northwell)
In Progress - Spring 2019
pa11y on Phase2 GitLab
pa11y Dashboard Update
pa11y Airtable API
pa11y on Octane
8 Hours (Mike Potter)
pa11y Screenshot Capture | NOT MVP
The goal of this is to get pa11y to be available on new development builds “out of the box.” it is now available and may be utilized by a developer. Jonathan Adams performed this in Spring of 2018 with the goal of applying it on the Autism Speaks project.
Tanner Heffner took his pa11y dashboard ‘prototype’ on Memorial Sloane Kettering (MSK) and refined it be more organized and deliver high-level information such as a pie chart of errors, organization of failures with a list of URLs, and the inverse: providing a table with the URL and associated failures.
A challenge of the pa11y results is understanding clearly the actions needed to remediate:
What does the failure mean? How does it tie back to the Web Content Accessibility Guidelines?
What should be fixed first, as it pertains to the client’s core business goals?
How many of the failures can be fixed with 1 error?
How can the fixes be validated?
AirTable is a dynamic database with multiple tables feeding into one. This is notable in the triaging and evaluation of tickets since the context, how to fix, patterns/features, and frequency of error can be extracted from views within AirTable. Additionally, a feature of AirTable includes the ability to generate client-facing reports, so there is the benefit of creating highly-polished report documentation highlighting the pa11y results.
Anyone within the Northwell project may view the pa11y results for two design systems at a Northwell URL. This is notable because anyone can view the latest pa11y result(s) without reliance of accessing a developer’s local environment.
Jake Strawn has made it so that in new (Docksal) local environments, Northwell developers may generate and view pa11y results locally on two different design systems. All commands happen in Docker containers rather than the host machine(s) as to ideally provide parity between Mac OS and Windows environments.
We will be attempting to provide Pa11y results on specific, predefined Drupal pages for the Northwell Drupal sites. With pa11y’s flexibility to scan any URL, we can simply provide it a list of URLs (local or remote) to access and be able to compare pa11y results from the design system instances from results once those same elements are used in Drupal.
The benefit of getting pa11y setup within Phase2 GitLab environments would be so that Product Managers and Project Managers may view pa11y results and support the development team in allocating time and resources to remediation. This creates greater transparency to the accessibility process.
Tanner Heffner has wire-framed a more dynamic dashboard that better communicates the state of the accessibility testing results on an environment. Tanner anticipates 2-3 weeks of effort to complete. Jake Strawn and Jennifer Grace, by nature of their current involvement on the Northwell pa11y project, are interested in this work should they finish ahead of schedule with the remediation fixes. Tanner has this sketched out in the Phase2 Repo https://github.com/phase2/pa11y-dashboard/issues/3#issuecomment-467110342
The current AirTable work is a manual process that takes upwards 4 hours to prepare per environment. You may view the manual documentation process to get a sense of the numerous steps involved. This will also bring in AirTable logic and content to the updated pa11y dashboard, creating a robust system of triaging, evaluation, and prioritization. This will be one step closer to accomplishing our core use for SiteImprove.
Mike Potter has expressed interest in making pa11y part of Octane. The ability to have pa11y be a part of a new Drupal build is especially attractive to further standardize our work and process.
Screenshots are technically possible in the pa11y container already (and through chauncey's VRT work), however, it's a matter of wiring everything up, and ensuring screenshots are attached to the appropriate result/issue.
During the pa11y collab lunch (video) it was noted that we wouldn’t want to take a screenshot of every issue, as that could take up excess disk space as well as drastically increase the amount of time it takes tests to run. As a “compromise,”, there could be a button on the dashboard per issue that says "show me a screenshot."