Triaging pa11y Results

These steps are intended for Product Managers to triage pa11y results in order to create JIRA tickets. Step by Step instructions are provided here.

Grab the pa11y report

Work with a Developer to get the pa11y report exported to a spreadsheet, like this:

1. Initial pa11y dump in Google Sheets

sample spreadsheet of a series of pa11y failures

2. Clone an AirTable Project

Open up the AirTable Worksheet for the project, being sure to use a table with pa11y pre-set.

An empty table for pa11y results

3. Google Sheet: Copy URLs on each row

In the spreadsheet, copy all the URLs for each Failure. You will need to select-all from the Formula Bar. We want to get each of these URLs into their own row on the AirTable.

Select-All and Copy all URLs from the Formula Bar.

4. Paste URLs into AirTable

Paste these URLs into AirTable. They all should appear on their own row. You will be prompted from AirTable for confirmation to "expand" table to accommodate additional rows. Select 'Yes.'

All the URLs are pasted into AirTable.

5. Grabbing pa11y error "name"

Next, in the spreadsheet, reference the Error "name", selecting only the last part of the full error name, for instance, "H91.InputText.Name"

Select the cell with the Error Name; and select the text following the string of numbers.

6. Paste in pa11y Error Name into AirTable

In AirTable, paste this into the "Name" and drag to copy for all the URLs that were pasted.

Paste, then Drag to Fill!

7. Find the Match

In AirTable, navigate to the "Advisory Technique" field. Begin typing the letters in Column A, "Name" to get a list of matches. Once you have found the "Match" select it.

Enter the first few characters in the "Advisory Technique" column to find the match

When starting from an existing AirTable base, the "Advisory Technique" column might be hidden, so check that it is not hidden.

Some pa11y tests like H91 have multiple error codes (like InputText.Name, and InputText.Area) and you may not immediately see the one you are searching, so keep typing to find the exact match as it may be further down the list.

8. Auto-Filled Magic: The What and Why

You will notice WCAG Criteria, Advisory Technique Title and Advisory Technique Summary columns will get auto-filled. Then Fill down to rest of the rows that contain the URLs that you just pasted.

Three Columns are auto-populated with better information

9. Extra Step: Preparing for better Sorting and Organization

It is also helpful to identify and group & sort by Type of Pattern (atom, organism, molecule) and the Pattern (e.g., "Cards). We can extract this information from the URL. Let's download the AirTable that we just created.

Select the ... from the menu bar, and 'Download CSV'

10. Preparing the Excel File

We will first split the URLs by the delimiter /

Use the Text-to-Columns section in Excel
Select "Other" and insert / as the delimiter.
Select the Destination to be the first empty column, such as $J$1

Now we can delete the parts of the URL that we don't need.

Highlight parts of the URL we won't be needing.

You can also delete the column that contains the URL ending in .html

We now need to get rid of the 01- that are in 2 columns. We will trim these values. Insert a column before Col K and enter this formula:

=RIGHT(K1,LEN(K1)-3)

The next part will be sorting, but let's preserve the original sort by assigning numeric values in Column L.

=L2+1

Drag to the rest of the URLs.

Important! Copy the #s and paste into the adjacent column; Paste Special > Values. This preserves the original #.

You will now have the ability to get this order of URLs back to the same order as in AirTable.

Next, we will sort by Col K:

Then you will insert a new column to the left; begin typing the pattern that you see in the first few characters, e.g., "atoms" "molecules". Drag to fill rows for each type. Perform this for all the pattern URLs: atoms, molecules, organisms, templates, pages, styleguide

Insert a new column before L. Similar to removing the 03-'s, we will use the =RIGHT formula to remove the atoms- organisms - and so forth for all pattern types. Since each is a unique character count, you will need to update the formula, using the below as a guide, focusing on the last digit.

=RIGHT(M1,LEN(M1)-6) (atoms)
=RIGHT(M11,LEN(M11)-10) (molecules)
=RIGHT(M17,LEN(M17)-10) (organisms)
=RIGHT(M23,LEN(M23)-6) (pages)
=RIGHT(M122,LEN(M122-11) (styleguide)
=RIGHT(M132,LEN(M132-10) (templates)

Then copy, insert new column Paste Special > Values. Once you've created this new column, you may delete the original column with the formulas.

Sort (again) on the row of #s to return to the original sort.

11. Transferring to AirTable

Copy Columns K & L.

Paste into AirTable, under Pattern Type and Patterns

Voila!

Now your pa11y report can be sorted and organized by a variety of groups.

12. AirTable Views: Sorting & Grouping

The AirTable project is set up with views:

  1. By Failure (As you see it)

  2. By Pattern (See all the errors by Pattern, e.g., "Header")

  3. By Pattern Type ( See all errors grouped by Type, e.g., all atoms, all molecules)

View, selectable in top left, "By Failure"
View, selectable in top left, "By Pattern"

13. Preparing JIRA Tickets

Create JIRA tickets by Error Type:

  1. Set your AirTable View "By Failure"

  2. Go to your JIRA Project

  3. Create a New Story

  4. Copy the "Advisory Technique Title" into the Summary Field

  5. In the Body, Type in the Description ("Advisory Technique Summary"), the WCAG Criteria affected.

  6. Copy the Pattern Type, Patterns, & URLs and go to a Markdown Editor: https://thisdavej.com/copy-table-in-excel-and-paste-as-a-markdown-table/

  7. Paste the Markdown into JIRA ticket and Save.

Sample of a pa11y-reported JIRA ticket