There is a sentiment that accessibility isn’t a checklist, meaning that if you’re really trying to make a site accessible, you don’t just get to check some things off a list and call it perfect. The list may be imperfect and worse, it takes the user out of the equation, so it is said.
Karl Groves once argued against this:
I’d argue that a well-documented process which includes checklist-based evaluations are better at ensuring that all users’ needs are met, not just some users.
I mention this because you might consider an automated accessibility testing tool another form of a checklist. They have rules built into them, and they test your site against that list of rules.
I’m pretty new to the idea of these things, so no expert here, but there appears to be quite a few options! Let’s take a look at some of them.
aXe
The Accessibility Engine for automated testing of HTML-based user interfaces. Drop the aXe on your accessibility defects!
aXe can take a look at an HTML document and find potential accessibility problems and report them to you. For example, there are browser extensions (Firefox / Chrome) that you give you the ability to generate a report of accessibility errors on the page you’re looking at.
At it’s heart, it’s a script, so it can be used in all kinds of ways. For example, you could load up that script in a Pen and test that Pen for accessibility.
There is a CLI so you can integrate it into build processes or testing environments or deployment flows or whatnot.
Looks like maybe intern-a11y can help script aXe for extra functionality.
Pa11y
Pa11y is your automated accessibility testing pal. It runs HTML CodeSniffer from the command line for programmatic accessibility reporting.
Pa11y is another tool along these lines. It’s a script that can test a URL for accessibility issues. You can hit it with a file path or URL from the command line (pa11y http://example.com) and get a report.
As well as use it from a Node environment and configure it however needed. It’s actually intentionally meant to be used only programmatically, as it’s the programmatic version of HTML_CodeSniffer, the bookmarklet/visual version.
There is also a native app version called Koa11y if that makes usage easier.
Seren Davies recently wrote about a specific scenario where they picked Pa11y over aXe:
We began by investigating aXe CLI, but soon realised it wouldn’t fit our requirements. It couldn’t check pages that required a visitor to log in, so while we could test our product pages, we couldn’t test any customer account pages. Instead we moved over to Pa11y. Its
beforeScriptstep meant we could log into the site and test pages such as the order history.
Google Accessibility Developer Tools
Google is in on the game with Accessibility Developer Tools.
Its main component is the accessibility audit: a collection of audit rules checking for common accessibility problems, and an API for running these rules in an HTML page.
It’s similar to the others in that it’s designed to be used different ways, like as Grunt task, from the command line, or the browser.
Addy Osmani has a11y, powered by Chrome Accessibility Tools, which appears to provide a nicer API and nicer reporting.
It seems like most of Google’s website auditing weight is thrown behind Lighthouse these days though, which include accessibility tests. For example, the “Buttons Have An Accessible Name” test, but that test is actually aXe under the hood.
It’s unclear to me if Lighthouse runs a complete and up-to-date aXe audit or not, and if the Accessibility Developer Tools are sort of deprecated in favor of that, or what.
Automated Accessibility Testing Tool (AATT)
PayPal is in on the game with AATT, a combination and extension of already-mentioned tools:
Browser-based accessibility testing tools and plugins require manually testing each page, one at a time. Tools that can crawl a website can only scan pages that do not require login credentials, and that are not behind a firewall. Instead of developing, testing, and using a separate accessibility test suite, you can now integrate accessibility testing into your existing automation test suite using AATT.
AATT includes HTML CodeSniffer, aXe, and Chrome developer tool with Express and PhantomJS, which runs on Node.
It spins up a server with an API you can use to test pages on other servers.
accessibilityjs
GitHub themselves recently released accessibilityjs, the tool they use themselves for accessibility testing. They use it on the client side, where when it finds an error, it applies a big huge red border and applies a click handler so you can click it to tell you what the problem is.
They scope it to these common errors:
ImageWithoutAltAttributeErrorElementWithoutLabelErrorLinkWithoutLabelOrRoleErrorLabelMissingControlErrorInputMissingLabelErrorButtonWithoutLabelErrorARIAAttributeMissingError
Tenon.io
Tenen.io is perhaps the easiest of all of them to get started with, as the homepage has a validator right up top where you can copy and paste HTML or drop in a URL to validate.
Tenon.io can identify 508 and WCAG 2.0 issues in any environment – even on your developer’s laptop. Because production is a bad place to discover bugs.
It has a free 30 day / 500 API call trial, and is a paid product beyond that.
Tenon.io integrates in loads of places. Karl himself told me:
We have a CLI. We have Grunt & Gulp plugins, Node modules, and plugins for Chrome, Firefox, IE, and Opera. PHP Classes, Ruby Gems, CMS integrations for WordPress, Drupal, etc.
Honorable Mentions
I’m not intentionally trying to feature or hide any particular accessibility testing tool. All this stuff is new to me. It just seemed like these were a lot of the big players. But web searching around reveals plenty more!
- Tanaguru: “Automated accessibility (a11y) testing tool, with emphasis on reliablity and automation”
- The A11y Machine “is an automated accessibility testing tool which crawls and tests pages of any web application to produce detailed reports.”
- tota11y: “an accessibility (a11y) visualization toolkit”
My work uses the Siteimprove service for scanning parts of our site. They have released a Chrome plug-in as well that uses much of the same technology as their paid service. I like particularly that it categorizes the errors it finds by the WCAG level (A, AA, AAA) as well as referencing the part of the specification being checked and whether it has detected an actual error or something that should be manually checked. (Think “this
<img>doesn’t have an alt attribute” vs. “does the alt attribute adequately describe the image?”) They also link both to the specification and to the WCAG’s pages regarding suggested techniques for complying to the specifications.Another vote for the Siteimprove Chrome Extension! We’d always used Wave in the past, but Siteimprove’s Chrome Extension is thorough and free to anyone who wants it. It also has a modern design and feel which is refreshing.
Might I mention the WAVE extension? https://chrome.google.com/webstore/detail/wave-evaluation-tool/jbbplnpkjmmeebjpijfedlgcdilocofh
Great for a variety of a11y tests, including proper heading level nesting/order.
Another vote from me for the WAVE plugin!
I also like Colorblinding and NoCoffee for emulating different levels and types of vision. Chromevox is good to practice with for experiencing the web through keyboard navigation and a screenreader – I definitely feel I should be putting in more time trying to navigate this way to get a better idea of what it would be like to only be able to navigate this way, what is most challenging about it, what sites leave out, etc.
And to keep things in perspective, I like these lists from gov.uk that show different sets of accessibility characteristics. It’s a good reminder that needs vary by person, and what is accessible for one isn’t necessarily accessible for others. https://accessibility.blog.gov.uk/2016/09/02/dos-and-donts-on-designing-for-accessibility/
Correct! Accessibility is not a checklist. However, these tools allow you to catch the low-hanging fruit (the typos, the oopses, and so on). That being said, the guy you quoted at the start of the post (Karl) applied his strong opinions about how not to do automated testing and wrote Tenon: https://tenon.io/
Accessible doesn’t mean usable.
A site may me perfectly ‘accessible’ yet be a monster to actually work with.
When talking about accessible we mostly think of our blind friends, but don’t forget the others.
Zoombrowsers, usually don’t like deep indenting as it’s extra scrolling (/searching).
Sighted non-mouse users, these won’t ‘hear’ the aria roles. Does it still makes sense to them? (aria role tablist is one to cause confusion)
Color blindness, more folks than you might think have this. Do you use color to present a status? (maybe change the icon too to make it more clear)
Does it still work if they use their mobile to visit your site?
Sometimes even a small change can make there life so much better. (even if they don’t complain about it)
For example, the tools test if your images have an ALT attribute but alt=”ytfytf” will also be marked as correct.
Adding navigation items in a list is good, but if it only contains two items you get “list list containing two items item one item two”. Is this overhead worth it or is it annoying?
Same goes with adding aria roles. You get points for adding them but in reality you should only add them is there is no other way using common html.
These tools might be good to test for the obvious flaws but it’s not a replacement for testing it with the folks who are using your product.
Ask them, get all these (probably) different preferences and try to merge that to a good result :)
Thanks for posting these, automated tools seem like the most cost efficient method to ensure a certain level of accessibility… in fact I can’t really think of any method that wouldn’t involve a list or set of guidelines on at least some level.
Thanks for the write-up! For aXe, you can script extra functionality using the axe-webdriverjs integration (among other things). That makes it easier to do things like script logins, and have test pages with iframes. The aXe chrome and firefox extensions test inside of iframes as well.
There’s also a prerelease axe-core version we’re working on that tests Shadow DOM: https://www.deque.com/blog/test-leading-edge-accessibility-axe-coconut-axe-core-3-0/
Good list of tools here, another vote for the WAVE plugin as mentioned above. As someone that does a lot of work for various banks who use 3rd party Accessibility companies its worth noting that a lot of whats in the WCAG guidelines is also a matter of personal preference to the tester.
I’ve lost count of the number of times my companies work has gone off to one of these third parties only for it to come back with recommendations. Once implemented the changes have to go back for retesting (usually by a different person) and they then recommend we do what we did originally (without knowing it’s what we did originally). Its often a never ending merry go round due to some of the guidelines being totally open to personal interpretation. Its often a total friggin nightmare to be perfectly frank.
It is important for me to get this possibility of technique which named accessbility in view to run my activities in relate with website that we will soon put into my disposal .Therefore I would ask you to proceed for implementation to grow our organanization.