feat(a11y): wire @axe-core/playwright into the e2e suite - #13251
Merged
Conversation
Phase 1 (internetarchive#13006) covers ARIA patterns in Lit components, but those tests run in jsdom, which never lays out or paints. Colour contrast, focus styling, and anything else needing rendered pixels can only be checked in a real browser. Adds a shared `a11yCheck` helper wrapping @axe-core/playwright with OL's WCAG 2.1 AA target, plus per-test scoping so a fix PR can assert just the rule it fixed while unrelated violations elsewhere stay outstanding. - `expectNoViolations` prints impact, offending markup, and remediation steps instead of Playwright's default array diff. - `THIRD_PARTY_FRAMES` is exported rather than applied by default, so a test that skips part of the page says so in its own body. The archive.org donation banner fails `image-alt` and isn't ours to fix. - One test asserts axe evaluated rules at all, so a bad selector can't leave the suite vacuously green. - tests/e2e/README.md documents the fix-PR pattern. - Ignore playwright-report/ and test-results/, which had no ignore rule. Closes internetarchive#13007
…ight-axe # Conflicts: # .gitignore
…ight-axe Resolves the package-lock.json conflict introduced by internetarchive#13006, which added jest-axe@10 (pinning axe-core to exactly 4.10.2) after this branch last synced with master. Regenerated rather than hand-merged: took master's lockfile wholesale and re-ran `npm install` against the merged package.json. npm keeps jest-axe's [email protected] in the top-level slot and nests [email protected] under @axe-core/playwright, so both packages get the version they pin.
Two fixes to the smoke tests. `goto` resolves on `load`, which can precede first paint. Axe's contrast rules read computed pixels, so scanning that early varies run to run. Anchor on #header-bar first, matching every other spec in this directory. The canary asserted `color-contrast` was in `passes`, but a rule that starts failing moves to `violations` — so the first real contrast regression would fail the canary too, reporting "axe is broken" when axe worked correctly. Count rules axe reached any conclusion about instead, so it keeps testing what it's named for.
Closed
50 tasks
RayBB
approved these changes
Aug 11, 2026
RayBB
left a comment
Collaborator
There was a problem hiding this comment.
Approving because the tests seem to pass and work overall on my machine with no issues.
However, here's one feedback that seemed legit from AI but I'm not 100% sure.
Anyway, merge when you're ready!
The THIRD_PARTY_FRAMES / donation-banner narrative doesn't reproduce (most important)
The PR says the home page scan fails on image-alt inside the archive.org donation banner unless the banner is excluded, and instructs reviewers to verify by dropping the exclusion. I did exactly that, and it doesn't fail.
- The banner is an iframe: #donato contains <iframe src="https://archive.org/includes/donate.php?…" title="Banner for donating to the Internet Archive"> — cross-origin from both localhost:8080 and openlibrary.org .
- @axe-core/playwright scans frames via axe.runPartial injection (confirmed in the installed dist), which cannot inject into cross-origin frames — so anything inside donate.php (including a close-button
with no alt) is invisible to axe.
- My probe, with no exclusion: zero violations. With the exclusion: zero violations. No top-page
is missing alt ( topImgsMissingAlt: [] ).
- The iframe even has a title attribute, so frame-title passes too.
So today the exclusion is inert, and the "gate bites" verification described in the PR can't be demonstrated. Worse, test 1's green/red state is now hostage to an external service that serves different variants per request (the donation-identifier URL param changed between my runs): if donate.js ever injects inline markup, or a same-origin frame appears, the test can flip with no OL change.
Suggested fix: make test 1 hermetic by stubbing/blocking archive.org ( context.route('https://archive.org/**', route => route.abort()) ) — axe can't scan the banner anyway, so you lose nothing, and the test becomes deterministic. At minimum, correct the PR description's verification claim, since it's currently misleading.
The donation banner's content rotates by campaign, and axe does scan inside cross-origin frames under Playwright, so an unblocked scan could flip between runs with no Open Library change. Abort archive.org requests before navigating and correct the comments/README that pinned the image-alt violation to the banner's current markup. Claude-Session: https://claude.ai/code/session_012W2ka8av54rbJviSEDHXW7
This was referenced Sep 2, 2026
This was referenced Sep 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #13007
Wires
@axe-core/playwrightinto the existing Playwright suite so a11y fix PRs can prove their fix in a real browser. [feature]Phase 1 (#13006) covers ARIA patterns in Lit components, but those run in jsdom, which never lays out or paints. Colour contrast and anything else needing rendered pixels can only be checked here.
Technical
@axe-core/playwrightdevDependency (4.12.1)tests/e2e/a11y.ts— shareda11yCheck(page, options)defaulting to WCAG 2.1 AA, withrules/include/exclude/disableRulesscopingtests/e2e/a11y.spec.ts— 3 smoke teststests/e2e/README.md— full docs.gitignoreforplaywright-report/andtest-results/, which had noneNot wired to CI, deliberately. No workflow runs
npm run test:e2eand this PR doesn't add one. axe-core bumps introduce new rules, so gating on it could fail unrelated PRs. Suggested follow-up: a non-blocking reporting job on a pinned version first.Baseline.
/,/search?q=tolkien, and/subjects/historyare clean at WCAG 2.1 AA. Following @RayBB's review, the a11y specs now blockarchive.orgbefore navigating: axe does scan inside cross-origin iframes under Playwright, and the donation banner's content rotates by campaign (the variant served during development failedimage-alton its close button; the current one doesn't), so leaving it in would tie the scan's result to whatever archive.org serves that day.THIRD_PARTY_FRAMESstill excludes any iframe, exported rather than applied as a silent default. Ten nodes come backincompletefor contrast (mostly the gradient borrow buttons, where axe can't determine a background colour) and will never fail a test. Worth a follow-up.Testing
Also
npx jest(26 suites, 495 tests, unaffected) andnpm run lint(0 errors).To confirm the gate bites, plant a violation: add this line to the first test in
a11y.spec.tsaftergotoSettledand rerun.It fails on
image-altwith the offending markup and remediation steps. (An earlier version of this description said dropping the iframe exclusion would fail on the donation banner. That was true of the banner variant served at the time, but the banner rotates — which is exactly why the specs now block it.)Adding or editing tests
Scan a page:
For a fix PR, scope to the rule you fixed so it stays green while unrelated violations are outstanding:
Then check it fails without your fix. A scoped rule matching no elements passes trivially, which looks identical to a fix that works. More in
tests/e2e/README.md.Screenshot
No UI changes — test infrastructure only.
Stakeholders
@mekarpeles @RayBB @cdrini @jimchamp