Skip to content

feat(a11y): wire @axe-core/playwright into the e2e suite - #13251

Merged
lokesh merged 8 commits into
internetarchive:masterfrom
lokesh:13007/feat/playwright-axe
Aug 11, 2026
Merged

feat(a11y): wire @axe-core/playwright into the e2e suite#13251
lokesh merged 8 commits into
internetarchive:masterfrom
lokesh:13007/feat/playwright-axe

Conversation

@lokesh

@lokesh lokesh commented Aug 3, 2026

Copy link
Copy Markdown
Collaborator

Closes #13007

Wires @axe-core/playwright into the existing Playwright suite so a11y fix PRs can prove their fix in a real browser. [feature]

Phase 1 (#13006) covers ARIA patterns in Lit components, but those run in jsdom, which never lays out or paints. Colour contrast and anything else needing rendered pixels can only be checked here.

image

Technical

  • @axe-core/playwright devDependency (4.12.1)
  • tests/e2e/a11y.ts — shared a11yCheck(page, options) defaulting to WCAG 2.1 AA, with rules / include / exclude / disableRules scoping
  • tests/e2e/a11y.spec.ts — 3 smoke tests
  • tests/e2e/README.md — full docs
  • .gitignore for playwright-report/ and test-results/, which had none

Not wired to CI, deliberately. No workflow runs npm run test:e2e and this PR doesn't add one. axe-core bumps introduce new rules, so gating on it could fail unrelated PRs. Suggested follow-up: a non-blocking reporting job on a pinned version first.

Baseline. /, /search?q=tolkien, and /subjects/history are clean at WCAG 2.1 AA. Following @RayBB's review, the a11y specs now block archive.org before navigating: axe does scan inside cross-origin iframes under Playwright, and the donation banner's content rotates by campaign (the variant served during development failed image-alt on its close button; the current one doesn't), so leaving it in would tie the scan's result to whatever archive.org serves that day. THIRD_PARTY_FRAMES still excludes any iframe, exported rather than applied as a silent default. Ten nodes come back incomplete for contrast (mostly the gradient borrow buttons, where axe can't determine a background colour) and will never fail a test. Worth a follow-up.

Testing

npx playwright install chromium chromium-headless-shell
npx playwright test                          # 23 passed (20 existing + 3 new)
npx playwright test --grep @a11y --headed    # watch it run

Also npx jest (26 suites, 495 tests, unaffected) and npm run lint (0 errors).

To confirm the gate bites, plant a violation: add this line to the first test in a11y.spec.ts after gotoSettled and rerun.

await page.evaluate(() => document.body.insertAdjacentHTML('beforeend', '<img src="x.gif">'));

It fails on image-alt with the offending markup and remediation steps. (An earlier version of this description said dropping the iframe exclusion would fail on the donation banner. That was true of the banner variant served at the time, but the banner rotates — which is exactly why the specs now block it.)

Adding or editing tests

Scan a page:

import { a11yCheck, expectNoViolations, THIRD_PARTY_FRAMES } from './a11y';

await page.goto('/my-page');
expectNoViolations(await a11yCheck(page, { exclude: THIRD_PARTY_FRAMES }));

For a fix PR, scope to the rule you fixed so it stays green while unrelated violations are outstanding:

expectNoViolations(await a11yCheck(page, { rules: ['frame-title'] }));

Then check it fails without your fix. A scoped rule matching no elements passes trivially, which looks identical to a fix that works. More in tests/e2e/README.md.

Screenshot

No UI changes — test infrastructure only.

Stakeholders

@mekarpeles @RayBB @cdrini @jimchamp

lokesh added 2 commits August 3, 2026 16:36
Phase 1 (internetarchive#13006) covers ARIA patterns in Lit components, but those tests
run in jsdom, which never lays out or paints. Colour contrast, focus
styling, and anything else needing rendered pixels can only be checked in
a real browser.

Adds a shared `a11yCheck` helper wrapping @axe-core/playwright with OL's
WCAG 2.1 AA target, plus per-test scoping so a fix PR can assert just the
rule it fixed while unrelated violations elsewhere stay outstanding.

- `expectNoViolations` prints impact, offending markup, and remediation
  steps instead of Playwright's default array diff.
- `THIRD_PARTY_FRAMES` is exported rather than applied by default, so a
  test that skips part of the page says so in its own body. The
  archive.org donation banner fails `image-alt` and isn't ours to fix.
- One test asserts axe evaluated rules at all, so a bad selector can't
  leave the suite vacuously green.
- tests/e2e/README.md documents the fix-PR pattern.
- Ignore playwright-report/ and test-results/, which had no ignore rule.

Closes internetarchive#13007
lokesh added 3 commits August 4, 2026 10:37
…ight-axe

Resolves the package-lock.json conflict introduced by internetarchive#13006, which added
jest-axe@10 (pinning axe-core to exactly 4.10.2) after this branch last
synced with master.

Regenerated rather than hand-merged: took master's lockfile wholesale and
re-ran `npm install` against the merged package.json. npm keeps jest-axe's
[email protected] in the top-level slot and nests [email protected] under
@axe-core/playwright, so both packages get the version they pin.
Two fixes to the smoke tests.

`goto` resolves on `load`, which can precede first paint. Axe's contrast
rules read computed pixels, so scanning that early varies run to run.
Anchor on #header-bar first, matching every other spec in this directory.

The canary asserted `color-contrast` was in `passes`, but a rule that
starts failing moves to `violations` — so the first real contrast
regression would fail the canary too, reporting "axe is broken" when axe
worked correctly. Count rules axe reached any conclusion about instead,
so it keeps testing what it's named for.
@github-project-automation github-project-automation Bot moved this to Waiting Review/Merge from Staff in Ray's Project Aug 5, 2026

@RayBB RayBB left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving because the tests seem to pass and work overall on my machine with no issues.

However, here's one feedback that seemed legit from AI but I'm not 100% sure.

Anyway, merge when you're ready!

The THIRD_PARTY_FRAMES / donation-banner narrative doesn't reproduce (most important)
The PR says the home page scan fails on image-alt inside the archive.org donation banner unless the banner is excluded, and instructs reviewers to verify by dropping the exclusion. I did exactly that, and it doesn't fail.

  • The banner is an iframe: #donato contains <iframe src="https://archive.org/includes/donate.php?…" title="Banner for donating to the Internet Archive"> — cross-origin from both localhost:8080 and openlibrary.org .
  • @axe-core/playwright scans frames via axe.runPartial injection (confirmed in the installed dist), which cannot inject into cross-origin frames — so anything inside donate.php (including a close-button with no alt) is invisible to axe.
  • My probe, with no exclusion: zero violations. With the exclusion: zero violations. No top-page is missing alt ( topImgsMissingAlt: [] ).
  • The iframe even has a title attribute, so frame-title passes too.
    So today the exclusion is inert, and the "gate bites" verification described in the PR can't be demonstrated. Worse, test 1's green/red state is now hostage to an external service that serves different variants per request (the donation-identifier URL param changed between my runs): if donate.js ever injects inline markup, or a same-origin frame appears, the test can flip with no OL change.
    Suggested fix: make test 1 hermetic by stubbing/blocking archive.org ( context.route('https://archive.org/**', route => route.abort()) ) — axe can't scan the banner anyway, so you lose nothing, and the test becomes deterministic. At minimum, correct the PR description's verification claim, since it's currently misleading.

The donation banner's content rotates by campaign, and axe does scan
inside cross-origin frames under Playwright, so an unblocked scan could
flip between runs with no Open Library change. Abort archive.org
requests before navigating and correct the comments/README that pinned
the image-alt violation to the banner's current markup.

Claude-Session: https://claude.ai/code/session_012W2ka8av54rbJviSEDHXW7
@lokesh
lokesh merged commit 9042789 into internetarchive:master Aug 11, 2026
5 checks passed
@github-project-automation github-project-automation Bot moved this from Waiting Review/Merge from Staff to Done in Ray's Project Aug 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

A11y Phase 2: Playwright + axe-core integration for page-level WCAG testing

3 participants