Vetrix Docs

Accessibility regression gating

This document is the operator-facing spec for the a11y regression gate on Vetrix. It covers three moving parts:

  1. The branch-protection configuration that makes a11y jobs required checks on develop and master.
  2. The Playwright a11y suite gate that is required on MRs touching web/.
  3. The nightly full-route axe run against gitvetrix.test and the auto-filed bug-ticket path when it fails.

The plan it implements is plans/accessibility.md, §6 (verification checklist) and the "How to read this plan" preamble that lists axe / pa11y and keyboard-only verification as continuous-run expectations rather than one-shot checks.

Dependency note. The required-check contexts ci/a11y/axe-pa11y and ci/a11y/playwright are produced by the axe/pa11y gate job and the Playwright a11y job respectively. Until both jobs are reporting successfully on the protected branch, those contexts will be "never reported"; an MR adding this scaffolding must NOT be followed by a branch-protection apply until at least the axe/pa11y gate is green — otherwise no MR can merge. See §5 "Rollout" below.

1. Required checks on protected branches (AC 1)

The canonical configuration lives at deployments/branch-protection/vetrix-a11y.yml. Summary:

Branch Required reviewers Required status checks Restrict pushes
develop 1 ci/test, ci/web-lint, ci/a11y/axe-pa11y, ci/a11y/playwright no
master 2 ci/test, ci/web-lint, ci/a11y/axe-pa11y, ci/a11y/playwright yes

Apply with:

ISSUE_TRACKER_USER=vetrix-admin \
ISSUE_TRACKER_PASS=**** \
./scripts/a11y/apply-branch-protection.sh

The apply script is idempotent: it diffs desired state against GET /branches/protection and PUTs anything that drifted. Rerunning is safe.

A companion lint — ./scripts/a11y/lint-branch-protection.sh — walks the YAML and verifies every require_status_checks entry has an emitting job somewhere in the repo (vetrix-ci.yml + automations). Vetrix's API silently accepts unknown contexts, so the lint is the canonical check for typo drift.

2. Playwright a11y suite — required on web/ MRs

The Playwright a11y suite lives at web/playwright.config.ts (project a11y). Its CI counterpart runs inside the .vetrix/automations/web-ci.yml workflow. The job MUST emit the ci/a11y/playwright context for every MR, including MRs whose diff does not touch web/ — otherwise Vetrix's required- check evaluator treats the skip as a missing check and refuses to merge.

The workflow therefore declares two sibling jobs:

jobs:
  playwright-a11y:
    runs-on: linux
    if: ${{ contains(vetrix.changed_paths, 'web/') }}
    # real run
  playwright-a11y-skip:
    runs-on: linux
    if: ${{ !contains(vetrix.changed_paths, 'web/') }}
    steps:
      - name: Finalise context
        # Emits the same status context with conclusion=success so the
        # branch-protection evaluator treats the MR as having satisfied
        # the check.
        run: echo "ci/a11y/playwright skipped for non-web MR — reporting success"

Both jobs report the same context name; only one runs on any given MR.

3. Nightly full-route axe run (AC 2)

3.1 Schedule + trigger

  • Workflow: .vetrix/automations/a11y-nightly.yml
  • Cron: 0 3 * * * (03:00 UTC daily). Spaced from mirror/backup schedulers by a safe margin; tight enough to catch same-day content regressions.
  • Manual re-run: workflow_dispatch — the "Run workflow" button on the automation's UI page. An optional target_url input points the runner at a release-candidate deployment instead of the default staging host.

3.2 What the runner does

  1. Installs @axe-core/cli, pa11y, and Playwright with Chromium.
  2. Reads the authoritative route list from web/a11y-routes.json (authenticated + unauthenticated routes).
  3. For each route, requests the page via a headless browser, runs axe-core, records any serious or critical violation.
  4. Writes web/reports/a11y-nightly.json and web/reports/a11y-nightly.xml (JUnit for the artifacts UI).
  5. Exits non-zero if any serious/critical violation is present.

3.3 What happens on failure

The workflow's last step runs scripts/a11y/file-bug-ticket.sh when the axe step failed. The script:

  1. Authenticates to https://api.gitvetrix.com with the repository-scoped ISSUE_TRACKER_USER / ISSUE_TRACKER_PASS secrets.
  2. POSTs an issue titled [a11y][nightly] Regression on https://www.gitvetrix.test/ (YYYY-MM-DD) against rts2271/Vetrix containing the JSON violation summary, a link back to the failing run, and a pointer to §4 of this doc.
  3. Surfaces the full API response body on any non-2xx — both the login call and the issue POST. This is required: a silently-successful bug-ticket creator hides regressions worse than the axe failure itself.

3.4 Configuring the secrets

In the repo that owns the upstream mirror, add two CI secrets:

  • ISSUE_TRACKER_USER — Vetrix service account username (must have repo:issues:write on the target repo).
  • ISSUE_TRACKER_PASS — its password. Password-login is used (not a token) so the same credential pair works against both gitvetrix.com and gitvetrix.test without per-environment token rotation.

These secrets are wired into the workflow via secrets.ISSUE_TRACKER_USER / secrets.ISSUE_TRACKER_PASS — they never touch the nightly axe step itself, only the bug-ticket step.

4. Triage runbook

When an auto-filed ticket appears:

  1. Open the pipeline run (link in the ticket body).
  2. Inspect a11y-nightly-report artifact → a11y-nightly.json.
  3. Re-run the workflow once via workflow_dispatch to confirm the regression is reproducible, not a network/content flake.
  4. If reproducible:
    • File a child task against the owning component (UI primitive, shared shell, or page) and set its parent_number via PUT /issues/{n}/parent (PATCH is a silent no-op on this endpoint).
    • Link the root-cause fix MR to the ticket.
  5. If the regression is a known-flake class (third-party embed, iframe from a vendor whose markup we don't control), add the rule id + selector to web/a11y-routes.json#ignore and close the ticket wontfix. Never close silently without updating the allow-list — the next nightly will refile the same ticket.

Closing the ticket uses POST /api/v1/repos/{owner}/{repo}/issues/{n}/close (not PATCH).

5. Rollout

Ordered so a failure at any step is recoverable:

  1. Land the axe/pa11y gate workflow so ci/a11y/axe-pa11y starts reporting on the protected branch. Verify the context name matches deployments/branch-protection/vetrix-a11y.yml exactly.
  2. Land the Playwright a11y suite so ci/a11y/playwright reports (including the non-web-MR skip-finaliser described in §2).
  3. Land the regression-gating scaffolding — the nightly cron, bug-ticket script, branch-protection YAML, and lint script. No required checks change yet because the config hasn't been applied.
  4. Apply the branch-protection config. Run scripts/a11y/apply-branch-protection.sh against api.gitvetrix.com. This is the point where merges begin requiring the new contexts.
  5. Verify. Open a throwaway MR touching README; confirm all four contexts report and merge is gated accordingly.

If step 4 happens before step 1 or 2, no MR can merge to the protected branch — fix by temporarily removing the offending context from the live protection rule via the API, land the dependency, then reapply.

  • .vetrix/automations/a11y-nightly.yml — the nightly workflow.
  • scripts/a11y/file-bug-ticket.sh — bug-ticket helper.
  • scripts/a11y/apply-branch-protection.sh — apply the YAML.
  • scripts/a11y/lint-branch-protection.sh — catch context-name drift.
  • deployments/branch-protection/vetrix-a11y.yml — source of truth.
  • ../../user-docs/protected-branches.md — Vetrix API reference for branch rules.
  • plans/accessibility.md — the plan this gating enforces.