Vetrix Docs

Runbook — accessibility regression gate

This runbook covers the CI accessibility gate that scans every route in scripts/a11y/routes.json (every page in plans/accessibility.md §4) through @axe-core/playwright plus a pa11y cross-check.

The gate is wired in vetrix-ci.yml as the a11y-gate job. It runs on every push and merge request, and fails the pipeline on any Serious or Critical axe violation.

What the gate touches

  • vetrix-ci.yml — a11y-gate job (Vetrix schema; no GitLab keys)
  • docker-compose.yml (repo root) — dev stack with postgres, api, web services. Aliased to db and server so the prod override (deployments/docker/docker-compose.prod.yml) still applies.
  • scripts/a11y/gate.mjs — the driver
  • scripts/a11y/routes.json — the route list (101 routes; §4.1 – §4.14)
  • web/package.json — npm run test:a11y
  • web/a11y-report/ — output artifact directory (uploaded by CI)

Bringing the stack up locally

The gate scans whatever A11Y_BASE_URL resolves to. For a local run against https://www.gitvetrix.test:

# From the repo root
docker compose -f docker-compose.yml up -d web api postgres

# Wait for the web service to answer
for i in $(seq 1 60); do
  curl -skf https://www.gitvetrix.test/privacy >/dev/null && break
  sleep 2
done

# Run the gate (axe + pa11y, all 101 routes)
cd web && npm ci && npx playwright install --with-deps chromium
npm run test:a11y

The docker-compose.yml at the repo root is the base file — the production override (deployments/docker/docker-compose.prod.yml) extends it. Bringing the stack up without the base file fails because the prod override targets services named db and server. The base file aliases postgres → db and api → server so both names resolve.

Output artifacts

The driver writes three files under web/a11y-report/:

File Purpose
summary.json machine-readable per-route axe + pa11y findings
summary.html human-readable report (open in a browser)
junit.xml JUnit-formatted failure list for downstream tooling
docker.log docker-compose logs for web/api/postgres, ALWAYS
captured (success and failure) so an a11y regression
can be told apart from a stack bring-up failure

docker.log lives inside web/a11y-report/ rather than at the repo root. The Vetrix pipeline schema only supports artifacts.paths: (no artifacts.when:, no artifacts.reports.junit:, no artifacts.expire_in:); shipping the log alongside the rest of the report avoids needing extra artifact paths the parser would reject.

Why the CI YAML looks the way it does

vetrix-ci.yml carries no top-level version: key, so it is parsed against the v1 schema (internal/cicd/parser/v1.go; the version dispatcher that selects v1 vs v2 is internal/cicd/parser/parser.go). Everything in this section describes v1.

The v1 parser accepts only:

  • top-level: stages, variables, branch_order, version, plus job-name keys
  • per-job: stage, image, commands, variables, only, except, secret, environment, allow_failure, artifacts.paths

It explicitly rejects before_script, after_script, script, services, artifacts.when, artifacts.reports, artifacts.expire_in, the top-level GitLab keys workflows, include, and default (incompatibleTopLevelKeys in internal/cicd/parser/v1.go), plus the GitHub-Actions keys on, runs-on, steps, jobs. Using any of these GitLab- or GitHub-isms in vetrix-ci.yml causes the parser to reject the pipeline at trigger time.

allow_failure: is not on that rejection list. It is a first-class boolean job field, valid under both the v1 and the v2 schema: parsed by both (rawJob.AllowFailure in internal/cicd/parser/v1.go, rawJobV2.AllowFailure in internal/cicd/parser/v2.go), defaulting to false in both, persisted to the pipeline_jobs.allow_failure column, and read back by the engine for the soft- vs hard-failure split (jobOutcome, defined in internal/cicd/engine_webhook.go and called from internal/cicd/engine.go). A non-boolean value is a parse error under both schemas. For the full soft-failure semantics see pipeline-reference.md § Tolerating job failures.

Three quirks of the rewrite to be aware of when you edit it:

  1. No before_script. Stack bring-up, the health-check loop, npm ci, and playwright install are inlined into the job's commands: list.
  2. No after_script. Diagnostics capture (docker compose logs) is collapsed into the same shell expression as the gate run, with set +e + exit $rc so the gate's exit code propagates while the logs are still always captured. Vetrix's worker joins commands: with && (buildShellScript in cmd/worker/docker_executor.go), so a bare line after a failing command would be skipped.
  3. No services: docker:dind. The runner image must already provide a working docker compose CLI against a reachable daemon; the default Vetrix runner mounts the host docker socket.

Environment overrides

All inputs are env-var driven on the gate side:

Env var Default Effect
A11Y_BASE_URL from routes.json Override the scan target
A11Y_USER kcoder Login username
A11Y_PASS (dev seed fixture — see below) Login password
A11Y_REPORT_DIR a11y-report Output directory
A11Y_FAIL_ON serious,critical Comma-list of axe impact levels that gate
A11Y_ROUTES (empty — all) Comma-list of route ids to scan
A11Y_SKIP_PA11Y (unset) 1 to skip pa11y (axe-only fast iteration)
A11Y_TIMEOUT_MS 30000 Per-route navigation timeout

A11Y_PASS has no literal default in this table on purpose. The value is a development seed fixture — not a live credential — committed across the backend's test/bench fixtures (vetrix backend repo: bench/, scripts/transfer-fixtures/, tests/web/selenium/, and an authapi integration test) for local development and CI use. This page follows the pattern in authoring-vetrix-pipelines.md § Variables, interpolation & secrets: never inline a secret-shaped value, even a fixture one, so a reader cannot copy a stale or wrong value out of the docs. Set A11Y_PASS from the runner-host secret binding (CI) or the dev-stack bootstrap seed (local) — see playwright-stack-target.md § Credentials for how the equivalent Playwright-side credential is sourced.

For a quick local pass against three routes:

A11Y_ROUTES=login,welcome,repositories A11Y_SKIP_PA11Y=1 \
  npm --prefix web run test:a11y

Triage workflow

  1. Pipeline failure on a11y-gate — open the artifact bundle, read web/a11y-report/summary.html. Failing routes carry a red badge.
  2. Suspect a stack bring-up issue, not a real a11y regression — read web/a11y-report/docker.log. Look for non-2xx responses on /privacy or postgres init errors.
  3. A pa11y-only finding — pa11y is a cross-check; its issues are reported in summary.json under each route's pa11yIssues but do NOT gate. They land in the report for triage only.
  4. Want to land a fix and re-run — push the fix, the gate runs on the next CI build. To run locally first, follow the "Bringing the stack up locally" section.

Adding a new §4 route

  1. Add an entry to scripts/a11y/routes.json under either authenticated or unauthenticated. Required fields: id (kebab-case, unique), path (rooted with /, may use {placeholder} tokens against the fixtures block), and section (the 4.N subsection it inherits from).
  2. If the new path uses a placeholder not yet in fixtures, add one that resolves against the deterministic seed repo. The schema-guard test (web/src/__tests__/lib/a11yRoutesSchema.test.ts) fails any PR that adds a {placeholder} without a matching fixture.
  3. Run the schema-guard test: npm --prefix web test -- a11yRoutesSchema.
  4. Optionally do a one-route smoke pass: A11Y_ROUTES=<your-id> A11Y_SKIP_PA11Y=1 npm --prefix web run test:a11y.