CI Runner Docker-daemon (DinD) requirement

Some pipeline jobs run docker inside the job container — they stand up sibling containers, build images, or push to a registry. The two such jobs in the canonical vetrix-ci.yml are:

go-integration-test — brings up an ephemeral postgres:16-alpine container with docker run -d, applies migrations, and runs the integration suite against it.
build-image — runs docker build / docker push to build and publish the server image.

Both jobs only work when the runner makes a Docker daemon reachable from inside the job container. The Vetrix worker does not provide one automatically. This document is the provisioning contract for self-hosted runners.

Audience: operators running their own (generic / self-hosted) v1 worker (cmd/worker). The commitshield dev-stack runners already satisfy this contract — see Verification. The CICDv2 control plane (runnerctl / host-agent) uses a different, outer-DinD model; see How this differs from CICDv2.

What the worker does — and does not — provide

The v1 worker executes each job with docker run against the daemon the worker process itself can reach (the DOCKER_HOST of the worker process; the platform default socket when unset). The full job-container docker run argv is assembled by buildDockerRunArgs in cmd/worker/docker_executor.go (vetrix backend repo). It emits only:

the per-job workspace bind mount (-v <host-workspace>:/workspace, when RUNNER_WORKSPACE_DIR is set), and
the canonical CI_* environment variables plus the job's declared variables:.

By default — i.e. unless RUNNER_DOCKER_SOCKET is set (below) — it does not:

bind-mount /var/run/docker.sock into the job container.

This default is intentional. The runner deliberately withholds host credentials — including the Docker socket — from job containers unless an operator explicitly provisions them. See the security notes in internal/cicd/runner.go (vetrix backend repo) ("Runner credentials (Docker socket, artifact keys) are never passed into the job container unless explicitly declared as secrets") and internal/runnerctl/executor/doc.go (vetrix backend repo) ("never to the host's docker socket").

Consequence: if a job runs docker and the runner has not been provisioned per this document, the job fails at its first docker command with Cannot connect to the Docker daemon.

Workspace git contract

The runner materialises the repo source into the per-job workspace (/workspace) before the container runs, when RUNNER_WORKSPACE_DIR and the repo's bare disk path are both available. There are two materialisation modes; both are implemented in cmd/worker/docker_executor.go (vetrix backend repo):

Mode	Trigger	What's in `/workspace`	`.git` present?
archive (default)	every job	repo source tree at the build ref, streamed via `git archive` (`populateWorkspace`)	No
git checkout (opt-in)	job sets `VETRIX_WORKSPACE_GIT` to a truthy value (`1`/`true`/`yes`/`on`)	a real working tree: local clone of the bare repo with the build commit checked out (`populateWorkspaceGit`)	Yes

Default mode is git archive. It is cheap and image-agnostic, but it extracts a snapshot of the tree — there is no .git directory. Any job command that invokes git (git rev-parse, git fetch, git diff, git log) will fail with fatal: not a git repository / not inside a git work tree. The archive mode is correct for the common case (compile, lint, run tests against the checked-out source); it is not suitable for jobs that need git history.

Git-dependent jobs must opt in. A job that runs git declares the variable in its variables: block:

my-git-job:
  variables:
    VETRIX_WORKSPACE_GIT: "1"
  commands:
    - git rev-parse HEAD          # works: .git is present
    - git fetch origin develop    # works: origin = the bare mirror
    - git diff origin/develop...HEAD -- 'internal/api/*.go'

In git-checkout mode the runner does a --local --no-checkout clone of the repo's bare mirror into the workspace and checks out the build commit. Because origin points at that mirror, base refs are fetchable (`git fetch origin

` and `origin/` resolve) without network or credentials. The canonical consumer is `router-test-coverage` (it diffs `HEAD` against `origin/develop`); see its job definition and `scripts/check-router-test-coverage.sh` (vetrix backend repo).

The clone is --local, so it reuses the bare repo's object store (hardlinks where the filesystem allows) — it is not a network operation and adds negligible cost beyond the archive path. It is opt-in only so jobs that don't need git don't pay for the .git directory.

`RUNNER_DOCKER_SOCKET` — opt-in socket pass-through

The worker exposes one explicit, opt-in knob to satisfy option (a) below:

Env var	Default	Meaning
`RUNNER_DOCKER_SOCKET`	(unset)	Host path of the Docker socket to bind-mount into every job container, at the canonical container path `/var/run/docker.sock`. Set it to `/var/run/docker.sock` for the standard host socket. Empty = no socket mounted (the withholding default above).

When set, the worker appends -v <RUNNER_DOCKER_SOCKET>:/var/run/docker.sock to the docker run argv it builds for each job (buildDockerRunArgs in cmd/worker/docker_executor.go (vetrix backend repo), wired from RUNNER_DOCKER_SOCKET in cmd/worker/main.go (vetrix backend repo)). The container-side path is fixed at the platform-default /var/run/docker.sock regardless of the host path, so a job's bare docker CLI and build-image's DOCKER_HOST: "unix:///var/run/docker.sock" both reach the host daemon with no further wiring. This is the sibling-container DinD posture of option (a).

Security: a job that can reach the host socket has root-equivalent access to the runner host (see option (a)). Enable RUNNER_DOCKER_SOCKET only on runners that exclusively execute trusted, first-party pipelines. For untrusted jobs use option (b) instead and leave RUNNER_DOCKER_SOCKET unset.

Note: build-image sets a job-level DOCKER_HOST: "unix:///var/run/docker.sock". The worker forwards that as an -e DOCKER_HOST=... env var; for it to resolve, the socket must actually be present at that path inside the container — which is exactly what RUNNER_DOCKER_SOCKET (option (a)) provides. Option (b) instead overrides DOCKER_HOST to a tcp://… daemon via runner-side secret injection.

`RUNNER_JOB_NETWORK` — opt-in job-container network attachment

The worker exposes a second explicit, opt-in knob (mirroring RUNNER_DOCKER_SOCKET) that provides the dev-stack reachability attachment as a repo-configurable env var:

Env var	Default	Meaning
`RUNNER_JOB_NETWORK`	(unset)	Name of a Docker network to attach every launched job container to, via `--network <name>`. Set it to the dev-stack bridge that carries nginx `172.18.16.10` so e2e/a11y jobs can reach `E2E_BASE_URL`. Empty = no `--network` flag; the container joins Docker's default bridge (unchanged behaviour).

When set, the worker appends --network <RUNNER_JOB_NETWORK> to the docker run argv it builds for each job (buildDockerRunArgs in cmd/worker/docker_executor.go (vetrix backend repo), wired from RUNNER_JOB_NETWORK in cmd/worker/main.go (vetrix backend repo)). The launched job container is then attached to that named network, so it can reach the dev-stack nginx at 172.18.16.10 / E2E_BASE_URL without a manual docker network connect per job — exactly the network half of the reachability contract below. When empty (the default) no flag is emitted and the container runs on Docker's default bridge, so existing deployments are unaffected.

Note: RUNNER_JOB_NETWORK covers the network attachment half of reachability. The name resolution half — www.gitvetrix.test resolving to 172.18.16.10 inside the job — is still an operator concern (extra_hosts/dnsmasq, or pointing E2E_BASE_URL at the IP), as the network name alone does not add the host alias.

Runner identity — `RUNNER_ID`

Each runner registers a row in ci_runners keyed by its name. UpsertRunnerHeartbeat upserts ON CONFLICT (name), so a runner that keeps a stable name reuses its existing row across restarts. A runner whose name changes on every recreate instead leaves the old row behind as a stale, permanently-offline entry — over time the table fills with dead rows.

Operators MUST set a stable RUNNER_ID in production. A good value is a short, human-meaningful, per-runner-unique name such as runner-prod-01.

The worker (cmd/worker) resolves its identity in this order (see resolveRunnerID in cmd/worker/runner_id.go (vetrix backend repo)):

RUNNER_ID env var — if set (non-empty after trimming), it wins unconditionally. This is the recommended production mechanism.
Persisted id — otherwise the worker reads a runner-<uuid> from a runner_id file under RUNNER_STATE_DIR (falling back to RUNNER_WORKSPACE_DIR when RUNNER_STATE_DIR is unset). On first run it generates the id and writes it there; subsequent runs read it back. This keeps identity stable across container recreates as long as that directory lives on a persistent volume. An empty/corrupt file self-heals (regenerated). If the directory is unusable (cannot create/read/write), the worker logs a warning and falls through to step 3.
Legacy hostname fallback — otherwise runner-<hostname>. This is non-deterministic: container hostnames change on recreate, so every recreate registers a NEW ci_runners row. The worker logs a warning when it takes this path. Do not rely on it in production.

Why not just hostname? With the hostname fallback, container recreates (deploys, restarts, host migrations) produce a fresh hostname each time, so the ci_runners table accumulates offline rows with no stable identity. Setting RUNNER_ID — or mounting a persistent RUNNER_STATE_DIR — avoids this.

Provisioning options

Pick one. Both make a Docker daemon reachable from inside the job container; they differ in security posture.

Option (a) — bind-mount the host Docker socket

Set RUNNER_DOCKER_SOCKET=/var/run/docker.sock on the worker process (see the table above). The worker then bind-mounts that host socket into every job container at /var/run/docker.sock. The job talks to the host's daemon and spawns sibling containers next to itself.

The worker process itself must of course be able to reach that socket: either run the worker on a host where the path exists, or run the worker in a container that already has the socket mounted in (so the path it passes through to jobs is one it can itself see). The two are independent — DOCKER_HOST controls which daemon the worker talks to; RUNNER_DOCKER_SOCKET controls what gets mounted into jobs. For the common sibling-DinD setup both point at the same host socket.

This matches what the in-tree jobs assume: go-integration-test discovers the ephemeral Postgres by its bridge-network container IP (siblings on the same daemon), and build-image points DOCKER_HOST at unix:///var/run/docker.sock. Both jobs install their own docker CLI into the job image at runtime (go-integration-test: apt-get install -y docker.io; build-image: the docker:25-cli base image plus apk add curl), so the only thing the runner must add is the socket — which RUNNER_DOCKER_SOCKET does.

Security trade-off — root-equivalent host access. A job that can talk to the host's Docker socket can start a privileged container, bind-mount /, and escalate to root on the host. Treat any pipeline that reaches the host socket as fully trusted code running as host root. Only use this on runners that exclusively execute trusted, first-party pipelines (which is the posture of the commitshield dev-stack runners).

Option (b) — sidecar / TCP `DOCKER_HOST` daemon

Provision a separate Docker daemon (a Docker-in-Docker sidecar, or a dedicated daemon host) and point the job at it over TCP by exporting DOCKER_HOST=tcp://<daemon-host>:2376 (use the runner-side secret-injection mechanism, see pipeline-reference.md → Secrets). The job's docker CLI then drives that daemon instead of the host's.

Security trade-off — network exposure to guard. A TCP Docker daemon is an unauthenticated root shell for anyone who can reach the port unless you put TLS client-certificate auth in front of it (tcp://…:2376 + DOCKER_TLS_VERIFY=1

DOCKER_CERT_PATH). Never expose 2375 (plaintext, no auth). Bind the daemon to a private network segment reachable only by the runner, and rotate the client certs. This option keeps the host's daemon out of reach of job code — the blast radius is the sidecar daemon, not the runner host — at the cost of a network surface you must lock down.

Dev-stack runner provisioning (reproducible)

The commitshield dev-stack runner (mydev_vetrixworker, cmd/worker) uses option (a). The RUNNER_DOCKER_SOCKET knob mounts the socket into jobs; without it build-image fails with Cannot connect to the Docker daemon and go-integration-test fails with docker: not found / postgres did not become ready in 60s. Provisioning the dev stack to satisfy the DinD contract is exactly two host-side facts:

Mount the host Docker socket into the worker container so the worker process can reach the daemon at all. In the dev-stack compose service for mydev_vetrixworker:

services:
  mydev_vetrixworker:
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    environment:
      # pass the socket through to every job container.
      RUNNER_DOCKER_SOCKET: /var/run/docker.sock

Set RUNNER_DOCKER_SOCKET=/var/run/docker.sock on that same service (shown above). This is the only setting required; without it the worker does not mount the socket into jobs.

That is the complete contract for the dev stack: the worker can reach the daemon (volume), and it forwards the socket into jobs (RUNNER_DOCKER_SOCKET). go-integration-test and build-image install their own docker/curl into the job image at runtime (see option (a)), so no job-image change is needed.

The compose file lives outside this repo (it is dev-ops infrastructure), so the YAML above is the contract an operator applies, not a file checked in here. The repo-side half of the fix — reading RUNNER_DOCKER_SOCKET and emitting the -v mount — is in cmd/worker.

Dev-stack e2e/a11y job reachability

The browser-driven jobs — a11y-gate, web-a11y-e2e, security-scannerfix-cross-browser, repositories-landing-responsive, slow-chromium-e2e (vetrix-ci.yml) — do not stand up their own stack. They run their suites against the already-running dev stack at E2E_BASE_URL (https://www.gitvetrix.test). Each job's first substantive step is bash scripts/cicd/e2e-bootstrap.sh, which (a) verifies the Docker socket with docker version and (b) polls E2E_BASE_URL until it answers.

For step (b) to succeed, the job container must be able to resolve and route www.gitvetrix.test to the dev-stack nginx at 172.18.16.10. A bare job container is on the default Docker bridge and cannot — which is the stack did not become healthy in 120s symptom. The repo side provides the bash bootstrap (no daemon-start, poll-and-fail-fast); the operator must apply the network half on the dev-stack host. Pick one:

Attach job containers to the dev-stack network (preferred). The worker bind-mounts the host socket and spawns sibling job containers on that same daemon (option (a) above). Put those siblings on the bridge that carries nginx 172.18.16.10. Two host-side facts:
- The dev-stack network (the one mydev_nginx / 172.18.16.0/24 sits on) must be reachable from the job containers. The simplest action is to set RUNNER_JOB_NETWORK to that bridge, so the worker appends --network <name> to each job's docker run (buildDockerRunArgs in cmd/worker/docker_executor.go (vetrix backend repo)). Without that knob the operator must instead run the dev-stack containers and the job containers on the same user-defined bridge, or docker network connect <devstack-net> <job-container> for the job's lifetime.
- www.gitvetrix.test must resolve to 172.18.16.10 from inside the job container. On the dev-stack network that is automatic if nginx is reachable by IP and the job sets E2E_BASE_URL to the IP, OR the operator adds a host alias. The reproducible knob is documented below.
Inject a host alias mapping www.gitvetrix.test → 172.18.16.10. Independent of networking, ensure DNS//etc/hosts inside the job resolves the .test name to nginx. On the dev-stack host this is the same extra_hosts / dnsmasq entry the rest of the stack already uses for *.gitvetrix.test.

Reproducible operator contract (dev stack). The job container needs, in addition to the socket already documented above:

# dev-ops infrastructure (compose lives outside this repo) — the CONTRACT an
# operator applies so e2e/a11y jobs can reach the live stack:
#
# (a) the job's sibling containers join the dev-stack bridge that carries
#     nginx 172.18.16.10 (same network as mydev_nginx), AND
# (b) www.gitvetrix.test resolves to 172.18.16.10 inside the job
#     container (the *.gitvetrix.test dnsmasq/extra_hosts mapping the
#     dev stack already ships).

The worker provides a first-class RUNNER_JOB_NETWORK knob (documented above) for the network half of this attachment: set it to the dev-stack bridge that carries nginx 172.18.16.10 and the worker appends --network <name> to each job's docker run, so the per-job docker network connect is no longer required. The name resolution half (item 2 above — www.gitvetrix.test → 172.18.16.10 inside the job) remains an operator step. The repo side still fails fast with a pointer to this section (scripts/cicd/e2e-bootstrap.sh) when the route is missing, so a mis-provisioned runner is obvious instead of timing out opaquely.

Why not docker compose up a fresh stack in-job? Standing up a fresh stack in-job would run docker compose -f docker-compose.yml up -d web api postgres against the frontend repo's compose file, not this one's, and standing up a second stack per job is wasteful and racy on a shared runner. Targeting the already-running dev stack at E2E_BASE_URL is both correct and cheaper.

web/ (frontend) dependency. The four Playwright suite jobs run npm/npx playwright against the frontend web/ tree, which is not in this backend repo. The operator must make web/ available to those jobs' workspace (a checkout/mount of the vetrix-frontend repo). The bootstrap-shell + reachability behavior is independent of and complete without that; the suites only execute once web/ is present. a11y-gate's scanner (scripts/a11y/gate.mjs) is in this repo but still needs the axe/pa11y/playwright npm deps from the frontend web/ package.

Verification

Both go-integration-test and build-image pass on develop / master on a correctly-provisioned runner. The commitshield dev-stack runners provide a reachable Docker daemon (option (a) posture).

To confirm a new self-hosted runner satisfies the contract (a daemon reachable from inside the job container, via option (a) or (b) above) before relying on it, run a one-off job whose commands: is a single docker version — the same fail-fast precondition build-image uses. A clean Client + Server handshake means the contract is met; Cannot connect to the Docker daemon means it is not.

Repo-side coverage. The launch-config behavior — that the worker emits -v <host-socket>:/var/run/docker.sock when RUNNER_DOCKER_SOCKET is set, and emits no socket mount when it is unset — is unit-tested without a live daemon in cmd/worker/docker_executor_test.go (vetrix backend repo) (TestBuildDockerRunArgs_MountsDockerSocketWhenConfigured, TestBuildDockerRunArgs_DockerSocket_HostPathMappedToCanonicalContainerPath, TestBuildDockerRunArgs_NoDockerSocketByDefault, TestBuildDockerRunArgs_WorkspaceAndSocketCoexist, TestDockerExecutor_Run_MountsConfiguredDockerSocket). These pin the argv contract; the live docker version handshake above is the end-to-end check once a runner is provisioned per the dev-stack section.

Runner registration, heartbeats, and the stale-runner reaper

Each v1 worker self-registers in the ci_runners table on startup and keeps a liveness signal current. The lifecycle:

On startup the worker upserts its ci_runners row (keyed by RUNNER_ID) to status='idle' and stamps last_seen_at.
Every 15 s it sends a heartbeat, refreshing last_seen_at and setting status to busy (jobs in flight) or idle.
On graceful shutdown (SIGINT / SIGTERM) it flips its own row to status='offline'.

A worker that dies ungracefully — OOM kill, SIGKILL, host crash, network partition — never runs that shutdown step, so its row is stranded at idle/busy with a frozen last_seen_at. Such rows pollute runner-availability views and mask dead runners.

The reaper sweep

Every worker runs a periodic stale-runner reaper that calls MarkStaleRunnersOffline: any ci_runners row whose last_seen_at is older than the staleness threshold (or NULL) and is still idle/busy is flipped to offline. The UPDATE is idempotent and cluster-wide — it reaps any stale runner, not just the local worker's own row — so running it on every worker is safe; redundant passes are no-ops. The sweep runs one immediate pass on startup, then on the configured interval, and exits cleanly when the worker receives a shutdown signal (it shares the worker's cancellation context).

Worst-case detection latency is threshold + interval.

Env var	Default	Meaning
`RUNNER_STALE_THRESHOLD`	`90s`	How old a runner's `last_seen_at` may get before it is reaped. The default tolerates ~6 consecutive missed 15 s heartbeats — past a transient stall or a slow DB round-trip, but far short of the multi-day staleness this guards against.
`RUNNER_STALE_SWEEP_INTERVAL`	`30s`	How often the reaper runs. Independent of the threshold.

Both accept Go duration strings (e.g. "2m", "45s"). An invalid or non-positive value logs a warning and falls back to the default. The defaults live in internal/cicd/stale_runner_reaper.go (vetrix backend repo) (DefaultStaleRunnerThreshold, DefaultStaleRunnerSweepInterval); the wiring is in cmd/worker/main.go (vetrix backend repo).

Pipeline reconcile sweep

The runner rolls a pipeline up to success/failed (with finished_at set) after each job's terminal transition — but that only covers transitions the current worker observes. A pipeline whose jobs all finished under a previous runner — the worker was restarted after the last job's terminal event, or that event was missed — never receives a reconcile call and stays state='running', finished_at=NULL even though every job is terminal.

Every worker runs a pipeline reconcile sweep that closes this gap. It lists every pipeline in state='running' (Store.ListRunningPipelineIDs, cluster-wide) and calls the same idempotent Engine.ReconcilePipeline the runner uses inline. That method owns the terminal decision and is reused, not duplicated:

any job still pending/running (or no jobs) → left running;
all jobs terminal, any failed/cancelled → pipeline failed;
all jobs terminal, every one success → pipeline success.

The sweep runs one immediate pass on startup — this recovers across-restart orphans — then on the configured interval, and exits cleanly on the worker's shutdown signal (it shares the cancellation context). It is idempotent and cluster-wide, so running it on every worker is safe; a still-running pipeline is a no-op, and a redundant pass cannot re-terminalise an already-terminal pipeline. A per-pipeline reconcile error is logged and skipped so a sibling orphan still gets its chance in the same pass.

Env var	Default	Meaning
`PIPELINE_RECONCILE_SWEEP_INTERVAL`	`60s`	How often the reconcile sweep runs. This is a recovery path (the runner reconciles live transitions inline), so a minute-scale cadence is ample; a sweep also always runs immediately on startup.

Accepts Go duration strings (e.g. "2m", "30s"). An invalid or non-positive value logs a warning and falls back to the default. The default lives in internal/cicd/pipeline_reconcile_sweep.go (vetrix backend repo) (DefaultPipelineReconcileSweepInterval); the wiring is in cmd/worker/main.go (vetrix backend repo).

How this differs from CICDv2

The CICDv2 control plane (runnerctl + host-agent) does not use the mechanism above. There, runnerctl runs inside a per-host privileged outer DinD container and the executor talks to that outer daemon's socket — never the host's — to spawn the inner job container. See internal/runnerctl/executor/doc.go (vetrix backend repo) ("DinD posture") and the operator runbook (now maintained in runbook-docs) §1 for that provisioning path. This document covers only the v1 worker (cmd/worker).

The v2 controller that would have scheduled onto that host pool was sunset and its dispatch-side code removed, so the v1 worker is the only path that executes jobs. For that story, and for why the runner_hosts DinD host pool is expected to be empty, see dispatchers.md.