Worker Docker-socket access (group_add GID)

Worker Docker-socket access (`group_add` GID)

Audience: operators deploying the containerized Vetrix CI worker (appmerc_vetrixworker / vetrix-worker image, compose --target worker).

This supersedes the generic usermod -aG docker advice in runners.md for containerized worker deployments. In a container the in-image group name is irrelevant; only the numeric GID the runtime grants matters (see below).

Why this matters

The worker does not talk to Docker over an API token — it shells out to the host docker CLI to launch each pipeline job container. To do that it must reach the bind-mounted daemon socket:

volumes:
  - /var/run/docker.sock:/var/run/docker.sock
environment:
  - RUNNER_DOCKER_SOCKET=/var/run/docker.sock

On a typical host /var/run/docker.sock is owned root:docker, mode 660 — i.e. not world-accessible. The worker container runs unprivileged (image default USER vetrix, pinned at runtime to the host identity via user: "${DOCKERUSERID}:${DOCKERGROUPID}"). The Linux kernel authorises socket access by numeric GID: the worker process must carry, as a supplementary group, the exact numeric GID that owns the socket on this host. That is the entire job of the group_add: entry.

Group membership is by GID number, not name. Creating a docker group inside the image (the DOCKER_GID build-arg) only helps if its GID happens to equal the host socket's GID. The authoritative, host-portable mechanism is the runtime group_add: numeric GID below.

Symptoms of a misconfigured GID

Symptom	Meaning
Every job (incl. trivial `hello-world` on `alpine`) fails ~1s in with `exit_code=126`, job log shows `docker: permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock … connect: permission denied`	The worker is running and claiming jobs, but its `group_add` GID does not match the socket's owning GID. ← this doc fixes that.
All jobs stuck `pending`, `runner_id: null`, never start	The worker process isn't running/polling at all — usually the container failed to boot (see "Do not drop `user:`" below), not a socket-GID problem.

The hello-world job is the canary: pure echo, no git/registry/source/service/DB dependency. If it fails at exit 126, the fault is in container launch (the socket), never the job logic.

Step 1 — Find the socket's owning GID (per host)

Run on the Docker host (not inside a container):

stat -c '%g' /var/run/docker.sock      # the numeric GID that owns the socket  ← use THIS
getent group docker | cut -d: -f3      # usually the same; the docker group's GID

Use the number from stat. Common values are distro/install-specific (999, 998, 988, …) and change after a Docker package/daemon reinstall — never assume or hardcode a "known good" value.

Step 2 — Set `group_add` on the worker service

appmerc_vetrixworker:
  user: "${DOCKERUSERID}:${DOCKERGROUPID}"   # REQUIRED — see Step 3
  volumes:
    - /var/run/docker.sock:/var/run/docker.sock
  environment:
    - RUNNER_DOCKER_SOCKET=/var/run/docker.sock
  group_add:
    - "<GID from Step 1>"                     # e.g. "999" — must equal stat output

Prefer not to hardcode it across hosts. Drive it from the environment so each host gets its own value:

  group_add:
    - "${DOCKER_SOCKET_GID}"

# in the deploy .env (or export before `docker compose up`):
DOCKER_SOCKET_GID=$(stat -c '%g' /var/run/docker.sock)

Step 3 — Do not drop the `user:` line to "work around" socket perms

Removing user: "${DOCKERUSERID}:${DOCKERGROUPID}" does not grant socket access (the image's default vetrix user still isn't in the host docker GID) and it breaks the worker's boot: the worker writes durable logs to RUNNER_LOG_DIR=/var/log/vetrix-worker and uses a workspace dir — both bind-mounted from host paths owned by ${DOCKERUSERID}:${DOCKERGROUPID}:

  volumes:
    - ./site/vetrix/worker-logs:/var/log/vetrix-worker
    - ./site/vetrix/data/runner-ws:/…/runner-ws

If the container runs as any other uid it can't write those dirs → the worker fatals at startup → it never polls → all jobs sit pending. Keep user: pinned; fix the socket via group_add only.

Step 4 — Apply and verify

docker compose up -d appmerc_vetrixworker
docker compose logs --tail=30 appmerc_vetrixworker     # expect a clean "worker starting"
docker compose exec appmerc_vetrixworker id            # supplementary groups must include the Step-1 GID
docker compose exec appmerc_vetrixworker docker info   # must succeed — NO "permission denied"

Then trigger a pipeline (push to a built branch) and confirm hello-world reaches succeeded and emits its echo output.

Why it "works on local dev" but not prod

The compose file is portable; host identities are not. Two per-host facts differ:

Socket GID — dev's /var/run/docker.sock GID matches what its worker carries (or the dev socket is permissive); prod's is a different root:docker GID → exit-126 on prod only.
Bind-mount dir ownership — dev dirs are writable by whatever the worker runs as; prod dirs are strictly owned by ${DOCKERUSERID}, so the user: pin is load-bearing there.

Always derive the GID on the host you are deploying to (Step 1). A value copied from dev — or a guessed constant — is the usual root cause of a prod-only CI outage.

Troubleshooting quick table

You see	Fix
`exit_code=126`, `docker: permission denied … docker.sock`	`group_add` GID ≠ host socket GID → set it to `stat -c '%g' /var/run/docker.sock`, recreate.
Worker container Exited/Restarting; logs show a fatal or no `worker starting`	Likely `user:` removed or bind-mount dir ownership mismatch → restore `user:`, ensure host dirs are owned by `${DOCKERUSERID}:${DOCKERGROUPID}`.
All jobs `pending`, 0 claimed	Worker not polling → check it's running the `worker` build target and booted cleanly.
Socket GID changed after a Docker upgrade	Re-run Step 1 and update `group_add`; this is expected after daemon/package reinstalls.

See also: pipeline-variables.md (per-branch CI/CD variables and the worker SECRET_ENC_KEY requirement).