CICDv2 operator runbook

Operator-facing procedures for the Vetrix CICDv2 self-hosted execution fleet. The runbook is grouped by operational verb; each section is self-contained and may be run without reading the others.

Section	Verb
§1 Install host-agent and attach a new host	mint attach token + register
§2 Graceful drain + decommission	drain a host, then remove it from the fleet
PKI rotation	rotate leaf certs / replace CA
Push auto-trigger + rollout	enable push-triggered pipelines + gate develop/staging

1. Install host-agent and attach a new host

This section walks an operator through bringing a fresh Linux VM into the CICDv2 fleet end-to-end:

Verify the host meets the prereqs.
Install the host-agent binary and its systemd unit.
Mint a one-time host_attach_token against the tenant API and run the install snippet on the new host (the attach-token flow).
Verify the host appears in the fleet and can run a no-op job.
Use §1.6 troubleshooting if any step fails.

Source references for this section: cmd/host-agent/, internal/api/hosts_attach_vbe261.go (POST /api/v1/{owner-scope}/hosts:attach), internal/api/hosts_detach_vbe262.go (DELETE /api/v1/{owner-scope}/hosts/{id}), internal/api/internal_host_pool_register.go (POST /internal/host-pool/register), and the host minimum requirements analysis in the system-docs repo.

1.1 Prereqs

A Vetrix CICDv2 host runs the host-agent (one process per node) and spawns Docker-in-Docker job containers under cgroup v2 limits. The minimum supportable node is:

Surface	Minimum	Recommended for ~8 concurrent jobs
Architecture	`x86_64` (amd64)	`x86_64`
vCPU	4	12
RAM	8 GiB	32 GiB
Disk	60 GiB free (10 GiB `/`, 30 GiB `/var/lib/docker`, 5 GiB `/var/log/vetrix*`, 1 GiB `/var/lib/vetrix/host-agent`)	200 GiB SSD
Linux kernel	>= 5.10 (cgroup v2 unified, overlayfs, user namespaces)	5.15+ (Ubuntu 22.04 LTS, RHEL 9 with AppArmor)
Docker Engine	>= 24.0 (storage driver `overlay2`)	25.x / 26.x
nftables	`nft` binary >= 0.9	1.0+
LSM	AppArmor enabled + active; seccomp v2 (`CONFIG_SECCOMP_FILTER=y`)	same
Capabilities for host-agent service user	`CAP_NET_ADMIN`, `CAP_SYS_ADMIN`, `CAP_NET_RAW` (or run as root)	same
Outbound network	HTTPS + mTLS to controller; HTTPS to MinIO; no listening port required	same
Clock sync	NTP / chrony enabled; skew vs controller < 60 s	< 5 s

Verify the local kernel and Docker versions before installing:

uname -r                                              # need >= 5.10
stat -f -c %T /sys/fs/cgroup                          # must be cgroup2fs
docker version --format '{{.Server.Version}}'         # need >= 24.0
docker info --format '{{.Driver}}'                    # must be overlay2
nft --version                                         # need >= 0.9
aa-status >/dev/null && echo apparmor-ok              # must report enabled
awk '/Seccomp:/ {print $2}' /proc/self/status         # must be > 0

The full requirements analysis (per-job RAM model, tmpfs scratch budget, registration-time preflight matrix) lives in the host minimum requirements analysis in the system-docs repo. That document is the source of truth — this runbook restates the operator-facing minimum and leaves the engineering rationale there.

1.2 Install the host-agent binary

1.2.1 Get the binary

On this rc only the build-from-source path is supported. The controller-served download endpoint is planned but not yet shipped (see callout below).

Build from source on a node with Go >= 1.25 toolchain available:

sudo install -d -o root -g root -m 0755 /var/lib/vetrix/host-agent/src
sudo git clone vetrix-git-kcoder:vetrix/vetrix.git /var/lib/vetrix/host-agent/src
sudo env -C /var/lib/vetrix/host-agent/src go build -o /usr/local/bin/host-agent \
    -ldflags "-X main.version=$(git -C /var/lib/vetrix/host-agent/src describe --tags --always) -X main.commit=$(git -C /var/lib/vetrix/host-agent/src rev-parse --short HEAD)" \
    ./cmd/host-agent
sudo chmod 0755 /usr/local/bin/host-agent
/usr/local/bin/host-agent version

Expected output (host-agent version is the documented invocation for fleet-inventory tooling and prints on a single line):

host-agent v0.12.x (abcdef0)

Planned — pre-built release download. A future phase will serve signed binaries from https://api.gitvetrix.com/dl/host-agent/<version>/linux-amd64/host-agent. Until that ships, build from source on each host or distribute the binary out-of-band from your own artefact store. Once the endpoint is live this section will gain a curl-style snippet alongside the build-from-source path; do not assume the URL above is reachable on this rc.

1.2.2 Create the service user and directories

The host-agent should not run as root in steady state. Create a dedicated unprivileged service user with the capabilities the runtime needs:

sudo useradd --system --home /var/lib/vetrix/host-agent --shell /usr/sbin/nologin vetrix-agent
sudo install -d -o vetrix-agent -g vetrix-agent -m 0750 /etc/vetrix
sudo install -d -o vetrix-agent -g vetrix-agent -m 0700 /etc/vetrix/cicd-pki
sudo install -d -o vetrix-agent -g vetrix-agent -m 0700 /etc/vetrix/cicd-pki/hosts
sudo install -d -o vetrix-agent -g vetrix-agent -m 0750 /var/lib/vetrix/host-agent
sudo install -d -o vetrix-agent -g vetrix-agent -m 0750 /var/log/vetrix-host-agent

If running as a non-root user, grant the binary the network-admin capabilities the nftables and Docker control paths require:

sudo setcap 'cap_net_admin,cap_net_raw,cap_sys_admin+eip' /usr/local/bin/host-agent

1.2.3 Mint the mTLS leaf certificate

The host-agent authenticates to the controller's /internal/ listener with an mTLS client certificate. The certificate is issued from the CICDv2 CA on the controller side using vetrix-cli:

# Run this on the CONTROLLER host (or wherever /etc/vetrix/cicd-pki
# lives), not on the new agent.
sudo vetrix-cli cicd-pki issue-host agent01.example.com \
    --dir /etc/vetrix/cicd-pki

Copy the resulting files to the new agent host at 0o600 ownership vetrix-agent:vetrix-agent:

Local source	Remote destination
`/etc/vetrix/cicd-pki/ca.crt`	`/etc/vetrix/cicd-pki/ca.crt`
`/etc/vetrix/cicd-pki/hosts/agent01.example.com.crt`	`/etc/vetrix/cicd-pki/hosts/agent01.example.com.crt`
`/etc/vetrix/cicd-pki/hosts/agent01.example.com.key`	`/etc/vetrix/cicd-pki/hosts/agent01.example.com.key`

Routine cert rotation (every 30 days or on suspected compromise) is covered by the dedicated cicd-cert-rotation.md runbook. This runbook only covers initial issuance.

1.2.4 Write the host-agent config

/etc/vetrix/host-agent.toml is the documented default location; override with --config. The schema is defined in cmd/host-agent/config.go. All four sub-tables ([host], [controller], [mtls], [heartbeat]) are required; unknown keys fail loudly.

# /etc/vetrix/host-agent.toml

[host]
# Must match the CN on the mTLS cert above. Mismatch fails heartbeat
# with 403.
fqdn = "agent01.example.com"

# Free-form label the scheduler uses for coarse routing. Today only
# linux-amd64 ships; arm64 is roadmap.
host_class = "linux-amd64"

# Where host-agent writes the controller-issued host_id after register.
# Must be writable by the service user. Mode 0o600.
host_id_path = "/var/lib/vetrix/host-agent/host_id"

# Scheduler-side capacity accounting. The scheduler subtracts in-use
# CPU / memory from these when matching tag-targeted jobs.
capacity_cpu = 8
capacity_mem_bytes = 34359738368   # 32 GiB

# Free-form labels for scheduler matching. Empty array == shared pool.
tags = ["linux", "amd64", "docker"]

[controller]
# Controller's mTLS-terminated /internal/ base URL. Trailing slash
# optional. Set this to the production controller; for local dry-runs
# point at api.gitvetrix.test.
url = "https://api.gitvetrix.com"

# Single-use plaintext token an operator minted via §1.3 below.
# Required for `register`; can be left empty after the first successful
# register (subsequent `run` invocations skip the redeem step when
# host_id_path is populated).
attach_token = ""

[mtls]
ca_path           = "/etc/vetrix/cicd-pki/ca.crt"
client_cert_path  = "/etc/vetrix/cicd-pki/hosts/agent01.example.com.crt"
client_key_path   = "/etc/vetrix/cicd-pki/hosts/agent01.example.com.key"

[heartbeat]
# Cadence of POST /internal/host-pool/heartbeat. Default 30s matches
# the controller's liveness scan window.
interval = "30s"

# Per-request HTTP timeout. MUST be < interval (validation enforces).
timeout  = "10s"

Set tight permissions:

sudo chown vetrix-agent:vetrix-agent /etc/vetrix/host-agent.toml
sudo chmod 0640 /etc/vetrix/host-agent.toml

Validate the config with the dedicated dry-run subcommand. This exercises strict TOML decode + defaults + cross-field validation, makes no controller traffic, and needs no controller.attach_token, so it is safe to run on a freshly-provisioned node before register:

sudo -u vetrix-agent /usr/local/bin/host-agent config validate --config /etc/vetrix/host-agent.toml

On a valid file it prints config OK: /etc/vetrix/host-agent.toml and exits 0. If the config is malformed (or fails a cross-field invariant, e.g. heartbeat.timeout not < heartbeat.interval), host-agent exits non-zero with a host-agent: config validate "/etc/vetrix/host-agent.toml": … line on stderr.

Do not use host-agent run … --help to validate config: cobra resolves --help and prints the usage banner before any config is read, then exits 0. A broken toml would slip through unnoticed. Use config validate (above) — it is the only invocation that actually reads and checks the file without contacting the controller.

1.2.5 Install the systemd unit

Write /etc/systemd/system/vetrix-host-agent.service:

[Unit]
Description=Vetrix CICDv2 host-agent
After=network-online.target docker.service
Wants=network-online.target
Requires=docker.service

[Service]
Type=simple
User=vetrix-agent
Group=vetrix-agent
ExecStart=/usr/local/bin/host-agent run --config /etc/vetrix/host-agent.toml
Restart=on-failure
RestartSec=5s

# Re-open log files on SIGHUP for logrotate `create` mode.
KillSignal=SIGTERM
ExecReload=/bin/kill -HUP $MAINPID

# Capabilities required by nftables + Docker control paths.
AmbientCapabilities=CAP_NET_ADMIN CAP_NET_RAW CAP_SYS_ADMIN
CapabilityBoundingSet=CAP_NET_ADMIN CAP_NET_RAW CAP_SYS_ADMIN

# Hardening — relaxed where the runtime needs it (DinD requires the
# Docker socket; nftables apply requires NET_ADMIN; cgroup writes
# require the unified hierarchy).
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
ReadWritePaths=/var/lib/vetrix /var/log/vetrix-host-agent /var/run/docker.sock

[Install]
WantedBy=multi-user.target

Lint the unit before enabling (catches missing dependencies and typos without starting the service):

sudo systemd-analyze verify /etc/systemd/system/vetrix-host-agent.service

Reload systemd and enable the unit (do NOT start it yet — register must run first):

sudo systemctl daemon-reload
sudo systemctl enable vetrix-host-agent.service

1.3 Mint the host-attach token

The host-attach token is a 256-bit single-use credential the operator mints against the controller; the host-agent redeems it exactly once during register to receive its long-lived host_id. The token is returned in plaintext exactly once — only its SHA-256 hash is stored server-side. Default lifetime is 24 hours.

The endpoint is mounted at three owner-scopes; pick the one matching the tenant the new host should serve:

Scope	Path	Authz
Org	`POST /api/v1/orgs/{owner}/hosts:attach`	org owner OR `acl.AdminSystem` instance admin
User	`POST /api/v1/users/{user}/hosts:attach`	the named user OR `acl.AdminSystem`
Repo	`POST /api/v1/repos/{owner}/{repo}/hosts:attach`	repo owner OR `repo_admin` collaborator OR `acl.AdminRepoRoles` / `acl.AdminSystem`

The request body is optional — {} or empty is accepted. A non-empty body with unknown keys fails 400 so future TTL-override extensions surface loudly rather than silently dropping.

1.3.1 Example mint call (org scope, production)

curl -fsSL -X POST https://api.gitvetrix.com/api/v1/orgs/acme/hosts:attach \
    -H "Authorization: Bearer $VETRIX_OPERATOR_TOKEN" \
    -H "Content-Type: application/json" \
    -d '{}'

For a dry-run against the local dev stack, swap the host: https://api.gitvetrix.test/api/v1/orgs/{owner}/hosts:attach. Authentication still uses Authorization: Bearer <token>; never use localhost in operator-facing examples — the production deployment's ExternalURL is the source of truth and a baked-in localhost would break the install snippet.

1.3.2 Response shape

The handler returns 201 Created with this JSON envelope:

{
  "token":           "<43-char base64url plaintext, single use>",
  "expires_at":      "2026-05-18T10:42:00Z",
  "install_command": "curl -fsSL https://api.gitvetrix.com/install-host-agent.sh | sudo VETRIX_HOST_TOKEN=<token> bash",
  "host_attach_url": "https://api.gitvetrix.com/internal/host-pool/register",
  "scope_kind":      "org",
  "scope_id":        "00000000-0000-0000-0000-000000000000"
}

Field reference:

token — the plaintext bearer the host-agent presents to redeem. Returned exactly once; record it now. No server-side recovery is possible if it is lost (the database stores only the SHA-256 hash).
expires_at — RFC3339, UTC. 24 h from mint by default.
install_command — copy-paste line for the operator to run on the new host. Its base URL is resolved server-side via URLBuilder.HTTPBase() → Config.Server.ExternalURL → request fall-back (in that order; never a compile-time literal).
host_attach_url — the controller endpoint the host-agent will POST to with the redeemed token. Useful for operator-side firewall documentation. Note that this path is only reachable via the mTLS- terminated /internal/ listener — direct curl-style probing without a client cert fails the handshake.
scope_kind — one of "org" | "user" | "repo". Recorded on the host attach token row and on the cicd.hosts.attach_token.mint audit log entry.
scope_id — the owner UUID (org/user) or repo UUID (repo).

1.3.3 What gets written server-side

Every successful mint produces:

One row in host_attach_tokens with (token_hash, scope_kind, scope_id, expires_at, created_by). The plaintext is never persisted.
One audit_log row with action cicd.hosts.attach_token.mint and details {actor_user_id, scope_kind, scope_id, expires_at, owner, repo?}. Audit emission is best-effort — a failure does not block the response (the operator is already looking at the plaintext on screen; refusing to return would strand it).

1.4 Attach-token flow end-to-end

1.4.1 Drop the token onto the new host

Copy the plaintext token from the previous step into the host's config or pass it on the command line. The recommended path is to edit /etc/vetrix/host-agent.toml and set:

[controller]
attach_token = "<plaintext from §1.3.2>"

…then run register once:

sudo -u vetrix-agent /usr/local/bin/host-agent register \
    --config /etc/vetrix/host-agent.toml

Expected output on success:

registered host_id=11111111-2222-3333-4444-555555555555 state=registering

State is registering here, not online. register prints the controller's RegisterResult.state, which is stamped at register time before the host has ever heartbeated. The host is promoted registering → online on its first successful heartbeat (see §1.4.3 / §1.5.2); only then does the liveness predicate (state = 'online' AND last_heartbeat_at IS NOT NULL) mark it healthy in the §1.5.2 fleet view. That predicate governs the admin fleet view's health flag only — no dispatcher currently routes pipeline jobs onto host-pool hosts at all (see "What actually dispatches jobs" in dispatchers.md), so online here means the host's register/heartbeat lifecycle is healthy, not that it is receiving work. A host that is stuck at registering in the §1.5.2 fleet view after the agent has been run-ning for >1 heartbeat interval indicates the first heartbeat has not landed — confirm last_heartbeat_at is advancing.

The command:

Loads the config (fails fast on malformed TOML).
Reads the mTLS cert material lazily so a missing file gives a path-specific error.
Refuses to run if host_id_path already exists (remove the file to deliberately re-register).
POSTs RegisterRequest{AttachTokenPlaintext, FQDN, HostClass, CapacityCPU, CapacityMemBytes, Tags, Version} to controller.url + /internal/host-pool/register over mTLS.
Persists the returned host_id (UUID) to host_id_path at 0o600 atomically (write to .tmp, then rename).
Exits 0.

Once register succeeds the operator should remove the plaintext token from host-agent.toml — run does not require it once host_id_path is populated:

sudo sed -i 's/^attach_token = .*/attach_token = ""/' /etc/vetrix/host-agent.toml

1.4.2 (Alternative) install-snippet flow — PLANNED

Planned — not shipped on this rc. The mint response's install_command field already serialises a curl -fsSL https://api.gitvetrix.com/install-host-agent.sh | sudo VETRIX_HOST_TOKEN=<token> bash one-liner, but the controller does not serve /install-host-agent.sh on this release. Running the command above will return a 404. Follow §1.4.1 (hand-edit host-agent.toml, then host-agent register) until the install script ships.

When the script does ship it will be a thin convenience wrapper around the steps in §1.2 and §1.4.1, and operators who want to audit every byte running on their host should continue to prefer the explicit flow regardless.

1.4.3 Start the long-running agent

After register exits 0, start the systemd unit so the agent heartbeats every 30 s and shows healthy in the admin fleet view:

sudo systemctl start vetrix-host-agent.service
sudo systemctl status vetrix-host-agent.service

The unit's run subcommand:

Re-reads the persisted host_id from host_id_path and skips registration.
POSTs /internal/host-pool/heartbeat every heartbeat.interval.
Does not claim or execute pipeline jobs today. The register + heartbeat loop above is the entirety of what run does on this release — the DinD execution machinery (internal/runnerctl/executor/) exists in the backend repo, but nothing currently drives it, because no dispatcher feeds jobs to the host pool. See "What actually dispatches jobs" in dispatchers.md for the v1 queue-only path that does run pipeline jobs today (against ci_runners, not this host pool).

1.5 Verify

Run these — they exercise the two verification layers a host pool actually has today (token storage, runtime heartbeat). §1.5.4 below explains why end-to-end job dispatch is not a third, performable layer on this release:

1.5.1 Audit log entry

The mint emits a cicd.hosts.attach_token.mint audit row. An operator with acl.AdminAuditRead can confirm via the audit-log admin API:

curl -fsSL "https://api.gitvetrix.com/api/v1/admin/audit:cicd?action=cicd.hosts.attach_token.mint&per_page=5" \
    -H "Authorization: Bearer $VETRIX_OPERATOR_TOKEN"

Expect the most recent row's details to carry the scope you minted under.

1.5.2 Host appears in the fleet inventory

The instance-admin (acl.AdminRunners) fleet view shows every host_pool row with its capacity, state, and last heartbeat:

curl -fsSL "https://api.gitvetrix.com/api/v1/admin/hosts?per_page=20" \
    -H "Authorization: Bearer $VETRIX_OPERATOR_TOKEN"

The new host should appear with:

state: "online" (or "draining" if a detach was already issued)
last_heartbeat_at updating every heartbeat.interval
capacity_cpu / capacity_mem_bytes matching the toml values
tenant_pin_id equal to the scope_id returned in §1.3.2

A tenant-scoped GET endpoint (GET /api/v1/{owner-scope}/hosts) is not yet shipped on this rc. Tenants today verify attach via §1.5.1 (audit row) plus the presence of the host in the admin fleet view (with operator assistance) until the tenant-scope read lands.

1.5.3 Agent journal log lines

Tail the agent's journal — the registration and ready lines land immediately on run startup; heartbeat traffic is silent on the success path (see operational note below):

sudo journalctl -u vetrix-host-agent.service --since=-2m

Expected log lines (slog text handler to stderr; see cmd/host-agent/cmd_run.go):

time=… level=INFO msg="host-agent: registered" host_id=… state=registering
time=… level=INFO msg="host-agent: ready" host_id=… fqdn=agent01.example.com

state=registering on this line is expected. The registered log echoes the controller's register-time RegisterResult.state; promotion to online happens on the first heartbeat and is observable via the §1.5.2 fleet view, not in this journal line.

Operational note — no success-path heartbeat log. The agent emits log lines only on the heartbeat failure path (msg="host-agent: heartbeat failed" at WARN, and a one-off msg="host-agent: initial heartbeat failed" WARN on first-beat failure). A healthy agent therefore prints host-agent: registered

host-agent: ready on startup and then stays silent — absence of heartbeat failed warnings is the success signal. Verify liveness via §1.5.2 (last_heartbeat_at advancing in the admin fleet view), not by tailing the agent journal for a per-beat line.

1.5.4 End-to-end dispatch verification — not currently available

There is no way to verify a newly attached host by dispatching a pipeline job onto it, and this is not a gap in this runbook — it reflects the current system: no dispatcher feeds the host pool at all. Every pipeline job in production runs through the v1 queue-only path, which claims jobs out of the ci_runners table (the runner-agent registration table) and never reads runner_hosts (the table this host pool lives in). Triggering a pipeline against this host's tags will not land a job on it, however the tags, capacity, or heartbeat health are configured. See "What actually dispatches jobs" in dispatchers.md for the full picture, including why an empty or dispatch-less runner_hosts pool is expected and not a fault.

§1.5.1 (audit log entry) and §1.5.2 (fleet inventory — state: "online" with last_heartbeat_at advancing) are therefore the complete verification for a newly attached host on this release. Treat those two as sufficient confirmation that registration and heartbeat are healthy; there is no dispatch-based check to add on top of them until a dispatcher exists for this host pool.

1.6 Troubleshooting

Symptom	Probable cause	Resolution
`register: attach token is invalid, expired, or already used — ask an operator for a fresh token`	Token is past `expires_at`, has already been redeemed, or never existed.	Mint a fresh token via §1.3, drop the new plaintext into `[controller].attach_token` (or remove `host_id_path` and re-run the install snippet), and retry `host-agent register`. Tokens are single-use by design.
`register: host_id_path "…" already populated; remove the file to re-register`	A previous `register` succeeded.	Confirm the host is intentionally being re-registered (this leaves the previous `runner_hosts` row orphaned — issue a tenant-scoped DELETE first; see "Host showing as 'draining' / 'decommissioned'" below). Then `sudo rm "$host_id_path"` and rerun.
Connection refused / TLS handshake failure when running `register`	Controller URL wrong, firewall blocks egress to `controller.url`, or transparent egress proxy is stripping TLS (breaks mTLS).	(1) Confirm `controller.url` in toml resolves and is reachable: `curl -k --resolve … https://<controller>/healthz`. (2) Confirm there is no TLS-terminating proxy between agent and controller — mTLS requires end-to-end TLS. (3) Inspect the controller logs for the TLS error.
`tls: bad certificate` / `remote error: tls: unknown certificate authority`	mTLS client cert / CA bundle mismatch. The agent's CA does not match the controller-side trust bundle, or the client cert was issued by a different CA than the controller currently trusts.	(1) Verify `mtls.ca_path` on the agent matches `/etc/vetrix/cicd-pki/ca.crt` on the controller. (2) Confirm the leaf cert was issued by the current CA (`openssl verify -CAfile <ca> <leaf>`). (3) If the controller is in a dual-trust window during CA rotation, ensure both CAs are concatenated into the agent's trust bundle (see `cicd-cert-rotation.md` §3).
Heartbeat returns 403 with CN mismatch	`host.fqdn` in toml ≠ CN on the mTLS leaf. The heartbeat handler does a CN cross-check (`internal/api/host_pool_heartbeat.go`).	Re-issue the leaf with `cicd-pki issue-host <correct-fqdn>` OR correct `host.fqdn` in the agent toml. CN and config FQDN MUST match.
Host showing as `draining` after a DELETE	A tenant or operator issued a graceful detach via DELETE `/api/v1/{owner-scope}/hosts/{id}`. The state flip is intentional — draining removes the host from the `state = 'online'` candidate set (see §1.5.4) but lets in-flight jobs finish.	Poll the progress URL returned in the 202 envelope (`GET /api/v1/admin/hosts/{id}/drain`). Once `drained: true`, the admin reconciler removes the row (or an operator may issue POST `/api/v1/admin/hosts/{id}/decommission`). Re-attaching a host that was drained on purpose requires re-issuing a token and starting from §1.3.
Heartbeat OK but no pipeline jobs ever land on this host	Expected on this release, not a misconfiguration — no dispatcher feeds the host pool at all (see §1.5.4). Tags, `capacity_cpu`/`capacity_mem_bytes`, and tenant pin have no bearing on pipeline job placement today; they only matter for a future dispatcher.	Not something to fix here. If pipeline jobs need to run, confirm the v1 worker is polling and the runner is registered in `ci_runners` instead — see `dispatchers.md`. This host pool has no effect on pipeline job placement today.
`unknown keys: …` from `host-agent register` or `run`	Typo in the toml file. The loader rejects undecoded keys to surface misconfiguration loudly.	Fix the named key. Reference `cmd/host-agent/config.go` for the canonical schema.
Fleet view shows implausible `in_use_cpu` (e.g. ~2.5x or ~10x the real load) on a host	The agent assumes `procClockHz = 100` (USER_HZ) when converting `/proc/stat` jiffies to seconds, but this host runs a custom kernel built with a different `CONFIG_HZ` (250 or 1000).	Confirm the kernel tick with `getconf CLK_TCK` on the affected host (see §1.7). If it is not 100, the host needs a stock-HZ kernel or a patched agent build — the scheduler back-off math assumes 100.
Audit row missing for the mint	Auditor failure is logged but not fatal. The token is still valid.	Check controller logs around the mint timestamp for an audit-emission warning. Token validity is unaffected.

1.7 /proc polling and the `procClockHz = 100` assumption

The host-agent derives the in_use_cpu figure it reports in each heartbeat from two consecutive reads of /proc/stat (cached for 5 s; see procCacheTTL in cmd/host-agent/cmd_run.go). It converts the busy-jiffies delta into an average-cores-busy estimate with:

avgCoresBusy = busy_jiffies_delta / (elapsed_seconds * procClockHz)

procClockHz is the kernel CLOCK_TICK rate (USER_HZ). The agent hard-codes procClockHz = 100 rather than calling sysconf(_SC_CLK_TCK) at runtime. Every mainstream Linux distro kernel since the 2.6 series ships with CONFIG_HZ=100, so this value is correct for the entire stock fleet, and the in_use_cpu number is a coarse load estimate the scheduler subtracts from capacity_cpu — a small miscount does not change scheduling decisions.

Custom-kernel operators: verify USER_HZ before trusting load numbers. If you run a host on a self-built or vendor kernel, the hard-coded assumption can be wrong. Confirm the host's actual tick rate:
getconf CLK_TCK
If this prints 100 the agent's reported in_use_cpu is accurate. If it prints 250 or 1000, the conversion is off by that ratio (2.5x or 10x): the host will systematically over-report busy CPU in the fleet view and the scheduler will back off from it more than it should. The agent has no override knob for this today — such a host needs either a stock-HZ kernel or a patched agent build with procClockHz set to match getconf CLK_TCK. This is the root cause of the "implausible in_use_cpu" troubleshooting row in §1.6.

2. Graceful drain + decommission

This section walks an operator (or the owner of a tenant-attached host) through the two-phase removal of a CICDv2 runner host from the scheduler fleet without losing any in-flight job's output:

§2.1 When to drain — pick the right verb for the lifecycle event in front of you.
§2.2 Pre-drain inventory — list the active jobs the host is about to keep running.
§2.3 Drain procedure — both the operator-driven admin path and the owner-initiated tenant path.
§2.4 Wait for completion — poll the progress URL until the host is quiescent.
§2.5 Decommission step — remove the runner_hosts row, tear down tenant networks. Irreversible.
§2.6 Verify — confirm the host is gone from the admin fleet inventory.
§2.7 Recovery / re-enrollment — re-attach a host that was drained-and-removed by mistake.

Source references for this section: internal/api/admin_hosts_drain.go (admin drain + progress), internal/api/admin_hosts_decommission_vbe251.go (admin decommission), internal/api/hosts_detach_vbe262.go (owner-initiated detach), internal/api/admin_runner_hosts.go (admin fleet inventory), internal/cicd/host_drain.go (state-machine + transactional FOR UPDATE), internal/cicd/hostadmin/decommission.go (state vocabulary + precondition).

2.1 When to drain

Drain is the prelude to any operation that takes a host out of the fleet without disrupting an in-flight job. Reach for it when:

Trigger	Verb	Notes
Kernel update / OS upgrade requiring a reboot	drain → reboot → re-register (NOT decommission)	The host's `runner_hosts` row stays; only the agent process is restarted. Re-running `host-agent run` after the reboot resumes heartbeats.
Hardware swap (failing disk, planned hardware refresh)	drain → decommission → re-attach the replacement	Replacement is a different machine, so it earns a new `host_id`. Re-attach uses §1.3's token flow.
Capacity reduction (shrinking the fleet)	drain → decommission	Permanent removal.
Suspected compromise or PKI rotation drift	drain → decommission → re-issue cert → re-attach	Drain first to bound the blast radius without aborting honest jobs in flight.
Operator wants to evacuate a tenant pin from a host	tenant-issued DELETE (§2.3.2) — drain only	Owner-side detach reuses the same drain state-flip. Decommission stays an operator step.

Skip drain entirely only when you are forcibly killing a host that has already lost contact (no heartbeat for longer than RunnerHostStaleAfter). In that case the row will fall out of the scheduler's loadAliveHosts candidate scan on its own — file a follow-up to decommission once you have confirmed the host is dead.

2.2 Pre-drain inventory

There is no dedicated "active jobs on host X" admin endpoint in this rc. Use the admin fleet view to read the per-host in_use_cpu / in_use_mem_bytes counters — they are the scheduler's snapshot of what is pinned to the host right now:

# Production:
curl -fsSL "https://api.gitvetrix.com/api/v1/admin/hosts?per_page=100" \
    -H "Authorization: Bearer $VETRIX_OPERATOR_TOKEN" \
    | jq '.items[] | select(.id == "<host-id>")'

# Dev:
curl -fsSL "https://api.gitvetrix.test/api/v1/admin/hosts?per_page=100" \
    -H "Authorization: Bearer $VETRIX_OPERATOR_TOKEN" \
    | jq '.items[] | select(.id == "<host-id>")'

The response row carries (per admin.RunnerHost):

state — pre-drain you expect "online" (or "draining" if a detach already landed); post-drain "draining" until the row is removed.
in_use_cpu / in_use_mem_bytes — non-zero values indicate jobs pinned to this host. Use them as the "how busy is this host" hint before pulling the trigger.
last_heartbeat_at — a stale value here means the host may already be off the air; the drain still works (state flip is operator intent, not host liveness) but you should not expect in-flight jobs to complete.
tenant_pin_id — non-null means an org/user/repo owner attached this host. The owner can detach it themselves via §2.3.2.

The authoritative remaining-in-flight signal lives on the drain progress endpoint itself (in_flight_jobs field — see §2.4); the fleet-list view is the broad-strokes pre-drain check.

If you need an exact per-job listing while the host is still serving traffic, query pipeline_jobs directly via psql or the ops shell (look for WHERE host_id = $1 AND state = 'running'). A future ticket will surface this as an admin endpoint.

2.3 Drain procedure

2.3.1 Admin path (operator-driven)

The operator endpoint is the canonical drain trigger when the fleet is being managed by an instance admin. Authz is acl.AdminRunners (same scope the existing /api/v1/admin/runners CRUD uses); 401 anonymous / 403 non-admin via the router gate chain.

# Production:
curl -fsSL -X POST "https://api.gitvetrix.com/api/v1/admin/hosts/<host-id>/drain" \
    -H "Authorization: Bearer $VETRIX_OPERATOR_TOKEN"

# Dev:
curl -fsSL -X POST "https://api.gitvetrix.test/api/v1/admin/hosts/<host-id>/drain" \
    -H "Authorization: Bearer $VETRIX_OPERATOR_TOKEN"

Expected 202 Accepted envelope (the Location header carries the same progress_url):

{
  "host_id":        "11111111-2222-3333-4444-555555555555",
  "state":          "draining",
  "in_flight_jobs": 2,
  "drained":        false,
  "progress_url":   "/api/v1/admin/hosts/11111111-2222-3333-4444-555555555555/drain"
}

Response-code matrix (from admin_hosts_drain.go):

Code	Meaning	When
`202 Accepted`	drain initiated (or no-op re-drain of an already-draining host — idempotent)	`online → draining`, `draining → draining`
`400 Bad Request`	malformed host UUID in the URL	`{id}` is not a valid UUID
`401 Unauthorized`	no bearer / invalid bearer	the requireAuth gate
`403 Forbidden`	bearer lacks `acl.AdminRunners`	the requireAdminScope gate
`404 Not Found`	no `runner_hosts` row with that id	the FOR UPDATE select returned no row
`409 Conflict`	host is already decommissioned (terminal)	the row exists but `state = 'decommissioned'`
`503 Service Unavailable`	CICD persistence not wired on this deployment	the handler degrades open so the route stays mounted

The state transition is atomic under a SELECT … FOR UPDATE lock on runner_hosts (internal/cicd/host_drain.go), so a concurrent decommission cannot race the drain. Once state = 'draining', the host no longer satisfies the loadAliveHosts candidate predicate (WHERE state = 'online') — this is the eligibility check a dispatcher for this host pool would consult if one existed; none does today (see §1.5.4), so there is no live "dispatch cycle" for the host to drop out of, and no direct call into a scheduler from this handler either way.

A best-effort audit row (admin.host_drained) is written; audit failure never fails the request.

2.3.2 Owner-initiated path (tenant detach)

The tenant-facing counterpart lets the owner who attached the host detach it without operator help. The same drain state-flip is reused under the hood, so an admin-drained host and an owner-detached host land in identical runner_hosts state — there is no per-pathway drift in the scheduler's view.

Three mount points share one handler; pick the one matching the scope the host was attached under:

Scope	Path	Authz
Org	`DELETE /api/v1/orgs/{owner}/hosts/{id}`	org owner OR `acl.AdminSystem` instance admin
User	`DELETE /api/v1/users/{user}/hosts/{id}`	the named user OR `acl.AdminSystem`
Repo	`DELETE /api/v1/repos/{owner}/{repo}/hosts/{id}`	repo `PermRepoAdmin` holder OR `acl.AdminSystem`

Example (org scope):

# Production:
curl -fsSL -X DELETE "https://api.gitvetrix.com/api/v1/orgs/acme/hosts/<host-id>" \
    -H "Authorization: Bearer $VETRIX_OWNER_TOKEN"

# Dev:
curl -fsSL -X DELETE "https://api.gitvetrix.test/api/v1/orgs/acme/hosts/<host-id>" \
    -H "Authorization: Bearer $VETRIX_OWNER_TOKEN"

Expected 202 Accepted envelope:

{
  "host_id":                 "11111111-2222-3333-4444-555555555555",
  "state":                   "draining",
  "estimated_drain_seconds": 60,
  "in_flight_jobs":          2,
  "drained":                 false,
  "progress_url":            "/api/v1/admin/hosts/11111111-2222-3333-4444-555555555555/drain"
}

The progress_url deliberately points at the admin progress endpoint (GET /api/v1/admin/hosts/{id}/drain). Tenants who do not have acl.AdminRunners cannot poll it directly today; they should co-ordinate with an operator for the wait-for-completion phase, or infer completion from the audit_log (an absence of new pipeline_jobs rows pinned to the host over a job-timeout window is the operational signal until a tenant-scoped progress endpoint lands).

Response-code matrix:

Code	Meaning
`202 Accepted`	drain initiated (or no-op re-detach — idempotent)
`400 Bad Request`	malformed host UUID, missing `owner` / `repo` path segment
`401 Unauthorized`	no bearer
`403 Forbidden`	bearer is not the owner / repo admin / system admin, or `host_scope_mismatch` (typed error_code) — the host's `tenant_pin_id` does not match the URL scope you addressed it through
`404 Not Found`	no host row, or the host's `tenant_pin_id` is NULL (shared-pool host — tenants cannot enumerate the global fleet)
`409 Conflict`	host is decommissioned
`503 Service Unavailable`	CICD persistence not wired

Cross-tenant guard: a host attached under org A cannot be detached through /api/v1/users/{user}/hosts/{id} even by the matching user. Instance admins (acl.AdminSystem or legacy IsAdmin) bypass this check.

Every successful tenant detach emits a cicd.host.detach audit row (distinct from the admin admin.host_drained event so auditors can distinguish "tenant detached their own host" from "operator drained the fleet").

2.4 Wait for completion

Drain is asynchronous. The progress_url returned in the 202 body is a read-only endpoint that reports the live drain status; poll it until drained == true.

# Same URL the 202 Location header carries — admin-scoped GET.
curl -fsSL "https://api.gitvetrix.com/api/v1/admin/hosts/<host-id>/drain" \
    -H "Authorization: Bearer $VETRIX_OPERATOR_TOKEN"

Response shape (identical to the POST body so a single client can poll it):

{
  "host_id":        "11111111-2222-3333-4444-555555555555",
  "state":          "draining",
  "in_flight_jobs": 1,
  "drained":        false,
  "progress_url":   "/api/v1/admin/hosts/11111111-2222-3333-4444-555555555555/drain"
}

What "complete" looks like:

state flips from "draining" and, once in_flight_jobs reaches zero, the drain reconciler (internal/cicd/hostadmin/ drain_reconciler.go) automatically promotes the row to "drained" on its next pass (≤30 s by default). The progress body's drained: true flag is derived from state == 'draining' AND in_flight_jobs == 0; the row's state column then independently catches up to "drained" so the §2.5 decommission precondition is met without operator action.
in_flight_jobs ratchets monotonically down to zero as the host-pinned pipeline_jobs finish. The count is read fresh on every GET; there is no caching.
drained: true is the operator's green light to proceed to §2.5. Allow up to one reconciler interval (≤30 s) after drained: true for the row's state column to flip to "drained"; decommission may briefly 409 host_not_drained in that window — retry and it will succeed once the reconciler has run.

Polling pattern (bash):

HOST_ID=<host-id>
PROGRESS="https://api.gitvetrix.com/api/v1/admin/hosts/${HOST_ID}/drain"
while true; do
    body=$(curl -fsSL "$PROGRESS" -H "Authorization: Bearer $VETRIX_OPERATOR_TOKEN")
    drained=$(printf '%s' "$body" | jq -r .drained)
    inflight=$(printf '%s' "$body" | jq -r .in_flight_jobs)
    printf 'host=%s drained=%s in_flight=%s\n' "$HOST_ID" "$drained" "$inflight"
    [ "$drained" = "true" ] && break
    sleep 15
done

Timeout guidance: a healthy fleet's per-job ceiling is the configured CICDv2 job timeout (operator-set in app.toml, typically 1 h). A single sleepy job can therefore hold a host in draining for the full job timeout. If you cannot wait, escalate to a per-job cancel through the pipeline UI before the drain rather than after — decommission refuses with 409 host_not_drained while in_flight_jobs > 0 (the row stays draining until the last job finishes, at which point the drain reconciler promotes it to drained on its next pass).

Audit-log signal that the host has quiesced: when the drain reconciler promotes the row it emits a dedicated cicd.host.drained CICDv2 audit event scoped to the runner host (scope_kind = runner_host, scope_id = <host-id>). That event is the canonical "host is now fully drained and decommissionable" signal. Per-job completion events still fire as each pinned job finishes; the progress endpoint's in_flight_jobs == 0 remains the canonical polling signal — do not pattern-match free-form audit text as a substitute for either.

2.5 Decommission step

Automatic drain → drained promotion. The decommission handler requires runner_hosts.state == "drained" (internal/cicd/hostadmin/decommission.go:69+204). The drain handler only writes "draining" (internal/cicd/host_drain.go:139); the drain reconciler (internal/cicd/hostadmin/drain_reconciler.go) closes the gap. It is a background loop wired into server startup (cmd/server/main.go) that, on each pass (≤30 s by default), scans every runner_hosts row in state "draining", counts the in-flight pipeline_jobs still pinned to that host, and — once that count reaches zero — atomically promotes the row to "drained" and emits a cicd.host.drained audit event. The transition is automatic, idempotent, and safe to re-run: no operator action and no manual SQL are required to satisfy the decommission precondition.

Timing: after §2.4 reports drained: true, allow up to one reconciler interval (≤30 s) for the row's state column to flip. If you POST decommission inside that window you may see a single transient 409 host_not_drained with observed_state: "draining" — simply retry; the next reconciler pass promotes the row and the decommission succeeds.

Decommission is a separate, irreversible step from drain. Drain takes the host out of the state = 'online' candidate set (see §1.5.4 — no dispatcher reads it today); decommission removes the runner_hosts row and tears down tenant networks. Authz is acl.AdminRunners — same scope as drain — so the same operator can run both, but the precondition is that the host be drained first (see §2.4 for what drained means in practice; the drain reconciler promotes the row to "drained" automatically once the in-flight job count hits zero, so the precondition is met without manual intervention — see the callout above for timing).

# Production:
curl -fsSL -X POST "https://api.gitvetrix.com/api/v1/admin/hosts/<host-id>/decommission" \
    -H "Authorization: Bearer $VETRIX_OPERATOR_TOKEN"

# Dev:
curl -fsSL -X POST "https://api.gitvetrix.test/api/v1/admin/hosts/<host-id>/decommission" \
    -H "Authorization: Bearer $VETRIX_OPERATOR_TOKEN"

Expected 204 No Content on success — there is no response body.

Response-code matrix (from admin_hosts_decommission_vbe251.go):

Code	Meaning
`204 No Content`	row deleted, tenant networks torn down, audit row written
`400 Bad Request`	malformed host UUID
`401 Unauthorized`	no bearer
`403 Forbidden`	bearer lacks `acl.AdminRunners`
`404 Not Found`	no row with that id
`409 Conflict` (typed `host_not_drained`)	the row exists but `state != 'drained'`. The error details carry `observed_state` and `required_state: "drained"` so operator tooling can surface "drain-first" guidance without re-reading the row. If `observed_state` is `"draining"` and §2.4 already reported `drained: true`, this is the brief ≤30 s window before the drain reconciler's next pass — retry and it will succeed (see the callout at the top of this section).
`503 Service Unavailable`	decommission service not wired

The row removal and the tenant-network cleanup run inside a single transaction with the row locked FOR UPDATE, so a concurrent drain or detach cannot interleave with the delete. If the network cleanup errors, the row delete rolls back and the host stays in the fleet (drained, retry-safe). The audit row (admin.hosts.decommission) is emitted best-effort after the commit; an audit failure never unwinds the delete.

Irreversibility note. Decommission deletes the runner_hosts row. There is no soft-delete — the row is gone and the host_id will never reappear. If the same physical machine needs to come back into the fleet, the host-agent must re-register with a fresh attach token (§2.7), which mints a new host_id.

If you want to keep a quiesced host around as an inventory artefact (e.g. for a forensic readback before final removal), stop after §2.4 and skip §2.5 — the row stays in state = 'draining' indefinitely and is excluded from the state = 'online' candidate set a dispatcher would draw from (see §1.5.4).

2.6 Verify

Confirm the host is gone from the admin fleet inventory:

curl -fsSL "https://api.gitvetrix.com/api/v1/admin/hosts?per_page=100" \
    -H "Authorization: Bearer $VETRIX_OPERATOR_TOKEN" \
    | jq '.items[] | select(.id == "<host-id>")'

Expected: no output. The decommission deleted the row, so the admin-fleet list endpoint no longer carries it.

A note on terminal-state vocabulary: the constant HostStateDecommissioned = "decommissioned" is defined in internal/cicd/host_drain.go:51 and is the terminal sentinel value in the drain state-machine — the drain handler refuses (409) when it observes a row already in that state, treating it as "already removed". In practice, however, the admin inventory will not return a decommissioned-state row on the success path because §2.5's transaction DELETEs the row outright (no soft-delete; see the irreversibility note in §2.5). "decommissioned" therefore shows up only as a refusal sentinel inside the drain handler when a race condition or out-of-band write would otherwise let a removed host re-enter the drain flow; it is not a state an operator should expect to see in the admin fleet list under normal operation.

If the row is still present:

Re-check the response code from §2.5 — a 409 host_not_drained means either drain has not completed (loop back to §2.4 until drained: true) or you are inside the ≤30 s window before the drain reconciler's next pass promotes the row to "drained". In the latter case simply retry the decommission POST; the next reconciler pass clears the precondition automatically.
Confirm last_heartbeat_at is not advancing (the host-agent should not be sending heartbeats after a successful decommission — if it is, the operator did not stop the systemd unit on the host machine; do that now).
Check the controller's error.log for the host id around the decommission timestamp; an opaque 500 from §2.5 lands an error line here.

Cross-check the audit log (operator with acl.AdminAuditRead):

curl -fsSL "https://api.gitvetrix.com/api/v1/admin/audit:cicd?action=admin.hosts.decommission&per_page=5" \
    -H "Authorization: Bearer $VETRIX_OPERATOR_TOKEN"

Expect the most recent row to carry target_host_id matching the host you removed.

2.7 Recovery / re-enrollment

Decommission is irreversible — the original host_id is gone. If an operator drained-and-removed a host by mistake, the recovery path is to re-attach the same physical machine through the standard flow:

Mint a fresh host_attach_token against the scope the host was originally attached under. The minting steps are documented in §1.3 ("Mint the host-attach token"); do not duplicate them here — that section is the source of truth for token lifetime, single-use semantics, and the cicd.hosts.attach_token.mint audit event.
On the agent host: remove the persisted host_id file so the host-agent will re-register (the file path is the host_id_path value in /etc/vetrix/host-agent.toml, typically /var/lib/vetrix/host-agent/host_id):
```
sudo rm /var/lib/vetrix/host-agent/host_id
```
Drop the new plaintext token into [controller].attach_token in the agent toml (or pass it via the install snippet's VETRIX_HOST_TOKEN env var) and run register exactly as in §1.4.1.
Start (or restart) the vetrix-host-agent.service systemd unit. The agent will heartbeat under a new host_id; the original id is permanently retired.

The previous host's tags / capacity / tenant pin do not carry over — they are re-asserted by the host-agent.toml on the new registration. Verify the post-recovery values via the admin fleet view (§2.6) match what the previous host advertised.