Vetrix Docs

CI job dispatch — the v1 queue-only dispatcher

Vetrix dispatches every pipeline job through a single dispatcher: the v1 queue-only path. A second, controller-based dispatcher ("cicd_v2") was staged behind a feature flag for a time, but that effort was sunset and its supporting code was removed from the backend repo. There is no longer a flag, a router gate, or an engine-side seam that can select a v2 dispatcher — so there is nothing to select, and no deployment can be "on v2".

This page exists because the runner_hosts table (the DinD host pool built for the v2 controller) is empty in production, and that emptiness has been flagged as a possible fault. It is not a fault. This page records why the table exists, why it is empty, and which parts of the v2 host-pool work are still live.

Audience: operators and engineers diagnosing "my pipeline job never runs" or "is runner_hosts supposed to be empty?". For the v1 worker's Docker-daemon (DinD) provisioning contract see runners.md; for host lifecycle ops (register / drain / decommission), which are still live, see the operator runbook (now maintained in runbook-docs).

What actually dispatches jobs

v1 — QueueOnlyDispatcher (queue-only, uses ci_runners)

This is the only dispatcher. cmd/server wires it explicitly:

cicdEngine := cicd.NewEngine(cicdStore, cicd.QueueOnlyDispatcher{})

(see cmd/server/main.go (vetrix backend repo) around the cicdEngine construction). The engine dispatches through that one dispatcher unconditionally — internal/cicd/engine.go (vetrix backend repo) holds a single dispatcher field and no flag-time routing.

QueueOnlyDispatcher is a deliberate no-op on both Dispatch and Cancel — see internal/cicd/dispatcher_queue.go (vetrix backend repo). The pipeline lifecycle is split across two processes:

  • cmd/server holds the *cicd.Engine. Engine.Schedule writes the Pipeline + per-stage PipelineJob rows in JobStatePending. The inserted-pending row is the dispatch signal; the engine pushes nothing at an execution backend.
  • cmd/worker runs the execution loop. It polls Store.ListPendingJobs on a fixed cadence and atomically claims jobs via Store.ClaimJob. Docker / Firecracker execution happens inside the worker.

The worker resolves which runner may claim a job against the ci_runners table (the runner-agent registration table) plus runner-group eligibility — see Store.ListRunners in internal/admin/store.go (vetrix backend repo) and EvalRunnerEligibility in internal/admin/runner_dispatch.go (vetrix backend repo).

The v1 path does NOT use runner_hosts at all. runner_hosts is the host-pool table built for the v2 controller; it has disjoint columns and a disjoint lifecycle from ci_runners (called out explicitly in the header of internal/admin/runner_hosts.go (vetrix backend repo)).

v2 controller — sunset, code removed

The v2 design would have inverted the v1 model: instead of letting workers poll, a controller would fan jobs at the DinD host pool — the runner_hosts table — scheduling ephemeral runners onto hosts in an accepting state with a recent heartbeat.

That controller never shipped a dispatch loop, and the staged migration was sunset rather than funded. The backend repo no longer contains:

  • the cicd_v2 feature-flag reader and the NotImplementedV2Dispatcher placeholder (internal/cicd/feature_flag.go — deleted, historical reference only),
  • the router-level v1/v2 gate that returned a structured 503 and stamped the X-CICD-Controller response header (internal/api/cicdv2_router_gate.go — deleted, historical reference only),
  • the per-tenant flag overlay editor (internal/api/admin_cicd_flag.go — deleted, historical reference only), which served the now-removed GET / PATCH /api/v1/orgs/{owner}/cicd routes,
  • the engine-side SetFlagReader / SetV2Dispatcher seam and the flag-time activeDispatcher routing in internal/cicd/engine.go.

The POST /api/v1/runner-controller/jobs:next deprecation stub was removed alongside them.

What survives the sunset: the host-pool data plane and its operator surfaces are untouched and still live — the runner_hosts table and its migration, the admin read/drain/decommission endpoints under /api/v1/admin/hosts, the drain state machine in internal/cicd/host_drain.go (vetrix backend repo), the internal host-pool control-plane routes, and the mTLS / PKI machinery for host-agents. Those are maintained on their own merits; they are simply not fed by any dispatcher.

Not to be confused with the v2 pipeline schema. The version: 2 pipeline configuration schema is live and supported — see authoring-vetrix-pipelines.md. The sunset above concerns the cicd_v2 dispatch controller and its feature flag, a different thing with a confusingly similar name.

How to confirm which path a job took

There is only one path, so the question reduces to "is the worker picking the job up?".

  • Behavioural tell: pipeline jobs are picked up by a polling cmd/worker process claiming rows out of ci_runners. If jobs sit in pending forever, the worker is not running, is not reachable, or no runner is eligible — not a dispatcher-selection problem.
  • No controller header. The X-CICD-Controller response header no longer exists; the middleware that stamped it was removed with the gate. Its absence is expected, not a regression.
  • No flag endpoint. GET / PATCH /api/v1/orgs/{owner}/cicd (the per-tenant flag overlay) is gone and returns 404. The similarly named /api/v1/orgs/{owner}/cicd-policy and /api/v1/admin/cicd-defaults endpoints are a different surface — CI/CD policy defaults, not dispatcher selection.
  • The cicd_v2 app-settings row is inert. The historical seed migration 000181_cicd_v2_feature_flag.up.sql is immutable and still runs, so an app_settings row with key cicd_v2 and value "false" exists on every deployment. Nothing reads it, the admin settings surface no longer registers the key, and writing it changes no behaviour. Its presence is not evidence that a v2 path exists.

Is an empty runner_hosts a fault?

No. runner_hosts is the host pool built for the v2 controller. The queue-only path never reads it; job placement runs entirely off ci_runners + runner-group eligibility. An empty runner_hosts is expected and correct, not a defect, and there is no longer any flag that could make it load-bearing for dispatch.

Registering hosts into the pool remains a supported operation — the register / heartbeat / drain / decommission lifecycle is live, and the liveness predicate (state online with a heartbeat inside RunnerHostStaleAfter = 60s) still governs what GET /api/v1/admin/hosts reports as healthy. Operators who run hosts for the mTLS control plane will see rows there. Operators who do not will see none, and nothing about pipeline dispatch depends on the difference.

If pipeline jobs are hanging, look at the v1 worker and ci_runners, not at runner_hosts.

Source cross-reference

Current sources in the vetrix backend repo:

Concern File
Queue-only dispatcher (no-op; worker polls) internal/cicd/dispatcher_queue.go
Engine, Schedule, dispatchStage internal/cicd/engine.go
Production engine wiring (QueueOnlyDispatcher) cmd/server/main.go
Runner eligibility (ci_runners) internal/admin/runner_dispatch.go
Host pool — admin read surface + liveness predicate internal/admin/runner_hosts.go
runner_hosts schema / indexes / states db/migrations/000175_cicd_v2_runner_hosts.up.sql
Host drain state machine internal/cicd/host_drain.go
Inert historical flag seed db/migrations/000181_cicd_v2_feature_flag.up.sql
Host lifecycle ops (register / drain / decommission) operator runbook (now maintained in runbook-docs)
v1 worker Docker-daemon (DinD) provisioning contract runners.md

Removed from the backend repo in the sunset — historical reference only, do not expect to find these files:

Concern File
Feature-flag reader + v2 not-implemented stub internal/cicd/feature_flag.go
Router-level v1/v2 gate + X-CICD-Controller header internal/api/cicdv2_router_gate.go
Per-tenant flag overlay editor internal/api/admin_cicd_flag.go