Vetrix Docs

CICDv2 Security manual checklist execution

Related references: findings index and the threat model (now in system-docs under architecture/security/).

Scope

The CICDv2 security posture defines six manual security checks against the inner-job DinD posture, per-tenant network isolation, dispatch authz, and job-token lifecycle. This document records how to execute each item against the CICDv2 source, with one section per checklist item.

Verification modes

In-sandbox verification cannot exercise a live host kernel or a running DinD container. Each item therefore documents one of three modes:

  • static — the code path itself is the evidence (e.g., the source contains no host bind-mount). Verifiable by reading the source.
  • unit-test — a Go unit test pins the expected behaviour; a regression flips the test red.
  • live-probe-pending — the static + unit-test evidence is sufficient to predict the live behaviour but the kernel-level probe (e.g., nsenter against a running container) cannot be issued from a sandbox and must run on a live host.

For each item: code path, verification mode, evidence, and verdict are recorded.

1. nsenter shows no host paths inside the inner container

Checklist item (verbatim): nsenter no host paths visible.

Intent. A reviewer entering the inner job container's namespace via nsenter (or docker exec) MUST see only the bind mounts the executor was told to attach — never the host docker socket, never /var/lib/docker, never / of the host.

Code path verified.

  • internal/runnerctl/executor/dind.go — buildContainerConfig() (lines 523–633). The only bind mounts attached to the inner-job ContainerConfig.Binds are:
    1. spec.CacheMount.HostPath:spec.CacheMount.ContainerPath[:ro] — caller-supplied per-job build cache (lines 550–556).
    2. <bootstrap-dir>:/vetrix/bootstrap:ro — caller-supplied per-job bootstrap script's parent directory (lines 561–565).
  • internal/runnerctl/executor/executor.go — the Spec shape (lines 24–79) exposes no other bind-mount surface. There is no field for DockerSocket, no field for HostRoot, no field for arbitrary Binds[].
  • internal/runnerctl/executor/doc.go (line 34): "DinDExecutor … talks to that outer DinD's docker.sock to spawn the 'inner' job container — never to the host's docker socket."

Evidence.

  • Static grep across internal/runnerctl/executor/ shows zero literals for /var/run/docker.sock, /var/lib/docker, /var/lib/kubelet, or /:/host:

    $ grep -rn '/var/run\|docker.sock\|/var/lib/docker' internal/runnerctl/executor/
internal/runnerctl/executor/doc.go:34:// talks to that outer DinD's docker.sock to spawn the "inner" job
internal/runnerctl/executor/doc.go:35:// container — never to the host's docker socket. The DockerClient

    Both hits are comments stating the contract; no code path attaches such a mount.

  • Unit test pin: internal/runnerctl/executor/dind_security_test.go::TestDinDExecutor_NoHostPathsInBinds enumerates the ContainerConfig.Binds after Start() and asserts none of the host-escape sentinels appear. Future code that adds a host-socket bind regresses this test.

Verification mode. static + unit-test (live-probe-pending for the actual nsenter probe against a running DinD container).

Verdict. PASS (static + unit-test). Live-probe pending — see finding F2 in the findings index, which tracks the live-host nsenter probe.

2. nft list ruleset shows tenant rules

Checklist item (verbatim): nft list ruleset shows tenant rules.

Intent. Each tenant whose network exists on the host MUST have a corresponding chain in the inet vetrix_cicd table, programmed with the tenant's egress_allowlist and dispatched-to via an iifname match on the tenant bridge.

Code path verified.

  • internal/runnerctl/executor/nftables/nft.go — GenerateMultiTenantRuleset() (lines 199–266). Emits one chain tenant_<hash> per supplied TenantSpec; chain name is derived from ChainName(tenantID) (lines 164–166), a deterministic 8-hex short hash of the tenant UUID.
  • internal/runnerctl/executor/nftables/apply.go — Applier.Refresh() (lines 225–234) regenerates the entire ruleset and applies it atomically via nft -f - (lines 193–205). The ApplyRuleset path rejects an empty ruleset (line 197) so a generator bug that emitted nothing surfaces as an error instead of silently dropping tenant rules.

Evidence.

  • Unit test pins: internal/runnerctl/executor/nftables/apply_test.go::TestRefresh_CrossTenantBlock (lines 257–314) Refreshes with two tenants and parses the generated ruleset to assert (a) each per-tenant chain exists, (b) each chain's iifname dispatch line is present, and (c) the base chain still carries policy drop.
  • Golden-file fixtures internal/runnerctl/executor/nftables/testdata/0[1234]_*.nft.golden byte-pin the generator output for single/multi-tenant cases.

Verification mode. static + unit-test (live-probe-pending for nft list ruleset against the host kernel after a real refresh — that requires root and a loaded nftables module).

Verdict. PASS (static + unit-test). Live-probe pending — see finding F2 in the findings index, which covers the live nft list ruleset probe.

3. Cross-tenant call denied

Checklist item (verbatim): cross-tenant call denied.

Intent. Two surfaces:

  • Network (dataplane): A packet from tenant A's bridge destined for tenant B's CIDR MUST be dropped by the kernel ruleset.
  • API (controlplane): A request from tenant A's authenticated user/admin scope MUST NOT be able to read tenant B's CI/CD usage rows.

Code path verified.

Network half:

  • internal/runnerctl/executor/nftables/nft.go — base chain (lines 247–254) dispatches to per-tenant chains via iifname only; the per-tenant chain contains only the established/related accept and that tenant's egress_allowlist. A packet entering on tenant A's bridge is jumped to A's chain, where no rule matches B's CIDR, so the packet falls through to the base chain's policy drop.
  • Existing test: TestRefresh_CrossTenantBlock (apply_test.go:257) parses the rendered ruleset and asserts neither tenant chain contains the other tenant's CIDR — a regression to oifname (which would let cross-tenant traffic be evaluated under the destination tenant's chain) flips this test red.

API half:

  • internal/api/usage_cicd_vbe260.go::UsageForOrg (lines 327–336) and UsageForUser (lines 344–353) — gate is requireOwnerOrAdminForUsage (lines 300–317): the caller must be the addressed owner OR claims.IsAdmin OR hold acl.AdminSystem. Mismatch → 403.
  • UsageForRepo (lines 364–384) — gate is canPerform(..., acl.PermRepoAdmin). Cross-tenant caller → 403.
  • The dispatch path proper (internal/cicd/runner.go + dispatch RPC) consumes the per-job VETRIX_JOB_TOKEN which is RepoID-scoped (see item 4). A token issued for tenant A cannot be replayed against tenant B because the registry-write authz binds the token's repo_id to the target repository (internal/registry/authz.go).

Evidence.

  • Unit-test pins for the API half: internal/api/usage_cicd_vbe260_router_test.go::TestUsageCICD_OrgOwnerGate_403WithMismatchedClaim (lines 157–170). Asserts that a non-owner, non-admin claim is rejected with 403.
  • Unit-test pin for the network half: TestRefresh_CrossTenantBlock (cited above).

Verification mode. static + unit-test (live-probe-pending for the dataplane half — requires two real tenant bridges + a probe packet).

Verdict. PASS (static + unit-test). Live-probe pending — see finding F2 in the findings index, which tracks the live dataplane probe.

4. Expired token rejected

Checklist item (verbatim): expired token rejected.

Intent. A VETRIX_JOB_TOKEN whose exp claim is in the past MUST be rejected by VerifyJobToken; the registry / artifact endpoint MUST refuse the push.

Code path verified.

  • internal/auth/job_token.go::IssueJobTokenForJob (lines 105–112) stamps ExpiresAt = now + jobTimeout + JobTokenTTLGrace where JobTokenTTLGrace = 60s (line 49).
  • internal/auth/job_token.go::VerifyJobToken (lines 137–152) parses via jwt.ParseWithClaims, which surfaces the jwt/v5 library's standard exp validation. An expired token returns a wrapped ErrInvalidToken.

Evidence.

  • Unit test pin already present: internal/auth/job_token_test.go::TestIssueJobToken_ExpiredToken (line 66 onward) — mints a token with a negative TTL and asserts VerifyJobToken returns an error.
  • Unit test pin already present: internal/auth/job_token_test.go::TestIssueJobTokenForJob_ExpiredToken (line 249 onward) — same shape for the per-job (pipeline_id + job_id) variant.

Verification mode. static + unit-test (full coverage — the JWT library's exp enforcement is the load-bearing piece and is already pinned).

Verdict. PASS (unit-test). No live probe needed; the JWT library is widely used and the test pins both code paths.

5. Allowlist reject works

Checklist item (verbatim): allowlist reject works.

Intent. A dispatched job whose OuterImage is not in the operator-managed allowlist MUST be rejected before any docker pull or docker run — i.e., the docker daemon never sees the disallowed reference, and an audit row is emitted to the controller's host-pool ingest.

Code path verified.

  • internal/runnerctl/allowlist/allowlist.go::Check (line 196 onward) returns ErrEmptyImageRef for empty input or ErrImageNotAllowed for any reference not matching an allowlist entry.
  • internal/runnerctl/allowlist/audit.go::GuardWithAudit (lines 258–283) wraps a DockerRunFunc so the Check call runs before run(imageRef) is invoked; on rejection it emits a RejectionRecord to the configured Sink and returns the typed sentinel without ever invoking the underlying docker call.

Evidence.

  • Unit test pins (existing): internal/runnerctl/allowlist/audit_test.go — covers the reject path (sentinel return + audit record), allowed path (no record), and the nil sink path.
  • Unit test pins (existing): internal/runnerctl/allowlist/allowlist_test.go — covers exact-match, glob-match, and reload semantics.
  • Additional pin: internal/runnerctl/allowlist/audit_security_test.go::TestGuard_RejectsBeforeRun — asserts the wrapped DockerRunFunc does not invoke the underlying run callback when Check rejects. Regression to a "check after run" ordering flips the test red.

Verification mode. static + unit-test.

Verdict. PASS (unit-test). No live probe required for the in-sandbox tier; an end-to-end live probe would dispatch a job with a known-bad outer image and confirm the controller's audit timeline shows an outer_image_rejected row — see finding F2 in the findings index.

6. seccomp blocks /proc-mount from inside DinD

Checklist item (verbatim): seccomp blocks /proc-mount from inside DinD.

Intent. A process inside the inner DinD container that attempts mount("proc", "/some/path", "proc", ...) MUST be blocked. A defence-in-depth structure splits responsibility across two layers (seccomp + AppArmor).

Code path verified.

  • internal/runnerctl/executor/seccomp/dind.json (the bundled profile loaded by seccomp.Load("")):
    • defaultAction = SCMP_ACT_ALLOW.
    • mount, umount, umount2, pivot_root are listed in the explicit-deny rule with action: SCMP_ACT_ERRNO and errnoRet: 1 (EPERM). The retained DinD-needed allow list contains only clone, setns, unshare — required for nested build tooling but not implicated in the /proc-mount primitive.
    • Pinning tests: internal/runnerctl/executor/seccomp/profile_test.go::TestLoad_DefaultDeniesMountSyscalls asserts each of mount, umount, umount2, pivot_root appears under SCMP_ACT_ERRNO and does NOT appear under any SCMP_ACT_ALLOW rule. TestLoad_DefaultAllowsDinDSyscalls was narrowed to the retained allow set (clone, setns, unshare).
  • internal/runnerctl/executor/apparmor/profile.txt — the bundled AppArmor profile (defence-in-depth layer 2):
    • Line 35: deny mount,
    • Line 36: deny umount,
    • Line 37: deny pivot_root,
    • Lines 42–49: explicit denies on @{PROC}/sys/kernel/**, /proc/sysrq-trigger, /proc/kcore, /proc/kallsyms, /proc/mem, /proc/kmem (write/lock/exec).

Resolution. The bundled DinD seccomp profile denies mount/umount/umount2/pivot_root at the kernel-syscall layer. The /proc-mount block does not depend on AppArmor being loaded — defence-in-depth runs as two independent layers (seccomp at the syscall layer + AppArmor at the LSM layer when available). On hosts where apparmor.IsAvailable() returns false, the seccomp layer alone still blocks mount(2), so the inner job container cannot mount /proc regardless of host kernel AppArmor support. The retained clone/setns/unshare allow set is still required for nested build tooling (fork emulation, namespace operations, user-namespace sandboxing); nested container runtimes inside the inner container talk to the outer DinD daemon via docker.sock and the outer daemon (in its own privileged outer container with its own profile) performs all real mount operations.

Code path also verified.

  • internal/runnerctl/executor/dind.go::buildContainerConfig (lines 589–596) — when Security.AppArmorProfile == "", the executor consults apparmorAvailable(); only when it returns true is the bundled profile name appended to SecurityOpts. When false, no AppArmor opt is emitted — but the seccomp opt is always emitted independently (line 589–591), so the seccomp deny rule applies regardless of host AppArmor support.

Evidence.

  • Static reads of dind.json and profile.txt (cited above).
  • Pin: internal/runnerctl/executor/seccomp/profile_test.go::TestLoad_DefaultDeniesMountSyscalls — parses the bundled JSON and asserts each of mount/umount/umount2/pivot_root is denied under SCMP_ACT_ERRNO and not allowed under any SCMP_ACT_ALLOW rule.
  • Pin: internal/runnerctl/executor/apparmor/profile_security_test.go::TestProfile_DeniesMountAndProcWrites — scans the bundled profile.txt and asserts each of the load-bearing deny mount / deny umount / deny pivot_root / deny @{PROC}/sys/kernel/** lines is present.

Verification mode. static + unit-test for both seccomp and AppArmor layers.

Verdict. PASS — both layers enforce the /proc-mount block independently. Run the live-host mount("proc", …) probe (tracked as finding F2 in the findings index) to confirm kernel-level behaviour matches the static evidence.

Summary of verdicts

# Item Verdict Mode
1 nsenter no host paths visible PASS static + unit-test (live-probe-pending: finding F2)
2 nft list ruleset shows tenant rules PASS static + unit-test (live-probe-pending: finding F2)
3 cross-tenant call denied PASS static + unit-test (live-probe-pending: finding F2)
4 expired token rejected PASS unit-test
5 allowlist reject works PASS unit-test
6 seccomp blocks /proc-mount from inside DinD PASS static + unit-test (seccomp denies mount/umount/umount2/pivot_root; AppArmor also enforces)

Two findings in the findings index cover this checklist:

Finding Severity Title Summary
F2 Medium CICDv2 manual checklist — live-host probes (nsenter, nft, dataplane cross-tenant, mount-from-DinD) Static + unit-test verification is complete; this finding tracks the kernel-level probes that can only run on a real host (nsenter, nft list ruleset, cross-tenant packet, mount("proc", …) from DinD inner). Severity is Medium pending live verification — not a known active gap, but unverified at the kernel tier.
F1 High (fixed) Seccomp profile permits mount(2); /proc-mount block depends entirely on AppArmor Fixed. The bundled DinD seccomp profile (internal/runnerctl/executor/seccomp/dind.json) denies mount/umount/umount2/pivot_root with SCMP_ACT_ERRNO (EPERM). The AppArmor profile (internal/runnerctl/executor/apparmor/profile.txt) retains its independent deny of the same operations as defence-in-depth. On hosts without AppArmor loaded (apparmor.IsAvailable() == false), the seccomp layer alone still blocks the syscalls — /proc-mount from inside the inner container is blocked at the executor layer regardless of host AppArmor support.

Severity rationale (F1 — High, fixed). The bundled DinD posture is documented as "seccomp + AppArmor" defence-in-depth on the /proc-mount primitive. Before the fix, the seccomp layer did not deny mount(2) and no AppArmor opt was emitted by dind.go:589–596 on AppArmor-absent hosts, collapsing two-layers-of-defence to zero executor-layer enforcement on those hosts. The fix restores the two-layer posture by tightening the bundled seccomp profile to deny mount/umount/umount2/pivot_root independently of AppArmor availability; on AppArmor-absent hosts the seccomp layer alone now suffices, and on AppArmor-present hosts both layers enforce. The historical severity rating is High.

Severity rationale (F2 — Medium). No known active gap; the unit-test + static evidence predicts the live behaviour. The Medium rating reflects "pending live verification on a kernel host" rather than an observed failure — the kernel-tier probes (nsenter, nft list ruleset, cross-tenant packet, mount("proc", …) from DinD inner) cannot run from a sandbox and must be exercised on a live host.

Pinning tests

  • internal/runnerctl/executor/dind_security_test.go — TestDinDExecutor_NoHostPathsInBinds pins item 1.
  • internal/runnerctl/allowlist/audit_security_test.go — TestGuard_RejectsBeforeRun pins item 5 ordering.
  • internal/runnerctl/executor/apparmor/profile_security_test.go — TestProfile_DeniesMountAndProcWrites pins item 6 AppArmor half.
  • internal/runnerctl/executor/seccomp/profile_test.go — TestLoad_DefaultDeniesMountSyscalls pins item 6 seccomp half: each of mount/umount/umount2/pivot_root is denied under SCMP_ACT_ERRNO and not allowed under any SCMP_ACT_ALLOW rule. TestLoad_DefaultAllowsDinDSyscalls covers the retained allow set (clone, setns, unshare).

Other tests cited above (TestRefresh_CrossTenantBlock, TestIssueJobToken_ExpiredToken, TestIssueJobTokenForJob_ExpiredToken, TestUsageCICD_OrgOwnerGate_403WithMismatchedClaim, TestLoad_DefaultAllowsDinDSyscalls) pin items 2, 3, 4, and (the load-side of) 6.

Live-host verification

To complete verification on a live host:

  1. Run the live probes for finding F2 against a staging host (or document the deferral with the operator owner).
  2. Update the §"Summary of verdicts" table if any verdict flips after live probing.