CICDv2 mTLS certificate rotation

Operator runbook for rotating the certificates that protect the CICDv2 control-plane. The PKI primitives are vetrix-cli cicd-pki init and issue-host; this runbook covers the rotation flow (vetrix-cli cicd-pki rotate-host).

The CICDv2 controller and host-agents trust a single private CA stored under --dir (default /etc/vetrix/cicd-pki/). The CA is intended to outlive the cluster — operators rotate the leaf certs every 30 days or after a credential-exposure incident. CA replacement is rare and gated behind the dual-trust flow in §3.

When to rotate

Trigger	Section	Verb
Scheduled 30-day leaf rotation	§1	`cicd-pki rotate-host`
Suspected leaf-key compromise on a single host	§1 + §2	`cicd-pki rotate-host --force` + revocation steps
CA key compromise	§3	full CA replacement
CA approaching `NotAfter` (≤ 90 days)	§3	CA replacement with dual-trust window
Host added to fleet	Bootstrap (`issue-host`)	`cicd-pki issue-host`

This document covers §1–§3. This runbook assumes the CA and at least one host cert already exist under --dir; initial bootstrap (cicd-pki init / issue-host) is covered separately.

1. Routine leaf rotation

Routine rotation re-issues hosts/<fqdn>.crt and hosts/<fqdn>.key under --dir, signed by the existing CA. The CA bundle is not touched. The controller and host-agent read the leaf on SIGHUP so rotation is zero-downtime.

1.1 Per-host rotation

sudo vetrix-cli cicd-pki rotate-host agent01.example.com \
    --dir /etc/vetrix/cicd-pki \
    --min-remaining 168h

What the command does:

Loads ca.crt + ca.key from --dir.
Loads the previous hosts/agent01.example.com.crt if it exists.
- If the previous cert was signed by a different CA, rotation aborts with ErrCAMismatch. This means the operator ran cicd-pki init --force since the last issuance — fall through to §3.
- If the previous cert still has more remaining validity than --min-remaining, rotation aborts with ErrPrematureRotation unless --force is passed. The 168h (7-day) floor is the recommended automation default; it stops cron from thrashing the certs.
Mints a fresh ed25519 leaf with the same SANs (DNS + IP) as the previous cert by default. Pass --no-carry-sans to drop them or --san to override.
Writes the new <fqdn>.crt / <fqdn>.key over the old files at 0600 inside the 0700 hosts/ directory.
Appends a JSON-line audit record to --dir/rotations.log.

1.2 After rotation

Reload the controller and any host-agent process that holds the cert:

sudo systemctl reload vetrix-controller
sudo systemctl reload vetrix-host-agent     # on the rotated host

Both processes re-open hosts/<fqdn>.crt on SIGHUP (sent by systemctl reload). Confirm the controller picked up the new serial:

sudo journalctl -u vetrix-controller --since=-1m | grep -i 'mTLS cert reloaded'

1.3 Fleet-wide scheduled rotation

A cron-driven sweep that rotates every host whose cert is within 14 days of expiry:

for fqdn in $(ls /etc/vetrix/cicd-pki/hosts | sed 's/\.crt$//' | sort -u | grep -v '\.key$'); do
    sudo vetrix-cli cicd-pki rotate-host "$fqdn" \
        --dir /etc/vetrix/cicd-pki \
        --min-remaining 336h \
        --output json
done
sudo systemctl reload vetrix-controller

The 336h (14-day) floor means rotation is a no-op for any host that has > 14 days of validity remaining, so this script is safe to run nightly. Hosts that are still > 14 days out exit non-zero with ErrPrematureRotation — pipe through || true if your scheduler treats that as an alert. Hosts on the rotation-due path produce one JSON object per success that your log shipper can index.

1.4 JSON output schema

--output json emits one object per invocation with this shape:

{
  "fqdn":            "agent01.example.com",
  "cert_path":       "/etc/vetrix/cicd-pki/hosts/agent01.example.com.crt",
  "key_path":        "/etc/vetrix/cicd-pki/hosts/agent01.example.com.key",
  "old_serial":      "1234…",
  "old_not_after":   "2026-06-15T00:00:00Z",
  "new_serial":      "5678…",
  "new_not_before":  "2026-05-16T00:00:00Z",
  "new_not_after":   "2026-06-15T00:00:00Z",
  "dns_names":       ["agent01.example.com", "agent01.internal"],
  "carried_sans":    true,
  "ca_serial":       "9999…",
  "ca_common_name":  "Vetrix CICDv2 CA",
  "rotation_log":    "/etc/vetrix/cicd-pki/rotations.log",
  "audit_recorded":  true
}

old_serial and old_not_after are empty strings when the FQDN had no prior cert (first-time issuance via the rotation verb).

2. Compromised leaf — emergency rotation

If a single host's private key is suspected compromised:

Quarantine the host — disable scheduling on it via the CICDv2 admin API, or mark the host disabled in cicd_runner_hosts.

Force-rotate the leaf:

sudo vetrix-cli cicd-pki rotate-host agent01.example.com \
    --dir /etc/vetrix/cicd-pki \
    --force

Reload the controller and host-agent as in §1.2.
Record the incident — --dir/rotations.log already has the audit entry; append a free-form note next to the matching serial in your incident-tracking system. The old_serial field in the audit entry is what you cite in the post-mortem.

There is no leaf revocation list (CRL) in the CICDv2 PKI; rotation is the revocation primitive. The controller only trusts the leaf serial it has currently loaded on disk, so as soon as SIGHUP is sent in step 3 the compromised leaf is no longer accepted by the TLS listener.

3. CA replacement — dual-trust window

CA replacement is required when:

The CA private key (ca.key) is suspected compromised.
The CA is approaching its own NotAfter (≤ 90 days). Default CA TTL is 10 years, so this is exceedingly rare.

The operation has three phases. Do not skip any phase — running cicd-pki init --force and re-issuing every leaf in one pass will lock the host-agents out of the control-plane until you re-deploy every host. The dual-trust window in phase 2 keeps the cluster reachable through the migration.

3.1 Phase 1 — mint the new CA alongside the old one

sudo vetrix-cli cicd-pki init \
    --dir /etc/vetrix/cicd-pki-next \
    --common-name "Vetrix CICDv2 CA (rotation $(date +%Y-%m))"

--dir is intentionally different from the live /etc/vetrix/cicd-pki/ so the new CA does not overwrite the old one yet.

3.2 Phase 2 — issue dual-trust leaves and deploy

For every host, issue a leaf signed by the NEW CA into a parallel directory:

sudo vetrix-cli cicd-pki issue-host agent01.example.com \
    --dir /etc/vetrix/cicd-pki-next

Concatenate the OLD and NEW CA certs to form the trust bundle used by both the controller and the host-agent (the TLS listener reads this bundle into its tls.Config.ClientCAs):

sudo bash -c 'cat /etc/vetrix/cicd-pki/ca.crt /etc/vetrix/cicd-pki-next/ca.crt > /etc/vetrix/cicd-pki/ca-bundle.crt'
sudo chmod 0600 /etc/vetrix/cicd-pki/ca-bundle.crt

Reload the controller and all host-agents so both CAs are trusted. At this point: old leaves still verify (signed by the old CA, which is in the bundle) AND new leaves verify (signed by the new CA, also in the bundle). The cluster has zero downtime.

3.3 Phase 3 — cut over and revoke the old CA

Once every host has its new-CA leaf deployed and you have verified that the controller is accepting new-CA leaves from every host:

# Promote the new CA to the canonical location.
sudo mv /etc/vetrix/cicd-pki     /etc/vetrix/cicd-pki-old
sudo mv /etc/vetrix/cicd-pki-next /etc/vetrix/cicd-pki
# Remove the dual-trust bundle so the controller stops accepting
# old-CA leaves.
sudo rm /etc/vetrix/cicd-pki/ca-bundle.crt
sudo systemctl reload vetrix-controller

Audit the cut-over by hand — rotations.log only records leaf rotations, not CA replacements. Append a note such as:

$(date -u +%FT%TZ)  CA-REPLACEMENT  old=<old-CA-serial>  new=<new-CA-serial>  operator=<you>

to your incident-tracking system or to /etc/vetrix/cicd-pki/CA-HISTORY (plain text, your call). There is no dedicated cicd-pki rotate-ca verb; CA replacement is the manual cp/mv flow above — see §4.

3.4 Rollback

If phase 3 leaves any host stranded (e.g. you missed a leaf), the old material is still at /etc/vetrix/cicd-pki-old/. Recover with:

sudo mv /etc/vetrix/cicd-pki     /etc/vetrix/cicd-pki-next
sudo mv /etc/vetrix/cicd-pki-old /etc/vetrix/cicd-pki
sudo bash -c 'cat /etc/vetrix/cicd-pki/ca.crt /etc/vetrix/cicd-pki-next/ca.crt > /etc/vetrix/cicd-pki/ca-bundle.crt'
sudo systemctl reload vetrix-controller

You are now back in phase 2's dual-trust state; resume from §3.2 after fixing whatever caused the lock-out.

4. Limitations

No cicd-pki rotate-ca verb. CA replacement (§3) is a manual cp/mv flow; there is no automated CA-rotation verb. The CA-replacement flow spans the controller's trust-bundle reload behaviour, so it stays a documented manual procedure.
CRL / OCSP. The CICDv2 PKI uses rotation-as-revocation (§2). Operators who need a true CRL must add it out-of-band.
Audit pipeline. rotations.log is JSONL on the local filesystem with a stable schema. The central audit pipeline does not tail this file; ship it out-of-band if you need it centralized.

5. Quick-reference verbs

Verb	Behaviour
`cicd-pki init`	Create a new CA at `--dir`. Refuses overwrite without `--force`.
`cicd-pki issue-host <fqdn>`	Mint a leaf signed by the CA at `--dir`. Refuses overwrite of an existing host cert without `--force`.
`cicd-pki rotate-host <fqdn>`	Re-issue the leaf for `<fqdn>` against the existing CA. Carries forward SANs by default. Refuses to rotate when previous cert has > `--min-remaining` validity unless `--force`. Appends to `rotations.log`.

6. Files this runbook touches

Path	Managed by
`/etc/vetrix/cicd-pki/ca.crt`	`cicd-pki init`
`/etc/vetrix/cicd-pki/ca.key`	`cicd-pki init`
`/etc/vetrix/cicd-pki/hosts/<fqdn>.crt`	`cicd-pki issue-host` / `rotate-host`
`/etc/vetrix/cicd-pki/hosts/<fqdn>.key`	`cicd-pki issue-host` / `rotate-host`
`/etc/vetrix/cicd-pki/rotations.log`	`cicd-pki rotate-host`
`/etc/vetrix/cicd-pki/ca-bundle.crt`	§3 only; manual