UsageAnalytics rollout runbook

Rolls out UsageAnalytics in two ordered releases plus a one-week drop-counter review cadence. Ordering matters: plumbing must bake for 48 h before the recorder is allowed to write.

Release 1 — plumbing, off by default (2026-04-22)

Ship:

All migrations 000128 — 000133 (daily partitions for analytics_events + rate_limit_violations, the three rollup MVs). Migration 000125 is the canonical source for user_group_rate_limits.beacon rows; there is no separate 000134 beacon seed.
Backend with the recorder + Partman wired but analytics.enabled=false in knownSettings defaults.
Frontend admin dashboards gated behind admin:analytics scope; user + repo surfaces behind analytics.enabled master-switch.

Operator checklist (Release 1):

Confirm the deployed binary picks up the new migration chain:
```
vetrix-server migrate status
# expected: 000133 applied
```

Confirm analytics.enabled is false post-deploy:

curl -u admin:... \
  https://api.gitvetrix.com/api/v1/admin/settings | \
  jq '.[] | select(.key=="analytics.enabled")'
# expected: { "value": "false" }

Announce Release 1 in the internal #releases channel:

UsageAnalytics plumbing shipped (Release 1). Recorder + Partman running; no events recorded yet. 48 h soak in progress; flip to analytics.enabled=true to follow.

Soak — 48 hours

Monitor /admin/analytics/health every 4 h for 48 h. Pass conditions:

Counter	Pass
`enqueued`	0 (recorder is off)
`written`	0
`dropped`	0
`queue_depth`	0
`ratelimit_violations_enqueued`	steady (RL epic baseline)
`ratelimit_redis_unavailable`	< 10 / h

Expected outcome: zero recorder activity (gate is off); Partman still runs the daily partition churn so missing-partition errors surface now instead of at flip time.

Record a screenshot or the raw JSON snapshot under bench/soak-reports/release1_soak.md. One sample every 4 h across the 48 h window is enough.

Release 2 — flip (target: 2026-04-24, 48 h after Release 1)

Pre-flip checklist:

Soak pass conditions above hold across every sample.
On-call is paged into the #analytics channel for the next hour.
bench/REPORT_usage-analytics-load.md numbers re-verified against the current prod load (spot-run bench/k6/run.sh usage-analytics-load at 25% rate — full 10k would impact users).

Flip: via the Admin Settings UI (/admin/settings → Analytics group → Enable analytics toggle → Save). Alternatively:

curl -u admin:... -X PATCH \
  -H 'Content-Type: application/json' \
  -d '{"value":"true"}' \
  https://api.gitvetrix.com/api/v1/admin/settings/analytics.enabled

Immediately after flip, watch /admin/analytics/health for 5 minutes. Expected:

enqueued starts climbing (10–30 events/s for a typical admin-heavy hour; higher under real traffic).
written tracks enqueued within a few seconds.
dropped stays at 0; a sudden jump > 100 indicates a DB write bottleneck — rollback is analytics.enabled=false via the same PATCH.
queue_depth stays bounded (channel is 10 000; we expect single-digit utilisation).

Announce:

UsageAnalytics is live (Release 2). Operators: drop counters are visible on /admin/analytics/health; daily review for the next week.

Daily drop-counter review (week after flip)

Runs for 7 consecutive days post-flip.

Daily checklist:

Snapshot /admin/analytics/health.
Confirm:
- dropped / enqueued ≤ 0.1% across the prior 24 h.
- queue_depth mean ≤ 100 at any 1-min sample.
- ratelimit_redis_unavailable stable (no step increase).
Log the sample in bench/soak-reports/postflip_dayN.md.

Exit conditions:

7 days clean → close out the rollout with a link to the final day's report.
Any day with > 0.1% drops or > 1 % 5xx rate → raise the Recorder channel cap (analytics.recorder_buf_size), file a follow-up investigation ticket, and delay the close until the next 7-day clean window starts.

No backfill

Historical data stays in audit_log and access.log. The Recorder does not replay anything that happened before Release 2. Callers querying the dashboards for pre-flip dates see empty buckets, which is correct.