Vetrix Docs

Rate-limit rollout playbook

Companion to rate-limit-shadow-to-enforced.md, which handles the actual shadow → enforced flip.

This document describes how the rate-limit subsystem ships initially and how the 48-h soak is operated. Every release runs with ratelimit.shadow_mode=true for the first 48 h on production; this playbook makes that operable.

1. Ship configuration

The release pins:

Setting Value Rationale
ratelimit.enabled true Master switch on — the middleware records everything.
ratelimit.shadow_mode true Log-only; no user is ever throttled during the soak.
ratelimit.violation_retention_days 30 Default retention window.

The seed rows in rate_limit_groups come from internal/ratelimit/seeds.go. Admin tier is action=allow (metered, never denied); General and Anonymous are action=throttle (denied when over budget, if shadow_mode=false).

During the soak, the yellow banner on /admin/rate-limits/settings surfaces the shadow-mode state so any operator hitting the page knows "nothing is actually being blocked".

2. Soak cadence

Fetch the following at T+0, T+24 h, T+48 h:

  1. /admin/rate-limits — screenshot the Throttled (24 h) stat card and the 3×7 matrix. Note the action chip on every cell.
  2. /admin/rate-limits/violations?action_taken=shadow — export CSV:
    curl -sS -H "Authorization: Bearer $ADMIN" \
  "https://api.gitvetrix.com/api/v1/admin/rate-limits/violations.csv?limit=500" \
  > soak-${TIMESTAMP}.csv
  3. /admin/health — capture the ratelimit.enqueued / ratelimit.written / ratelimit.dropped counters.

Archive all three snapshots together at each interval.

3. Soak report template

Compile the following as the final soak report at T+48 h.

# Rate-limit soak report — YYYY-MM-DD (T+48 h)

## Snapshots
- T+0 snapshot: _(paste link here)_
- T+24 h snapshot: _(paste link here)_
- T+48 h snapshot: _(paste link here)_

## Traffic summary
- Total `shadow_mode=true` violations recorded: **N**
- Unique users throttled: **M**
- Unique repos touched: **K**

## Top 10 throttled users
<paste from violations.csv, pivot by user_id>

## Top 10 throttled repos
<paste from violations.csv, pivot by repo_id>

## Budget safety check
For each scope, `(shadow_mode throttles over 48 h) / (expected budget
over 48 h)`. The threshold is **5×** — anything over gets a widened
seed before the shadow → enforced flip runs.

| Scope | Shadow throttles | Budget at 48 h | Ratio | Action |
|-------|------------------|----------------|-------|--------|
| api.read      | | | | |
| api.write     | | | | |
| api.expensive | | | | |
| git.read      | | | | |
| git.write     | | | | |
| registry      | | | | |
| beacon        | | | | |

## False-positive check
No user reports of false 429s during the soak (shadow mode cannot
actually throttle; confirm the support queue is clean).

## Recommendations
- [ ] Scope X widened from Y to Z (see link).
- [ ] OK to proceed with the shadow → enforced flip.
- [ ] (Or) Re-soak required.

4. Proceed conditions

Proceed to the shadow → enforced flip only when:

  • All seven scopes' safety factor ≤ 5×.
  • No support reports of false 429s.
  • ratelimit.dropped counter is < 0.1 % of ratelimit.enqueued.
  • Redis connectivity green on /admin/health throughout the soak.
  • Any scope that exceeded the safety factor has had its seed row widened before the flip; re-soak a minimum of 24 h after widening.

Abort and re-soak if any condition fails.

  • Dashboard landing: https://www.gitvetrix.com/admin/rate-limits
  • Violations feed: https://www.gitvetrix.com/admin/rate-limits/violations
  • Live tail: https://www.gitvetrix.com/admin/rate-limits/live
  • Settings page (shadow-mode banner): https://www.gitvetrix.com/admin/rate-limits/settings
  • Per-user self-service: https://www.gitvetrix.com/user/rate-limits
  • Per-user history: https://www.gitvetrix.com/user/rate-limits/history