Vetrix Docs

Runbook — Rate limiter: shadow → enforced

This runbook flips the instance-wide rate limiter from shadow mode (log-only) to enforced mode (issuing 429 responses) after the 48-h soak passes.

Rollback is a single PATCH — no server restart is ever required. Read the full document before starting; the pre-flight checklist is non-optional.

Prerequisites

  1. Soak report is green.

    • All seven scopes have shadow_mode violation counts within their documented budgets (the documented seed × 1.0–2.0 under normal load).
    • Any scope whose count exceeded 5× the seed has been widened; the widened seed rows are live in rate_limit_groups.
    • Top-10 throttled users + top-10 throttled repos produced zero false-positive reports (shadow mode cannot actually 429, but the feed should still look reasonable).

  2. Observability is healthy.

    • /admin/rate-limits loads for you and returns the 3×7 matrix.
    • /admin/rate-limits/violations returns a non-zero row count.
    • /admin/health counters ratelimit.enqueued / ratelimit.written are non-zero and ratelimit.dropped is at or near zero.
    • Redis connectivity is green on the health endpoint.

  3. Credential-guessing defence is the existing sliding-window limiter (internal/ratelimit/ratelimit.go, NOT the token-bucket path). That surface is outside this flip — it remains on regardless of ratelimit.enabled / ratelimit.shadow_mode. Verify after the flip that a failed-login spike still locks the account.

Pre-flight checklist

  • Soak report filed with T+0 / T+24 h / T+48 h snapshots.
  • /admin/rate-limits/dashboard — replaced by the landing page + Throttled (24 h) home StatCard. Both load.
  • ratelimit.shadow_mode=true in admin_settings (verify via /admin/rate-limits/settings — the yellow banner must be visible).
  • Baseline throttled-user count for the last 30 min captured (expected post-flip ≤ 2× baseline).
  • Rollback PATCH command (below) copied into a shell tab, ready.

Execution

Step 1 — Announce in #eng (30 min ahead)

Flipping rate-limit enforcement on at $(date -u +%H:%MZ). 48-h soak
green. Rollback is a single setting PATCH; no restart.
Watching /admin/rate-limits for the next 30 min. Ping me if anything
looks off.

Step 2 — Flip the setting

From an admin shell (must hold admin:rate_limit):

curl -sS -X PATCH https://api.gitvetrix.com/api/v1/admin/settings/ratelimit.shadow_mode \
  -H "Authorization: Bearer $ADMIN_ACCESS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"value":"false"}'

Or via the admin UI: Admin → Rate limits → Settings → Shadow mode, uncheck, Save changes.

Step 3 — Verify enforcement

From a non-admin test session (e.g. a General-tier user):

# With the seed budget (general · api.read = 600/60s at time of flip),
# fire 605 GETs in 60s. The 601st should return 429.
for i in $(seq 1 605); do
  curl -sS -o /dev/null -w "%{http_code} " \
    -H "Authorization: Bearer $GENERAL_USER_ACCESS_TOKEN" \
    https://api.gitvetrix.com/api/v1/repos
done | tr ' ' '\n' | sort | uniq -c

Expected: some mix of 200s and 429s (the exact count depends on timing; 429 must be non-zero). Confirm the 429 response carries:

  • X-RateLimit-Scope: api.read
  • X-RateLimit-Limit: 600
  • X-RateLimit-Remaining: 0
  • X-RateLimit-Reset: <epoch-ms>
  • Retry-After: <seconds>

Step 4 — Observe for 30 min

  • Watch /admin/rate-limits (the Throttled (24 h) StatCard).
  • Watch /admin/rate-limits/live.
  • Watch Prometheus: ratelimit_throttled_total growth rate.

Abort criteria: any of

  • throttled-user rate > 2× the soak baseline,
  • P90 API latency > 2× pre-flip,
  • any critical workflow (CI, git clone) reports 429 on a non-burst traffic pattern.

Hit Rollback (below) immediately on any abort condition.

Rollback

Single setting PATCH — no restart:

curl -sS -X PATCH https://api.gitvetrix.com/api/v1/admin/settings/ratelimit.shadow_mode \
  -H "Authorization: Bearer $ADMIN_ACCESS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"value":"true"}'

Or via the UI: Admin → Rate limits → Settings → Shadow mode, check it back on, Save changes. The yellow banner reappears instantly.

Existing in-flight tokens do not need to be cleared — the limiter middleware reads ShadowMode() on every decision and reverts to log-only on the next request.

Post-flip

  • Confirm /admin/rate-limits/settings reflects ratelimit.shadow_mode=false.
  • Confirm the yellow shadow-mode banner is gone.
  • Confirm a failed-login spike against a real account still locks the account (credential-guessing defence independent of this flip).
  • File post-flip observation report — top-10 throttled users, any scope that over-shot the expected budget, any follow-ups for scope budget tuning.

Post-mortem template

Use this section if anything goes wrong.

# Rate-limit flip post-mortem — YYYY-MM-DD

## Timeline (UTC)
- T+0:       Flip executed via PATCH.
- T+Nmin:    First alarm / user report.
- T+Xmin:    Rollback PATCH issued.
- T+Ymin:    Rate-limiter back in shadow mode, users unblocked.

## What happened
<narrative>

## Why the soak missed it
<root cause — missing traffic pattern, missing scope, clock drift, etc.>

## Remediation
- [ ] Widen seed for scope(s) that over-shot.
- [ ] Add Prometheus alert on <metric>.
- [ ] Add traffic pattern to soak playbook.
- [ ] Re-run 48-h soak before retrying flip.