Vetrix Docs

Credential Rotation Runbook

You are here because a git push was rejected with a message like: remote: ERROR secret detected at path/to/file:12 (rule aws-access-token); rotate the credential and rewrite history before pushing — see https://www.gitvetrix.com/docs/security/rotation

That check ran in-process at pre-receive and blocked the push before the credential hit any shared ref. The commit only exists on your local branch — no one else has it yet.

This page tells you what to do next: (1) rotate the credential at its source, (2) remove the bad commit from your local history, (3) re-push safely, (4) escalate if needed.

1. Rotate first, always

Do not skip this step, even if you know the push was rejected. A secret visible on your workstation is a secret visible to any process, browser extension, shell-history exporter, or sync agent that runs on your machine. Assume it is compromised the moment it was committed.

Find the leaked value and rotate it at the issuing system:

AWS

  1. aws iam list-access-keys --user-name <you>
  2. aws iam create-access-key --user-name <you> — write down the new key.
  3. Update every consumer to use the new key (CI variables, ~/.aws/credentials, container env, Parameter Store / Secrets Manager entries).
  4. aws iam update-access-key --access-key-id AKIA… --status Inactive on the old key, test for 24 h, then delete-access-key.
  5. aws cloudtrail lookup-events --lookup-attributes AttributeKey=AccessKeyId,AttributeValue=AKIA… — review recent usage for anything you didn't do.

Google Cloud

  1. Identify the leaked service-account key: gcloud iam service-accounts keys list --iam-account=<sa-email>.
  2. gcloud iam service-accounts keys create new.json --iam-account=<sa-email>.
  3. Roll consumers over to new.json.
  4. gcloud iam service-accounts keys delete <leaked-key-id> --iam-account=<sa-email>.
  5. Audit via Logs Explorer → protoPayload.authenticationInfo.serviceAccountKeyName for the leaked key id.

Azure

  1. Portal → Azure Active Directory → App registrations → → Certificates & secrets.
  2. New client secret → save the value.
  3. Update consumers.
  4. Delete the leaked secret. (Azure does not support revoke-in-place — deletion is the rotation action.)
  5. Review sign-in logs under Enterprise applications → → Sign-in logs for the affected app id.

GitHub (personal access tokens & OAuth app client secrets)

  1. Settings → Developer settings → Personal access tokens. Find the token by its last-4 or its note.
  2. Regenerate token — copy the new value into whichever secret store consumes it.
  3. For OAuth apps: Settings → Developer settings → OAuth apps → → Generate a new client secret, then update consumers and remove the old secret.
  4. Audit via Settings → Security log (personal) or Organisation → Security → Audit log (org).

GitLab (personal access tokens, deploy tokens, CI variables)

  1. User settings → Access Tokens — click Revoke next to the leaked token.
  2. Create a replacement with the narrowest scope and shortest TTL that still works.
  3. Update every consumer (.gitlab-ci.yml masked variables, repo deploy tokens, CLI config).
  4. Audit via Admin area → Monitoring → Audit Events.

Stripe

  1. Dashboard → Developers → API keys.
  2. Roll the leaked restricted key (or the live/secret key for publishable/secret-key leaks). Stripe keeps the old key active for 12 h so you can roll consumers; after that it is automatically invalidated.
  3. Update STRIPE_SECRET_KEY / STRIPE_RESTRICTED_KEY everywhere it is consumed.
  4. Audit via Dashboard → Developers → Logs, filter on the API-key prefix (sk_live_…/rk_live_…).

Twilio

  1. Console → Account → API keys & tokens.
  2. Delete the leaked API key or auth-token pair.
  3. Create new API key with the same scopes; update consumers.
  4. If the primary account auth token leaked, go through Account → Auth tokens and run the auth-token rotation flow (creates a secondary token, promotes it, deletes the primary).
  5. Check Monitor → Alerts → Anomalies for any unexpected usage.

Generic API key / bearer token

No specific issuer known? Treat it like a root credential:

  1. Log in to the system it belongs to and revoke it in the admin / developer area.
  2. Issue a new one.
  3. Update every consumer.
  4. If you cannot identify the issuer from the rule id alone, run git show HEAD:<path> | sed -n '<line>p' on your local commit (remember, it's still there in your reflog until you rewrite history — step 2 below) to read the key, then search your password manager or provider console for the matching entry.

Private-key PEM blocks

-----BEGIN … PRIVATE KEY----- is the highest-severity case — private keys cannot be rotated "in place", they must be regenerated and re-trusted.

  1. Generate a new keypair: openssl genrsa -out new.pem 4096 (or ssh-keygen -t ed25519 -f new.pem).
  2. Distribute the new public key to every system that previously trusted the leaked one (SSH authorized_keys, JWT-signing consumer lists, mTLS peer bundles).
  3. Remove the old public key from those systems.
  4. Revoke any certificate that chained to the leaked key (CA → revoke; update CRL/OCSP).
  5. If the key signed artifacts (releases, container images, JWTs), re-sign the still-valid artifacts under the new key and burn the old signatures.

2. Remove the leaked commit from your local branch

The commit is still on your workstation. Fix that before pushing again.

The easy case — the last commit

git reset --soft HEAD~1
# Your working tree still has the files. Remove the secret, then:
git add -A
git commit -m "<your original message>"

Earlier in history — interactive rebase

git rebase -i <sha-before-the-bad-commit>^
# Mark the offending commit as `edit`, save, then:
git rm --cached path/with/secret
# Edit the file to remove the secret, then:
git add -A
git commit --amend
git rebase --continue

Years-old leaks — git-filter-repo

# One-shot global scrub (rewrites every commit that mentions the path):
git filter-repo --path path/with/secret --invert-paths
# Or redact specific text across history:
git filter-repo --replace-text <(echo 'AKIA[A-Z0-9]{16}==>REDACTED')

git filter-repo is the modern replacement for git filter-branch and BFG; both earlier tools are still viable but filter-repo is substantially faster and safer.

3. Re-push safely

Once you have rewritten history, push again. Vetrix's pre-receive scan will run a second time against the new commits; if they are clean the push succeeds.

git push --force-with-lease origin <branch>

--force-with-lease refuses the push if someone else pushed to the branch between your fetches — you won't accidentally erase a coworker's commit. If you used the easy-case flow above (reset --soft) on a brand-new commit that never reached the server, a plain git push works too.

If the scan still trips, Vetrix will print a new error line with the new path:line. You have another leak somewhere — re-enter at step 1.

4. Escalation & break-glass bypass

When to page the security team

Page security@gitvetrix.com immediately if any of the following is true:

  • The secret belongs to a production system or a shared service account.
  • The leaked value was committed to a repository ref that exists on the server (Vetrix's pre-receive blocks new pushes, but existing refs are out of scope).
  • You cannot identify which credential leaked from the rule id alone.
  • Audit logs show activity you did not authorise during the window between commit and rotation.
  • The leak is a private-key PEM, a long-lived cloud access key, or any credential with cross-tenant scope.

Include in your message: the rule id, the repository, the path, and what you've done so far.

The secrets.bypass push option (break-glass only)

Users granted PermSecretBypass (by default: instance_admin only — see security.pre_receive_bypass_role) can push a flagged commit by adding the secrets.bypass push option:

git push -o secrets.bypass=true origin <branch>

Use this path only when you have verified that the detected string is not actually a credential — for example, a test fixture, a rotated-and-revoked value kept for documentation, or a known false positive for which you will file a baseline entry immediately after. Every bypass writes an audit_log row with action scan.pre_receive_bypass, the pushed ref updates, and every rule id that was overridden — the security team reviews these.

The bypass path can be disabled instance-wide by setting security.pre_receive_bypass_enabled = false (admin settings → Security). If that toggle is off, the push will be rejected even for PermSecretBypass holders.

If the detected finding is a legitimate long-lived false positive (the regex rule is too loose for your repo), commit the fingerprint into .vetrix/tooling.baseline.sarif in the default branch and re-push; baselined fingerprints are silenced in pre-receive but still visible on the security dashboard for periodic review. The baseline stale-entry threshold is security.baseline_max_age_days (default 90).

References

  • secrets.md — secret-detection rules + remediation
  • scanning.md — full scan-type reference
  • ../../admin-docs/admin/audit-log.md — querying audit entries (filter action=scan.pre_receive_bypass for bypass history)