Vetrix Docs

Reset search-index attempts-exceeded rows

Operator runbook for the POST /api/v1/admin/search/reset-attempts-exceeded admin endpoint.

This endpoint is the recovery escape hatch for the "transient-OpenSearch-outage-burned-the-retry-budget" condition: rows that the search-index outbox Nacked past max_attempts get parked with next_attempt_at ~1 year into the future. The scheduler's next_attempt_at <= now() predicate can never match those rows, so they stay stranded until an operator rearms them.

When to use

Use this endpoint after a transient OpenSearch outage when:

  1. OpenSearch has been confirmed healthy again, and
  2. the reindex status dashboard reports a non-zero attempts_exceeded count, or the SQL diagnosis below returns rows.

Do not use it while OpenSearch is still degraded — you will burn the retry budget a second time and end up back in the same state.

Diagnosis

Fetch outbox stats (also exposed in the admin UI):

curl -H "Authorization: Bearer $ADMIN_JWT" \
  https://api.gitvetrix.com/api/v1/admin/search/reindex/status

The attempts_exceeded field is the count of parked rows. Or query Postgres directly:

SELECT count(*)
FROM   search_index_outbox
WHERE  claimed_at IS NULL
  AND  next_attempt_at > now() + INTERVAL '1 year';

The claimed_at IS NULL filter mirrors the reset endpoint's predicate; in-flight rows are not parked-by-attempts and belong to the reset-stuck-rows runbook instead.

Required auth

acl.AdminSystem scope, enforced at the route layer via requireAdminScope. The bearer JWT must belong to an admin with the AdminSystem grant. A non-admin token returns 403. Calls made before the search-index pipeline is wired (cold-start / test-only deployments) return 503 with body search-index pipeline not configured.

Recovery

Request

curl -X POST \
  -H "Authorization: Bearer $ADMIN_JWT" \
  -H "Content-Type: application/json" \
  https://api.gitvetrix.com/api/v1/admin/search/reset-attempts-exceeded \
  -d '{"older_than": "1h", "limit": 1000}'

Request body (both fields optional; query-param equivalents ?older_than=&limit= accepted, body wins when both are supplied):

Field Type Default Meaning
older_than duration (no clamp) Only reset rows whose enqueued_at is older than this duration. Go time.ParseDuration format: "30m", "2h", "24h". A bare integer on the query string is interpreted as seconds.
limit int (no cap) Cap the number of rows touched in a single call. Use to chunk a fleet-wide recovery so a single UPDATE does not block the table. Rows are picked oldest-first by enqueued_at.

An empty body / {} resets every parked row with no clamp. A malformed body is treated as "no body" (the endpoint is a recovery tool — failing the call on a trailing-comma JSON glitch during an incident would be hostile).

Response

200 OK with:

{
  "reset_count": 42,
  "older_than_seconds": 3600,
  "limit": 1000,
  "older_than": "1h0m0s"
}

The clamps echo back so you can confirm what the SQL actually applied — e.g. if your older_than string fails time.ParseDuration the clamp is silently dropped and the response will show "older_than_seconds": 0.

5xx with verbatim error body if the UPDATE fails — the FE renders this as a toast (feedback_api_error_bodies convention).

What the UPDATE does

For each matched row:

  • attempts → 0 (clean slate for the exponential backoff)
  • next_attempt_at → now() (eligible for the next scheduler tick)
  • last_error → NULL
  • op / payload are not touched — the original upsert/delete intent is preserved.

This endpoint is UPDATE-only — it cannot delete rows. If parked rows need to be removed entirely rather than rearmed (for example, a producer bug enqueued a large batch of rows — code rows are the common case given their volume — that the producer no longer wants and that would only re-park on retry), the only mechanism is a raw SQL DELETE against search_index_outbox, scoped to the entity type and parked predicate you intend to clear. There is no admin-endpoint equivalent for deletion by design; run the DELETE before calling this endpoint so the reset does not needlessly re-arm rows you are about to discard.

Audit trail

Every successful reset records an audit_log row. To find the action:

SELECT actor_id, action, resource, details, created_at
FROM   audit_log
WHERE  action = 'search.reset_attempts_exceeded'
ORDER  BY created_at DESC
LIMIT  10;

details is JSON with:

{
  "reset_count": 42,
  "older_than_seconds": 3600,
  "limit": 1000,
  "scope": "admin:system"
}

The resource column is search_index_outbox. Audit writes are best-effort and do not fail the mutation; absence of an audit row does not mean the reset did not run — cross-check reset_count from the response.

Distinct from search.reset_stuck_rows (the stuck-claim recovery endpoint). Both share the AdminSystem scope but recover from different conditions; an audit reviewer should be able to tell them apart at a glance.

Aftercare

After a successful reset:

  1. Re-fetch /api/v1/admin/search/reindex/status and confirm attempts_exceeded is trending toward zero.
  2. Watch queue_depth — rearmed rows will move through the queue under normal scheduler pressure; a sudden spike followed by a steady drain is the expected shape.
  3. If attempts_exceeded does not drop, OpenSearch is likely still degraded — investigate the cluster before re-running the endpoint.
  • POST /api/v1/admin/search/reset-stuck-rows — companion endpoint for the distinct dead-worker condition (claimed_at != NULL past a threshold). Same AdminSystem scope; recovers a different condition than this endpoint.
  • code-search-backfill.md — companion endpoint for priming an empty code_search_index (a different recovery condition than a parked outbox).
  • Code-search index operations (architecture) — how this endpoint fits into the code_search_index table and the search-index outbox internally. This runbook is the authoritative operator procedure for the endpoint itself; the architecture page is for readers reasoning about the underlying data model.