Vetrix Docs

Reconciling dead-lettered search ACL-sync / delete outbox rows

The search index is fed asynchronously through the search-index outbox (table search_index_outbox). Every change that must reach the index — including a repo's ACL fan-out and repo deletions — is enqueued as an outbox row and applied by the indexer with bounded retries. Repo-scoped work is carried on entity_type='repo' rows:

  • the ACL fan-out (SyncACLForRepo) rides an op='upsert' row, and
  • a repo deletion (DeleteAllDocsForRepo) rides an op='delete' row.

(op is constrained to upsert / delete by a CHECK constraint — there is no sync acl op.) When a row exhausts its retries the Nack path dead-letters it: it leaves claimed_at NULL and pushes next_attempt_at ~100 years into the future (the AttemptsExceededInterval = '1 year' horizon), so the indexer's next_attempt_at <= now() claim predicate can never re-pick it. The counter search_outbox_dead_lettered_total is incremented.

This runbook is the canonical recipe for re-arming dead-lettered repo ACL-sync / delete rows so the indexer re-applies them, and for repairing or re-backfilling an index that has fallen out of sync.

There is no stored attempts_exceeded column. "Dead-lettered" is a derived predicate — claimed_at IS NULL AND next_attempt_at > now() + INTERVAL '1 year' — that the dashboard count (Stats.AttemptsExceeded, a COUNT(... ) FILTER (...) alias), the re-arm endpoint, and the fail-closed search exclusion all share. Every query below keys off that predicate, not a boolean flag.

1. The alert: search_outbox_dead_lettered_total rising

A rising search_outbox_dead_lettered_total means search-index outbox rows have exhausted their retries and were dead-lettered. For repo-scoped work this is the cases that matter:

  • entity_type='repo', op='upsert' — a repo's ACL block (who may see its docs in search) failed to apply and was parked. The index now holds a stale ACL for that repo, or that repo's docs are excluded entirely (see §4).
  • entity_type='repo', op='delete' — a repo deletion failed to propagate to the index. Without reconciliation the deleted repo's docs can linger in the index.

The dead-letter never clears itself: a parked row is not retried. Operator reconciliation is required to drain it.

Triage first

Inspect the parked rows before re-arming, to understand scope and the failure reason:

-- How many rows are parked, by entity_type / operation?
SELECT entity_type, op, COUNT(*)
  FROM search_index_outbox
 WHERE claimed_at IS NULL
   AND next_attempt_at > now() + INTERVAL '1 year'
 GROUP BY entity_type, op
 ORDER BY 3 DESC;

-- Which repos are affected, and why did they fail?
-- (entity_id is the repo id on entity_type='repo' rows.)
SELECT entity_id, op, attempts, last_error, enqueued_at
  FROM search_index_outbox
 WHERE entity_type = 'repo'
   AND claimed_at IS NULL
   AND next_attempt_at > now() + INTERVAL '1 year'
 ORDER BY enqueued_at DESC
 LIMIT 50;

If last_error points at a transient cause (search cluster unreachable, an index in index_not_found, a timeout), the row is a good candidate to re-arm once the underlying cause is fixed. If it points at a malformed payload, fix the root cause first — re-arming a row that will only fail again just refills the dead-letter.

2. Re-arm the dead-lettered rows

Re-arming is exposed as an admin HTTP endpoint backed by ResetAttemptsExceededRows (implemented in internal/searchindex/outbox_stats.go in the vetrix repo). It does not flip a flag: for every parked row (claimed_at IS NULL AND next_attempt_at > now() + INTERVAL '1 year') it sets attempts = 0, next_attempt_at = now(), and last_error = NULL, leaving op and payload untouched. That makes the rows eligible for re-claim again, so on the next indexer pass they are picked up and the ACL fan-out / delete re-applies.

Call the endpoint (admin bearer token, acl.AdminSystem scope):

# Re-arm every dead-lettered outbox row (drains the dead-letter).
curl -fsS -X POST \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  https://<vetrix-host>/api/v1/admin/search/reset-attempts-exceeded
# -> 200 {"reset_count": N, "older_than_seconds": 0, "limit": 0}

Two optional clamps are accepted as ?older_than= / ?limit= query params (or the same keys in a JSON body):

  • older_than — a duration (30m, 2h, 24h) or bare seconds; only rows enqueued before now() - older_than are reset. Scope a re-arm to "rows from before today's incident window" so you don't clobber a row that just exhausted.
  • limit — cap the rows touched in one call (a safety throttle for a fleet-wide backlog); re-run to drain in chunks.
# Example: re-arm at most 500 rows enqueued more than 2h ago.
curl -fsS -X POST \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  "https://<vetrix-host>/api/v1/admin/search/reset-attempts-exceeded?older_than=2h&limit=500"

The endpoint only touches Postgres and is deliberately not gated on OpenSearch reachability — its whole purpose is to recover from an OpenSearch outage. Every successful reset records an audit row (action search.reset_attempts_exceeded).

Confirm the drain and re-apply

-- Should trend to 0 after the reset + an indexer pass.
SELECT COUNT(*) FROM search_index_outbox
 WHERE claimed_at IS NULL
   AND next_attempt_at > now() + INTERVAL '1 year';

Watch search_outbox_dead_lettered_total stop rising and the indexer backlog drain. Once the re-armed repo upsert rows apply, the affected repos' ACL blocks are correct again and their docs return to search coverage (see §4).

3. Index repair / re-backfill

Re-arming only replays the parked rows. If an index has drifted far enough that replay alone won't restore it — or you simply want to rebuild the index from the source of truth — enqueue a full backfill, then re-arm so any still-parked repo upsert / delete work lands on the rebuilt index.

# 1. Enqueue a full backfill (walks every registered entity indexer
#    and inserts an upsert row per entity). The actual indexing happens
#    off-band in the search-index scheduler; this only enqueues.
docker exec <vetrix-server-container> \
  /usr/local/bin/vetrix-cli search backfill --all
# Add --watch to block until the backfill queue drains:
#   vetrix-cli search backfill --all --watch

# 2. Re-arm any rows still parked so repo upsert / delete replay.
curl -fsS -X POST \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  https://<vetrix-host>/api/v1/admin/search/reset-attempts-exceeded

vetrix-cli search backfill is a thin client over the admin endpoints POST /api/v1/admin/search/reindex (enqueue) and GET /api/v1/admin/search/reindex/status (queue stats, polled by --watch). The backfill is fleet-wide — there is no per-index flag; the server walks every registered EntityIndexer.

4. The fail-closed tradeoff

A dead-lettered repo ACL-sync row causes that repo's docs to be excluded from search — the behavior fails CLOSED, not open. This is the safe direction: a repo whose ACL could not be synced is hidden from search rather than served with a stale (possibly over-permissive) ACL.

The operational consequence is degraded search coverage:

  • Each dead-lettered repo drops out of site-search results until its ACL-sync row is re-armed and re-applied.
  • Once more than 1000 repos are dead-lettered, site-search degrades to pages-only — code/symbols/issues/MRs stop returning while the dead-letter is that large. (The exclusion list is capped at DeadLetteredRepoMax = 1000; past that the query layer fails closed wholesale rather than serve a partially-fenced set.)

So reconciliation serves two ends:

  1. Restore coverage — re-arming brings the excluded repos (and, past the 1000-repo threshold, the non-pages indexes) back into search.
  2. Clear the bounded exposure — because the code fails closed, the ongoing exposure is bounded (stale-ACL docs are no longer served from a dead-lettered repo). Draining the dead-letter clears even that bounded state. Reconcile promptly when the alert fires.

5. Verify

-- No rows remain parked.
SELECT COUNT(*) FROM search_index_outbox
 WHERE claimed_at IS NULL
   AND next_attempt_at > now() + INTERVAL '1 year';
-- Expect: 0

-- No repo ACL-sync / delete rows backed up overall.
SELECT op, COUNT(*) FROM search_index_outbox
 WHERE entity_type = 'repo'
   AND op IN ('upsert', 'delete')
 GROUP BY op;
  • search_outbox_dead_lettered_total is flat (no new increments).
  • Spot-check site-search for a previously-dead-lettered repo: its docs return again, scoped to the correct (now re-synced) ACL.
  • If you re-backfilled in §3, confirm the affected index doc count matches the source of truth (vetrix-cli search backfill --all --watch exits once the queue drains to zero).

6. Manual reset (fallback)

Prefer the admin endpoint in §2. Only if it is unreachable, re-arm the parked rows directly with the same UPDATE that ResetAttemptsExceededRows runs — reset the attempt counter, make the rows claimable again, and clear the stale error. There is no boolean flag to clear:

UPDATE search_index_outbox
   SET attempts        = 0,
       next_attempt_at = now(),
       last_error      = NULL
 WHERE claimed_at IS NULL
   AND next_attempt_at > now() + INTERVAL '1 year';

Resolve the root cause (last_error from §1) before running this, or the rows will simply re-exhaust their retries and re-park.

References

  • internal/searchindex/outbox_stats.go (vetrix repo) — ResetAttemptsExceededRows, the re-arm mechanism.
  • POST /api/v1/admin/search/reset-attempts-exceeded — the operator re-arm endpoint. POST /api/v1/admin/search/reindex (via vetrix-cli search backfill --all) — full backfill.
  • Table: search_index_outbox. Counter: search_outbox_dead_lettered_total.