Vetrix Docs

Architecture -- code-search index: default-branch repopulation and operations

Scope: how the code_search_index table and the search-index outbox work internally -- repopulating the default branch, handling a default_branch rename, the parked/stuck outbox states, the post-cleanup VACUUM/REINDEX step, the M5 branch-delete rate limit, and the L3 audit-log retention policy. This page documents endpoint mechanics and the underlying data model for readers reasoning about or changing the platform; it is no longer the one-page operator reference. For the step-by-step operator procedures themselves, follow the runbooks it links to inline below (backfill, reset attempts-exceeded, and stranded claims each have their own authoritative runbook).

Audience: platform operators with admin:system scope. All admin endpoints below are gated by requireAdminScope(acl.AdminSystem) at the router layer.

1. Repopulate the default branch (prime an empty code_search_index)

Use this when

  • /api/v1/search/code returns zero hits for ubiquitous tokens (func, package, README) -- the table was never primed.
  • After a deploy onto a fresh cluster, or after orphan reconciliation removed so many rows the live default-branch set needs re-seeding.

How this works

There is no POST /api/v1/admin/search/reindex?repo_id=... shape. The handlers that exist:

  • POST /api/v1/admin/search/reindex -- internal/api/admin_search_reindex.go:142 (AdminSearchReindexHandler.Start). Takes no repo_id query param and accepts no request body. It walks every registered EntityIndexer.BulkBackfill and re-enqueues existing search_index_outbox rows only. CodeAdapter.BulkBackfill walks the existing code_search_index table -- so on an empty table this endpoint enqueues zero code rows and cannot prime the default branch. Use it to re-fan an already-populated index to OpenSearch, not to seed one.
  • POST /api/v1/admin/search/code/backfill -- internal/api/admin_search_code_backfill.go:251 (AdminSearchCodeBackfillHandler.Run). This is the endpoint that actually produces default-branch rows. It enumerates every repository in the deployment, resolves each repo default branch (repository.default_branch; if that column is empty it falls back to the on-disk git HEAD via git.Manager.DefaultBranch), and for each repo calls search.Indexer.IndexRepository(ctx, repo.DiskPath, repo.ID, ref) with ref = that default branch. IndexRepository (internal/search/code.go:84) walks every text blob at ref and transactionally UpsertFiles it into code_search_index with ref = <default_branch> and enqueues the matching search_index_outbox code row in the same transaction. The scheduler then dispatches those rows to OpenSearch (vetrix-code + vetrix-symbols). This endpoint is deployment-wide -- it backfills every repo default branch in one call. There is no per-repo (?repo_id=) variant.

Authoritative operator procedure: invocation, auth requirements, the full gate/response-code list, the response-field table, monitoring steps, and failure modes are maintained in code-search-backfill.md in the runbook set -- follow that runbook, not this page, when actually running the backfill. This page documents how the endpoint works internally; it is not the operator procedure.

Per-repo fallback (CLI seam)

When you need to re-seed a single repository rather than the whole deployment, drive the same producer seam search.Indexer.IndexRepository(ctx, repo.DiskPath, repo.ID, repo.DefaultBranch) directly. The one-shot CLI for the code-index producer/cleanup family:

# Reconcile + (dry-run) inspect the code index.
# CAVEAT: `reconcile-code-index` is available only on builds that include
# the reconcile-code-index subcommand; on older builds use POST
# /api/v1/admin/search/code/backfill instead.
docker exec mydev_vetrix /usr/local/bin/vetrix-cli reconcile-code-index --dry-run
docker exec mydev_vetrix /usr/local/bin/vetrix-cli reconcile-code-index

reconcile-code-index walks every (repo_id, ref) pair, drops orphan refs, and (without --dry-run) is the single-shot operator path that shares the exact code path as the cron orphan sweep. Use it to verify 0 orphans found after a repopulation. The reconcile-code-index subcommand is available only on builds that include it; on older builds the deployment-wide /admin/search/code/backfill endpoint above is the supported path.

Verify the rows are default-branch rows

SELECT ref, count(*) AS files
FROM   code_search_index
GROUP  BY ref
ORDER  BY files DESC;
-- Expect the bulk of rows under ref = each repo default_branch
-- (e.g. main); orphan/feature refs should be absent or tiny.

2. default_branch rename handling

No special operator action is required. When a repository default branch is renamed (e.g. master -> main):

  • New pushes / merges index under the new default-branch ref via the post-receive / merge-side IndexRepository path.
  • Rows still carrying the old ref are orphans (no live git ref points at them). The cron orphan sweep reaps them within one sweep cycle.
  • Sweep cadence is the setting search.code_index_orphan_sweep_cron, default */5 * * * * (every five minutes). So a default-branch rename stale rows are gone within ~5 minutes with the default cron. An absent setting (no app_settings row -- the state of a fresh or restored install) resolves to this */5 default, so the orphan GC is on by default -- you do not have to set the key for pruning to happen. To explicitly disable the GC, set the value to off (also accepted: disabled, none, or an empty string); a malformed cron is rejected at admin-write time and, if it ever reaches the scheduler, falls back to */5 rather than never running.
  • Each sweep cycle that deletes orphans writes exactly one system.code_index.orphan_sweep audit row recording orphans_deleted=N.

To force immediate cleanup instead of waiting for the next cycle, run vetrix-cli reconcile-code-index (section 1, fallback) -- it shares the sweep code path and removes the old-ref orphans on the spot. Caveat (same as section 1): the reconcile-code-index subcommand is available only on builds that include it. On a build without it, do not wait-loop on a missing command -- the cron orphan sweep (search.code_index_orphan_sweep_cron, default */5 * * * *) already reaps the old ref within one cycle (~5 min), and the deployment-wide POST /api/v1/admin/search/code/backfill (section 1) is the supported path.

3. Clear a parked / stuck outbox and verify drain

If a transient OpenSearch outage burned through the retry budget, code (and possibly issue/repo) outbox rows park at attempts >= max_attempts and the scheduler stops claiming them.

Re-arming parked rows

  • POST /api/v1/admin/search/reset-attempts-exceeded -- internal/api/admin_search_reindex.go:410 (AdminSearchReindexHandler.ResetAttemptsExceeded). Re-arms parked rows (attempts -> 0, next_attempt_at -> now()) so the scheduler re-picks them up. Admin scope + ScopeAPIWrite. Only touches Postgres -- no OpenSearch gate (its whole purpose is recovering from an OpenSearch outage).

    Authoritative operator procedure: the request/response shape, the older_than / limit clamp parameters, invocation, and the audit trail are maintained in search-reset-attempts-exceeded.md in the runbook set -- follow that runbook, not this page, when actually re-arming rows.

  • For the distinct dead-worker rows stuck with claimed_at != NULL condition, use POST /api/v1/admin/search/reset-stuck-rows (internal/api/admin_search_reindex.go:277) instead -- same scope, optional threshold_seconds (60s floor; default 1h), search.reset_stuck_rows audit action.

    Authoritative operator procedure: the dashboard UI affordance (Reset stranded claims button), invocation, and how this condition differs from attempts-exceeded are maintained in the Stranded claims section of the Search Indexing operator runbook -- follow that runbook, not this page, when actually resetting stranded claims. This page documents how the endpoint works internally; it is not the operator procedure.

Verify the queue is draining

Poll the status endpoint (read-only, Postgres-only, always 200 -- works even mid-OpenSearch-outage):

curl -fsS -H "Authorization: Bearer $ADMIN_TOKEN" \
  "$API/api/v1/admin/search/reindex/status"
# -> 200 {"queue_depth":N, ... ,"attempts_exceeded":N, ...}

queue_depth should fall toward 0 over a couple of scheduler poll cycles (default search.poll_interval = 5s, search.batch_size = 500). attempts_exceeded should return to 0 after a successful reset + drain. Cross-check directly:

SELECT count(*) FROM search_index_outbox;                                  -- 0 when drained
-- :max_attempts below is the operator-configurable setting
-- search.max_attempts (internal/admin/settings.go:341, range 1..50;
-- search.SettingKeyMaxAttempts / DefaultMaxAttempts=10). Substitute
-- the configured value; if the setting has not been tuned it is 10.
SELECT count(*) FROM search_index_outbox WHERE attempts >= :max_attempts;  -- 0 after reset+drain (:max_attempts default 10)

4. Post-cleanup VACUUM / REINDEX

After a large orphan reconciliation (rows can drop from hundreds of thousands to a few thousand), reclaim space and refresh planner stats:

docker exec mydev_postgres psql -U vetrix -d vetrix \
  -c "VACUUM (ANALYZE, VERBOSE) code_search_index;"

VACUUM (ANALYZE) is the required step -- it reclaims dead tuples from the bulk delete and updates statistics so the GIN/index plans stay sane. Optional, only if GIN index bloat persists after the vacuum:

docker exec mydev_postgres psql -U vetrix -d vetrix \
  -c "REINDEX TABLE CONCURRENTLY code_search_index;"

REINDEX ... CONCURRENTLY avoids an exclusive table lock; run it off-peak and confirm no leftover INVALID indexes afterwards (\d code_search_index).

Verify storage dropped:

SELECT count(*), count(DISTINCT ref),
       pg_size_pretty(pg_total_relation_size('code_search_index')) AS size
FROM code_search_index;

5. M5 branch-delete rate limit

The Mode 1 (UI/API) branch-delete path is rate limited:

  • 30 branch deletes per 5-minute window, per authenticated actor.
  • On exceed: HTTP 429 with a Retry-After header.
  • The limiter fails open when REDIS_URL is not configured -- i.e. with no Redis the limit is not enforced and deletes are not blocked (availability over enforcement for this surface).

Operator note: a burst of legitimate cleanup (e.g. a script deleting many merged branches) can trip the 429. Either pace the deletes under 30 / 5 min per actor, or distribute across actors. Deleting a branch via the API also triggers DropRef (per-row code/delete outbox enqueue), so a large delete burst additionally produces outbox load -- watch queue_depth (section 3).

6. L3 audit-log retention policy (documented policy -- NOT enforced in code here)

audit_log has no TTL. The branch-delete GC and orphan sweep add a steady trickle (cron sweep ~ a few hundred rows/day plus N/day from branch deletes). The policy:

  • repo.branch.delete and repo.branch.delete.failed (audit actions defined at internal/api/branches.go:298-299): these are user actions -- keep indefinitely. Compliance / incident-forensics requirement; never age these out.
  • system.code_index.orphan_sweep (system-actor action emitted by the cron orphan sweep): operational only -- age out after 90 days. These rows are housekeeping telemetry, not a compliance record.

This is a documented retention policy only. No TTL job, migration, or pruning code is shipped by this work -- enforcement (a scheduled prune that deletes system.code_index.orphan_sweep rows older than 90 days while preserving all repo.branch.delete* rows) is intentionally out of scope here and tracked separately. Until an enforcement job exists, an operator may apply the policy manually:

-- Operational-only rows, safe to prune after 90 days. NEVER include
-- repo.branch.delete* in this predicate.
DELETE FROM audit_log
WHERE  action = 'system.code_index.orphan_sweep'
  AND  created_at < now() - interval '90 days';

repo.branch.delete* rows are out of any prune predicate by design.

Quick reference

Task Endpoint / command Notes
Prime/repopulate default branch (all repos) POST /api/v1/admin/search/code/backfill No body/params; writes code_search_index + outbox; admin:system
Per-repo re-seed (fallback) vetrix-cli reconcile-code-index [--dry-run] IndexRepository(repo, default_branch) seam. Available only on builds that include the reconcile-code-index subcommand; on older builds use POST /api/v1/admin/search/code/backfill (deployment-wide) instead
Re-fan existing index to OpenSearch POST /api/v1/admin/search/reindex Outbox re-enqueue only; cannot seed an empty table
Re-arm parked outbox rows POST /api/v1/admin/search/reset-attempts-exceeded older_than / limit clamps; audited
Re-arm dead-worker stuck rows POST /api/v1/admin/search/reset-stuck-rows threshold_seconds; audited
Verify drain GET /api/v1/admin/search/reindex/status Read-only; 200 even mid-outage
Reclaim space post-cleanup VACUUM (ANALYZE) code_search_index REINDEX ... CONCURRENTLY optional
default_branch rename (no action) sweep search.code_index_orphan_sweep_cron */5 * * * * Old ref reaped within one cycle
Branch-delete rate limit 30 / 5 min / actor 429 + Retry-After; fail-open without REDIS_URL
Audit retention policy only (not enforced here) keep repo.branch.delete* forever; age system.code_index.orphan_sweep at 90d