Code search-index bulk-backfill

Operator runbook for the POST /api/v1/admin/search/code/backfill admin endpoint (sibling of the /admin/search/pages/backfill handler).

This endpoint walks every repository's default branch on disk and populates code_search_index through search.Indexer.IndexRepository. Each upsert also enqueues a search_index_outbox row, and the outbox scheduler then dispatches those rows to the CodeAdapter, which writes to the vetrix-code and vetrix-symbols OpenSearch indexes (symbol fan-out via universal-ctags).

A single POST therefore primes both the Postgres fallback path AND the OpenSearch leg used by /api/v1/search?types=code and /api/v1/search/code.

When to use

Run this endpoint when any of the following are true:

A fresh deployment has just come online and code search is returning zero hits for every query. This is the canonical reproduction: nothing on the producer side has written rows into code_search_index yet because no post-receive hook fired before the deployment began serving requests.
The CI smoke test (TestCodeSearchSmoke_FreshDeploymentReturnsHitsForSeededToken) has failed and the recovery checklist in its output names this endpoint.
An OpenSearch reindex (mapping change, cluster recreate) has emptied the vetrix-code or vetrix-symbols index and you need the scheduler to refill from a known-good Postgres source of truth.
An operator has observed code_search_index drift relative to the on-disk git trees (e.g. a long-running mirror finished re-importing repos and the indexer was not invoked per push).

Do not use the general POST /api/v1/admin/search/reindex endpoint to recover from "code search returns zero hits". That endpoint walks every registered EntityIndexer's BulkBackfill, and CodeAdapter.BulkBackfill walks the (already empty) code_search_index table — re-enqueueing zero rows produces zero hits. The general endpoint can re-fan-out an already populated index but cannot prime an empty one.

Required auth

acl.AdminSystem scope, enforced at the route layer via requireAdminScope. The bearer JWT must belong to an admin with the AdminSystem grant.

Condition	Response
No `Authorization` header	`401`
Token with `is_admin=false`	`403`
OpenSearch backend not configured	`503 "opensearch backend not configured"`
Search-index pipeline (repos / indexer / branches / outbox) not wired	`503 "search-index pipeline not configured"`

The OpenSearch gate stays in place even though the handler writes to Postgres first — the whole point of the call is to land documents in the OpenSearch indexes via the scheduler, and running with an unreachable cluster would stall the outbox immediately.

How to invoke

curl -X POST \
  -H "Authorization: Bearer $ADMIN_JWT" \
  https://api.gitvetrix.com/api/v1/admin/search/code/backfill

No request body is required — the endpoint walks every repository the deployment can see via git.Store.ListAccessibleRepoIDs(_, _, true) (the isAdmin=true branch bypasses the ACL filter so every repo is included).

Response

202 Accepted with:

{
  "repos_walked": 47,
  "files_indexed": 3812,
  "errors": 0,
  "queue_depth": 3812,
  "status": "queued"
}

Field	Type	Meaning
`repos_walked`	int	Number of repositories successfully walked. A repo that returned an error mid-walk is NOT counted here — it lands in `errors` instead.
`files_indexed`	int	Best-effort delta between the pre-walk and post-walk `code_search_index` row count, summed across repos. Idempotent re-runs report `0` for a repo whose every file already matches the index.
`errors`	int	Number of repos that failed (missing default branch, broken pack, oversized blob, etc.). A non-zero value does NOT 5xx the request — partial backfill is still useful.
`queue_depth`	int	Post-walk `search_index_outbox` queue depth from `outbox.Stats`. Use to correlate the kickoff against the outbox scheduler dashboard.
`status`	string	Always literally `"queued"` — the actual OpenSearch writes happen off-band in the scheduler when it claims the freshly-enqueued outbox rows.

The handler does not spawn goroutines. By the time the 202 is written, every successful UpsertFile is committed; the scheduler will pick up the matching outbox rows on its next tick.

Monitoring progress

1. Poll the queue-depth endpoint

curl -H "Authorization: Bearer $ADMIN_JWT" \
  https://api.gitvetrix.com/api/v1/admin/search/reindex/status

Watch per_entity.code count down to 0. The queue_depth total is the sum across all entity types; if only the code backfill is draining, the two should agree.

2. Compare Postgres vs OpenSearch document counts

SELECT count(*) FROM code_search_index;

curl -H "Authorization: Bearer $OPENSEARCH_BASIC_AUTH" \
  https://$OPENSEARCH_HOST/vetrix-code/_count

At steady state the two counts should agree. A persistent gap after the outbox drains indicates a per-row index error — check the application error log for CodeAdapter.Index lines and follow the "Failure modes" section below.

3. Probe `/api/v1/search/code` directly

curl -H "Authorization: Bearer $ADMIN_JWT" \
  "https://api.gitvetrix.com/api/v1/search/code?q=package&limit=5"

A non-empty files array with total_files >= 1 confirms the producer-side pipeline is healthy. A total_files: 0 response after a successful backfill means the OpenSearch leg has not drained yet — re-check queue_depth. If queue_depth is also 0 and search still returns zero hits, see "Verifying the OpenSearch leg" below.

4. Watch the access log

grep '"POST /api/v1/admin/search/code/backfill"' /var/log/vetrix/access.log

One line per kickoff. The audit-log row recorded for the same call carries the per-call counters (repos_walked, files_indexed, errors, queue_depth) for post-hoc reconciliation.

Verifying the OpenSearch leg

The CI smoke test exercises the Postgres fallback path (search.Store.SearchCodeInRepos) because the CI runner image does not ship an OpenSearch sidecar. After every fresh deployment, run the manual OpenSearch-leg check below to confirm the OpenSearch leg:

# 1. Confirm the scheduler drained the outbox.
curl -H "Authorization: Bearer $ADMIN_JWT" \
  https://api.gitvetrix.com/api/v1/admin/search/reindex/status
# Expect per_entity.code == 0 once the backfill is done.

# 2. Confirm vetrix-code carries documents.
curl -H "Authorization: Bearer $OPENSEARCH_BASIC_AUTH" \
  https://$OPENSEARCH_HOST/vetrix-code/_count
# Expect >= the files_indexed value the backfill response reported.

# 3. Exercise the OpenSearch path end-to-end through the API.
curl -H "Authorization: Bearer $ADMIN_JWT" \
  "https://api.gitvetrix.com/api/v1/search?types=code&q=package"
# Expect a non-empty files[] in the unified-search response.

If step 3 returns hits but /api/v1/search/code does not, the deprecated per-type route is falling back to Postgres while the unified-search route reads OpenSearch — that is the expected behaviour today. Both routes are valid producer-side health probes.

Failure modes

OpenSearch is down

A 503 from this endpoint with body opensearch backend not configured means LoadOpenSearchClient() returned nil — either the cluster was unreachable at boot, or the ClientLoader has lost the connection. Recover the cluster first; the endpoint will start returning 202 again as soon as LoadOpenSearchClient() resolves (no API restart required).

Per-repo walk failures

The response body reports errors > 0 and repos_walked is below the total repo count. Common causes:

Symptom	Likely cause	Remediation
Empty `repos.default_branch` AND `git.Manager.DefaultBranch` returns ""	Bare repo with no HEAD (newly created, never pushed)	Push at least one commit to populate HEAD.
`IndexRepository` returns `git: object too large`	Repo carries a blob over the per-blob ceiling configured for the indexer	Bump `search.indexer.max_blob_size_bytes` or exclude the path via `.vetrixignore`.
`IndexRepository` returns `pack: …` corruption	On-disk corruption	Run `git fsck` on the disk path; re-clone from the upstream mirror if it is a managed mirror.

Per-repo errors do NOT 5xx the request — the handler walks the remaining repos. Re-running is safe: UpsertFile is idempotent on (repo_id, file_path, ref).

Scheduler stalled

If queue_depth does not drop after a successful enqueue, the outbox scheduler may have parked rows past max_attempts or hit the stuck-claim path. See:

search-reset-attempts-exceeded.md — rearm rows the Nack path parked past max_attempts.
The reset-stuck-rows companion endpoint — clear claims abandoned by a dead worker.

Both endpoints are safe to combine with this one: re-enqueue with /admin/search/code/backfill, then rearm with the reset endpoints, then poll queue_depth to confirm the queue is draining.

Restart during enqueue

The handler is fully synchronous — no goroutines, no in-memory state. A process restart mid-call cannot leave the outbox or code_search_index in an inconsistent state. If the call had time to write rows before the restart, those rows are committed and the scheduler will drain them. Re-run the endpoint after the restart; UpsertFile idempotency and outbox dedup absorb the overlap.

Steady-state regression after a clean backfill

If /api/v1/search/code starts returning zero hits after a clean backfill — for example, every push to repo X leaves the index stale while older repos still match — the post-receive hook is not invoking search.Indexer.IndexRepository. Steady-state pushes are kept current by the post-receive reindex hook; see code-search-incremental-indexing.md. If that hook is not firing, re-run this endpoint after batch imports or mass-push events as a workaround.

Audit trail

Every successful 202 records one audit_log row. To find recent backfill actions:

SELECT actor_id, action, resource, details, created_at
FROM   audit_log
WHERE  action = 'admin.code.bulk_backfill'
ORDER  BY created_at DESC
LIMIT  10;

details is JSON with:

{
  "actor_user_id": "<uuid>",
  "repos_walked":  47,
  "files_indexed": 3812,
  "errors":        0,
  "queue_depth":   3812,
  "scope":         "admin:system"
}

The resource column is search_index_outbox matching the search.reset_* and admin.pages.bulk_backfill family — the audit-log explorer groups every search-index recovery action on that resource string. Audit writes are best-effort and do not fail the mutation; the absence of an audit row does not mean the backfill did not run — cross-check repos_walked / files_indexed from the response body.

CI integration

A CI smoke test enforces this runbook end-to-end on every PR push that sets up a Postgres-backed test environment:

internal/api/code_search_smoke_test.go — TestCodeSearchSmoke_FreshDeploymentReturnsHitsForSeededToken seeds one repo via search.Store.UpsertFile, drives the production /api/v1/search/code handler, and asserts total_files >= 1. The test prints the recovery checklist (this endpoint plus the post-receive reindex hook) on a zero-hits failure.
The test runs as part of the go-integration-test CI job (vetrix-ci.yml) against an ephemeral Postgres. The go-test job skips it cleanly (no TEST_DSN) — that gap is covered by the manual operator runbook check documented above.
A companion TestCodeSearchSmoke_RouterSnapshot guards against a refactor that drops the /api/v1/search/code route entirely.

POST /api/v1/admin/search/reindex — general, every-entity-type reindex. Use it for an all-up reindex; use this endpoint for a code-only refill from an empty index.
code-search-incremental-indexing.md — the post-receive reindex hook (steady-state producer; this runbook is the one-shot recovery sibling).
pages-search-backfill.md — sibling runbook for the pages entity type.
search-reset-attempts-exceeded.md — companion endpoint for stalled outbox rows.
Code-search index operations (architecture) — how this endpoint fits into the code_search_index table and the search-index outbox internally: default-branch rename handling, orphan sweep, VACUUM/REINDEX, and related admin endpoints. This runbook is the authoritative operator procedure for the endpoint itself; the architecture page is for readers reasoning about the underlying data model.