Pages search-index bulk-backfill

Operator runbook for the POST /api/v1/admin/search/pages/backfill admin endpoint.

This endpoint enqueues a search-index backfill for the page entity type only. It is the recovery step an operator runs after a Wave A / Wave B pages schema migration lands in production, so the OpenSearch document mapping picks up the new repo_id, folder_path, enabled and deleted_at columns. The scheduler then drains the freshly enqueued outbox rows in the background.

When to use

Run this endpoint once per environment after each of these pages schema migrations lands in production:

pages.repo_id ADD COLUMN + backfill.
pages.folder_path ADD COLUMN + backfill.
Repo-association binding.
pages.enabled ADD COLUMN.
pages.deleted_at ADD COLUMN (soft-delete).

For any other reason — index drift suspected, an OpenSearch reindex, the document mapping was rewritten — this endpoint is also the right tool. It is safe to run any number of times; the outbox is idempotent against the partial unique index, so a duplicate POST while the previous run is still draining simply reports enqueued: 0 for any already-pending upserts.

Do not use the general POST /api/v1/admin/search/reindex endpoint when only pages need refreshing — that walks every registered EntityIndexer and burns the scheduler budget on issues / MRs / repos / packages for no reason.

Required auth

acl.AdminSystem scope, enforced at the route layer via requireAdminScope. The bearer JWT must belong to an admin with the AdminSystem grant. Failure modes:

Condition	Response
No `Authorization` header	`401`
Token with `is_admin=false`	`403`
OpenSearch backend not configured	`503 "opensearch backend not configured"`
Search-index pipeline (outbox / registry) not wired	`503 "search-index pipeline not configured"`
Pages adapter not registered	`503 "pages search adapter not registered"`

How to invoke

curl -X POST \
  -H "Authorization: Bearer $ADMIN_JWT" \
  https://api.gitvetrix.com/api/v1/admin/search/pages/backfill

No request body is required — the endpoint walks the entire pages entity space. Future per-page filters (e.g. by repo_id) can be layered on without changing this call shape.

Response

202 Accepted with:

{
  "enqueued": 1234,
  "queue_depth": 1234,
  "status": "queued"
}

Field	Type	Meaning
`enqueued`	int	Number of outbox rows this call appended. `0` on a duplicate POST against a queue that already has every page pending — outbox idempotency.
`queue_depth`	int	Post-call queue depth from `outbox.Stats`. Use to correlate the kickoff against the dashboard snapshot at the moment of emission.
`status`	string	Always literally `"queued"` — the actual index writes happen off-band in the scheduler, not inline.

The handler never spawns a goroutine. By the time the 202 is written, the outbox enqueue is committed; the scheduler will pick the rows up on its next tick.

Monitoring progress

The scheduler drains the outbox at its configured rate (see searchindex.scheduler.batch_size admin setting; default 100 rows per tick, one tick per second). To monitor progress:

1. Poll the queue-depth endpoint

curl -H "Authorization: Bearer $ADMIN_JWT" \
  https://api.gitvetrix.com/api/v1/admin/search/reindex/status

Watch per_entity.page count down to 0. The queue_depth total is the sum across all entity types; if only pages were enqueued the two should agree.

2. Compare Postgres vs OpenSearch document counts

SELECT count(*) FROM pages WHERE deleted_at IS NULL;

curl -H "Authorization: Bearer $OPENSEARCH_BASIC_AUTH" \
  https://$OPENSEARCH_HOST/pages/_count

At steady state the two counts should agree within the pages.deleted_at IS NOT NULL rows that are not indexed. A persistent gap after the queue drains indicates a per-row index error — see "Failure modes" below.

3. Watch the application access log

grep '"POST /api/v1/admin/search/pages/backfill"' /var/log/vetrix/access.log

One line per kickoff. The audit-log row recorded for the same call carries the enqueued count for post-hoc reconciliation (see "Audit trail").

Failure modes

OpenSearch is down

A 503 from this endpoint with body opensearch backend not configured means LoadOpenSearchClient() returned nil — either the cluster was unreachable at boot, or the ClientLoader has lost the connection. Recover the cluster first; the endpoint will start returning 202 again as soon as LoadOpenSearchClient() resolves.

Partial backfill

The endpoint surfaces a partial enqueue on a per-page BulkBackfill error (the pages.listPageIDsAfter cursor walk fails, or outbox.BulkBackfill Postgres-side errors mid-loop). The response is still 202 with the partial enqueued count — the scheduler will process what was enqueued, and re-running the endpoint picks up from where the previous run stopped (the cursor walk restarts from uuid.Nil but the outbox dedup absorbs the overlap).

A partial result is normal and recoverable. If a single POST returns enqueued: 0 against a known-non-empty pages table, the cursor walk failed on its first page — check the application error log for a pages.Adapter: listPageIDsAfter line and resolve the underlying Postgres issue (locking, statement timeout, etc.) before retrying.

Restart during enqueue

The endpoint is fully synchronous — no goroutines, no in-memory state — so a process restart mid-call cannot leave the outbox in an inconsistent state. If the call had time to write rows before the restart, those rows are committed and the scheduler will drain them. Re-run the endpoint after the restart; outbox idempotency absorbs the overlap.

Scheduler stalled

If queue_depth does not drop after a successful enqueue, the scheduler may have parked rows past the retry budget or hit the stuck-claim path. See:

search-reset-attempts-exceeded.md — rearm rows the Nack path parked past max_attempts.
The reset-stuck-rows companion endpoint — clear claims abandoned by a dead worker.

Both endpoints are safe to combine with this one: re-enqueue with /admin/search/pages/backfill, then rearm with the reset endpoints, then poll queue_depth to confirm the queue is draining.

Audit trail

Every successful 202 records one audit_log row. To find the action:

SELECT actor_id, action, resource, details, created_at
FROM   audit_log
WHERE  action = 'admin.pages.bulk_backfill'
ORDER  BY created_at DESC
LIMIT  10;

details is JSON with:

{
  "actor_user_id": "<uuid>",
  "enqueued": 1234,
  "queue_depth": 1234,
  "scope": "admin:system"
}

The resource column is search_index_outbox matching the search.reset_* family — the audit-log explorer groups every search-index recovery action on that resource string. Audit writes are best-effort and do not fail the mutation; the absence of an audit row does not mean the backfill did not run — cross-check enqueued from the response body.

The general /admin/search/reindex endpoint — the every-entity-type sibling; use that for an all-up reindex, this one for a pages-only refresh.