Vetrix Docs

Search Indexing -- Operator Runbook

The /admin/search dashboard surfaces the live state of the search-index outbox + scheduler. This document is the counterpart runbook -- when the dashboard surfaces a warning, follow the section that matches the badge. The non-alert sections below (Tunables, Queue depth, Manual sweep, Migration, Backfill duration, Tuning) are reference material for steady-state operation.

Overview

The search-index pipeline keeps OpenSearch in sync with the relational store using a transactional outbox (search_index_outbox):

  1. Application writes (issue create, MR comment, etc.) enqueue an upsert into the outbox in the same transaction as the data write.
  2. A background scheduler claims pending rows on a poll cadence, pushes them through the appropriate EntityIndexer adapter, and Acks the row on success.
  3. On failure the row is Nacked with an exponential backoff up to a configurable max-attempts cap. After the cap is hit the row is parked with next_attempt_at far in the future -- it stays visible to the dashboard but will not be re-claimed automatically.
  4. A periodic full-sweep goroutine walks every registered EntityIndexer.BulkBackfill on a cron expression so drift between the relational store and OpenSearch is repaired without operator intervention. Re-enqueuing an already-pending row is idempotent thanks to the partial unique index on (entity_type, entity_id) where claimed_at IS NULL.

The dashboard at /admin/search is the single operator surface for this pipeline. It exposes queue depth, last-tick timestamps, last-sweep timestamps, the attempts-exceeded counter, a per-entity breakdown, and the Trigger full sweep button.

Tunables

All tunables live in admin_settings and are validated on write. The scheduler re-reads most of them every poll iteration, so changes take effect within one poll cycle (no restart required). The two exceptions are called out below.

Key Default Bounds Reload cadence
search.poll_interval 5s 1s..60s every tick
search.batch_size 500 1..2000 every tick
search.full_sweep_cron 0 2 * * 0 5-field cron every minute
search.workers 2 1..16 Run start only
search.max_attempts 10 1..50 every tick
search.shutdown_grace 5s 0s..5m Shutdown only

Notes on each:

  • search.poll_interval -- how often the scheduler claims a batch of due rows. Lower values reduce indexing latency but cost one Postgres round-trip per worker per tick. The default 5 s lines up with the dashboard refetch interval.
  • search.batch_size -- how many rows a single tick claims. Larger batches amortise the round-trip but increase the work that a Shutdown grace window must drain. Match this to the OpenSearch bulk batch size to minimise per-row overhead.
  • search.full_sweep_cron -- 5-field cron expression that drives the drift-repair sweep. The default fires every Sunday at 02:00 (server time). Validation rejects malformed expressions on write, so a bad value never reaches the sweep goroutine. An empty string disables the sweep entirely.
  • search.workers -- number of concurrent drainer goroutines. The scheduler uses Postgres SKIP LOCKED so siblings partition work without coordination. Read once at Run start -- changing this requires a restart of the API process. The clamp 1..16 is enforced even if the admin DB stores an out-of-range value.
  • search.max_attempts -- per-row Nack cap. After this many consecutive failures the row is parked (next_attempt_at set to a year out) and surfaced on the dashboard as attempts-exceeded. Operator action then required -- see #stranded-claims below.
  • search.shutdown_grace -- how long Shutdown waits for in-flight tick goroutines before force-releasing their claims. Read once at Shutdown time. A value of 0s skips the wait entirely (claims are released immediately and any in-flight indexer call sees a cancelled context).

Queue depth interpretation

The dashboard headline number is the count of search_index_outbox rows where claimed_at IS NULL and next_attempt_at <= now(). In plain language: rows that are due for indexing right now and not already in flight on a worker.

  • Steady-state (~ 0) -- the pipeline is keeping up. A non-zero reading that returns to 0 within one or two refetches is normal background activity (a burst of comment writes, for example).
  • Sustained > 0 -- the workers are claiming rows but the rate of new writes is outpacing the rate of drains. Investigate:
    1. Is OpenSearch healthy? Slow bulk responses extend each tick.
    2. Is search.workers saturated? If queue depth grows linearly with tenant traffic, raise search.workers (restart required) or search.batch_size (live).
    3. Is one entity dominating the breakdown widget? A misbehaving indexer is easier to diagnose if the per-entity widget shows the imbalance.
  • Sustained > 10000 -- the scheduler is unlikely to catch up with the current settings. Triage as an incident: check API logs for repeated indexer errors, check OpenSearch cluster health, and consider pausing the producer (e.g. by stopping bulk-import jobs) while the queue drains.
  • Spike followed by drain -- expected after a manual sweep (POST /api/v1/admin/search/reindex) or a cron-driven full sweep. The dedupe partial index means a spike never re-indexes already- pending rows.

The attempts-exceeded counter shown alongside queue depth is not included in the queue-depth number -- parked rows have a far-future next_attempt_at and are not "due". Treat it as a separate signal (see #stranded-claims).

Manual sweep trigger

A manual sweep is the operator-initiated equivalent of the cron sweep: it walks every registered EntityIndexer.BulkBackfill, paginates through the canonical store, and enqueues upserts via the idempotent outbox path. Already-pending rows are coalesced; nothing is re-indexed twice.

From the UI

  1. Navigate to /admin/search.
  2. Click Trigger full sweep. The button is disabled while the request is in flight.
  3. A success toast reports Sweep enqueued -- N rows added to the queue (depth: M). The status widget refetches automatically; no manual refresh required.
  4. On 5xx the toast surfaces the verbatim API error body so the operator sees the upstream cause. The button re-enables on settled regardless of outcome.

From the CLI

The vetrix-cli binary exposes the same path for headless / cron contexts:

vetrix-cli search reindex

The subcommand POSTs to /api/v1/admin/search/reindex using the configured admin credentials and prints the JSON response body to stdout. Exit code 0 on 2xx, non-zero on any other status with the API error body written to stderr (per the operating-instructions "surface API error bodies on non-2xx" rule).

To inspect queue stats without enqueuing:

vetrix-cli search status

Prints the same body the dashboard polls (GET /api/v1/admin/search/reindex/status).

Concurrent triggers are safe -- the outbox dedupe coalesces overlap. A failed sweep can be re-run immediately; partial enqueues from the previous attempt are not lost.

Stranded claims

The /admin/search dashboard surfaces two distinct failure modes that both leave rows visible-but-not-progressing. Both are sometimes loosely called "stuck rows" in older docs and Go source comments, but they have different remediation paths and only one has a UI affordance.

Failure mode Row state UI affordance Operator action
Stranded claims claimed_at IS NOT NULL (orphan claim) "Reset stranded claims" button in /admin/search Click button (one click).
Attempts-exceeded attempts >= search.max_attempts None -- no button fixes this case Manual SQL UPDATE (see below).

The dashboard alert badge N rows exceeded retry budget. See runbook: stuck rows corresponds to the attempts-exceeded mode, NOT to stranded claims. The badge's inline secondary note on the page is deliberate -- the adjacent "Reset stranded claims" button does not clear attempts-exceeded rows, and clicking it will not move the counter the badge reports. Diagnose and reset attempts-exceeded rows via the manual SQL steps in the Attempts-exceeded subsection below.

Stranded claims (UI affordance)

A row is in the stranded claims state when claimed_at IS NOT NULL but the worker that took the claim is no longer running -- for example after an API process crash, an OOM kill, a forced restart, or a search-index outage that left in-flight ticks unfinished. The scheduler's SKIP LOCKED claim path will not re-claim these rows because they still appear claimed; without intervention they sit visible on the dashboard's queue-depth widget but make no progress.

The /admin/search dashboard exposes a Reset stranded claims button. It POSTs to /api/v1/admin/search/reset-stuck-rows (endpoint path kept under the legacy name for back-compat) and clears claimed_at on every row whose claim is older than the configured stale-claim threshold. Released rows become claimable again on the next poll tick and the queue-depth widget updates within one poll cycle (5 s default).

The button is the entire remediation -- no SQL is required. The button does NOT clear attempts, does NOT change next_attempt_at, and does NOT touch rows that are merely past their next_attempt_at (those are normal backoff candidates and the scheduler will pick them up on its own).

For how reset-stuck-rows and the claimed_at stranded-claim state fit into the code_search_index table and the search-index outbox internally, see Code-search index operations (architecture) -- section 3, "Clear a parked / stuck outbox and verify drain". This runbook is the authoritative operator procedure for the endpoint itself; the architecture page is for readers reasoning about the underlying data model.

Attempts-exceeded (manual SQL)

A row reaches the attempts-exceeded state when it has been Nacked more times than the configured search.max_attempts cap. The scheduler parks the row by setting next_attempt_at far in the future (roughly one year out) and will not retry it; an operator must investigate the underlying cause and manually reset the row. There is no UI button for this case -- the "Reset stranded claims" button above will not help.

Diagnose

Check the dashboard per-entity breakdown to see which entity type owns the parked row. Then query the search_index_outbox table for rows whose next_attempt_at is more than one year out -- those are the parked rows. Inspect the last_error column for the indexer error string from the final attempt.

SELECT id, entity_type, entity_id, attempts, last_error
  FROM search_index_outbox
 WHERE next_attempt_at > now() + interval '11 months'
 ORDER BY enqueued_at DESC
 LIMIT 50;

Common patterns:

  • OpenSearch mapping conflict -- the entity gained a field the cluster mapping does not allow. Resolve by updating the template (see ../../system-docs/search/index-mappings.md) and rolling the index alias forward, then reset the row. Caveat: the additive PUT _mapping update this implies is a no-op if the target index does not exist at all (see the bootstrap: creating an index vs. updating its mapping section) -- a missing index needs an API restart to get recreated, not a mapping PUT.

  • Entity deleted between enqueue and indexing -- the source row no longer exists. The adapter should now emit a tombstone delete instead; if you see this for an old row the adapter predates that fix. Reset the row to retry; the new path will Ack on the delete.

  • OpenSearch unavailable -- the cluster was down past the retry window. After the cluster is back, reset and the row will succeed on the next claim.

When diagnosing via logs, two failure signatures are worth grepping for directly in the API / worker process logs -- each corresponds to one of the patterns above:

docker compose logs --since=10m <api-service> <worker-service> \
  | grep -E "strict_dynamic_mapping_exception|no indexer registered for entity_type"
  • strict_dynamic_mapping_exception -- the document being indexed carries a field the target index's "dynamic": "strict" mapping does not allow (the OpenSearch mapping-conflict pattern above). Its reappearance after a mapping fix means the additive mapping update did not reach the live cluster -- check whether the index was missing entirely (see the bootstrap caveat above).
  • no indexer registered for entity_type -- a row for an entity type the running registry does not handle was claimed. This should not appear under normal operation; it points at a stale/rearmed row for a decommissioned entity type or a registry wiring bug, not at a transient condition to just reset-and-retry.
Reset

After fixing the underlying cause, clear attempts and next_attempt_at so the scheduler can claim the row again. The dashboard counter will return to 0 within one poll cycle (5 s default).

UPDATE search_index_outbox
   SET attempts = 0,
       next_attempt_at = now(),
       last_error = NULL
 WHERE id = '<row-id>';

To reset all parked rows at once (after, say, repairing the OpenSearch cluster):

UPDATE search_index_outbox
   SET attempts = 0,
       next_attempt_at = now(),
       last_error = NULL
 WHERE next_attempt_at > now() + interval '11 months';

Service unavailable

Triggered by the dashboard banner: Search indexing temporarily unavailable. Retrying...

The dashboard could not reach the status endpoint (GET /api/v1/admin/search/reindex/status). The verbatim API error body is rendered under the headline. React Query keeps polling at the 5 s cadence so transient outages self-heal without operator action.

If the banner persists past a few minutes:

  1. Check the API process logs for searchindex errors.
  2. Verify the Postgres pool has not exhausted its connections -- queue-stats issues a single CTE-style query, so a chronic 5xx is usually an upstream pool / migration issue.
  3. Once recovered the banner clears automatically when the next poll succeeds.

Migration runbook

The pipeline ships behind two migrations:

  • 000159_search_index_outbox -- creates the search_index_outbox table and the partial idx_outbox_claimable / idx_outbox_entity indexes.
  • 000160_search_index_state -- creates the per-worker scheduler state row used to record last-tick / last-sweep timestamps surfaced on the dashboard.

First-run backfill

A fresh deployment starts with an empty outbox -- no historical indexing has happened. Two paths to populate OpenSearch:

  1. Trigger a manual sweep (recommended for small / medium installs): click Trigger full sweep on /admin/search once the API process is up and the migrations have applied. The scheduler enqueues every registered entity, deduped against the unique index, and the workers drain at the configured rate. Watch queue depth on the dashboard until it returns to ~ 0.

  2. Wait for the cron sweep (set-and-forget for large installs): the sweep fires per search.full_sweep_cron. By default this is Sundays at 02:00. To accelerate the first sweep change the cron to a near-future minute (e.g. 42 14 * * *) and revert after the sweep completes.

Either path is idempotent. Re-running the manual sweep after the cron fires is safe (no duplicate work). The pipeline is designed so the operator never has to reason about "is this the first run?" -- the outbox + dedupe handles both cases.

Rollback

If 000160_search_index_state or 000159_search_index_outbox needs to come down, run the corresponding *.down.sql via the migrate binary:

vetrix-migrate down 1

Bring them down in reverse order (160, then 159). The scheduler will refuse to start with the outbox table absent; comment out the scheduler wiring in cmd/server/main.go if a long rollback window is needed.

Expected backfill duration

Backfill is the one-time pass that walks the relational source of truth for an entity type, enqueues every live row into search_index_outbox, and lets the scheduler drain those rows through the registered EntityIndexer. The pipeline is rate-limited at three points:

  • Adapter BulkBackfill reads (Postgres scan, page size = 500 rows per call for every adapter except repos which uses 1000; bounded above by searchindex.BulkBackfillPageMax = 1000).
  • Scheduler claim (search.batch_size, default 500) per search.poll_interval (default 5 s) per worker.
  • OpenSearch Bulk write (opensearch.BulkBatchSize = 500 documents per HTTP _bulk request).

At default settings -- search.workers=2, search.batch_size=500, search.poll_interval=5s -- a single worker can process at most one batch of 500 outbox rows per poll tick, i.e. ~100 rows/s. Two workers double that to ~200 rows/s. In practice the steady-state ceiling is lower because each row also triggers an adapter Index callback that runs its own per-row Postgres SELECT to hydrate the document body before the OpenSearch Bulk write.

The numbers below are defensible upper-bound estimates derived from the configured limits plus the documented OpenSearch single-node Bulk-API throughput envelope (~1k--2k docs/s sustained on commodity hardware with default JVM heap). They should be treated as sanity-check targets rather than measured production values. The bench/ rig does not currently include a backfill scenario; once one lands the table should be replaced with measured rows/s.

Entity Effective rate (default cfg) 10 k rows 100 k rows 1 M rows
Issues ~150 rows/s ~1 min ~12 min ~2 h
MRs ~150 rows/s ~1 min ~12 min ~2 h
Comments ~200 rows/s ~1 min ~9 min ~1 h 25 m
Repos ~120 rows/s ~1.5 min ~14 min ~2 h 20 m
Pages ~150 rows/s ~1 min ~12 min ~2 h
Code blobs ~60 docs/s ~3 min ~28 min ~4 h 45 m

Each entity row in the table assumes the scheduler is otherwise idle (no concurrent live writes competing for the outbox claim). When live traffic is active, subtract its sustained enqueue rate from the effective backfill rate before projecting -- the scheduler is FIFO across both sources and does not give backfill rows any priority.

Code-blob throughput is materially lower because the document body is larger (file content + tokenised symbol set) than a typical issue or MR row. That maps to a higher per-Bulk-request payload, which in turn hits the OpenSearch HTTP-body and JVM-heap limits sooner. Repos is slightly slower than the issue/MR row because the repos adapter hydrates ACL state during Index.

Bottleneck per entity type

Entity Dominant bottleneck
Issues OpenSearch Bulk write (small rows, fast DB scan).
MRs OpenSearch Bulk write (same shape as issues).
Comments OpenSearch Bulk write (very small payload, scan is cheap).
Repos Adapter Index ACL hydration round-trips (Postgres).
Pages OpenSearch Bulk write.
Code blobs OpenSearch Bulk-request payload size (large doc bodies).

Network and indexer-process CPU are not currently bottlenecks at default settings -- two worker goroutines on a single API instance sit well under one CPU core and well under typical 1 GbE link saturation (a 500-doc Bulk request for issue rows is on the order of hundreds of kilobytes; for code blobs it can reach a few megabytes but is still bandwidth-trivial at single-host scale).

Tuning guidance

Tune one knob at a time. After each change, watch the dashboard queue-depth widget for one full poll cycle (5 s) to confirm the queue is draining at the new rate without stuck-row growth.

  • Queue is steady or shrinking, latency is acceptable -- leave settings at defaults. The pipeline is doing its job.

  • Queue is growing during a known-bounded backfill (one-time cutover, large mirror import) -- bump search.workers first. Each additional worker adds ~100 rows/s of headroom. Capped at MaxWorkers = 16. Restart the API process for the change to take effect; search.workers is read once at scheduler start.

  • Queue is growing under steady-state live traffic, not a one-shot backfill -- this is a sign the indexer cannot keep up with write volume. Adding workers is a band-aid; root-cause is usually OpenSearch cluster capacity (heap pressure, slow indexing on a hot shard). Check OpenSearch cluster health before raising search.workers past 4.

  • CPU on the API process is the bottleneck (queue is draining fast but the rest of the API is sluggish) -- lower search.workers to 1 or shift indexing to a dedicated API instance with the rest of the stack scaled down.

  • search.batch_size knob -- raising past the 500 default only helps if the OpenSearch Bulk write is not the bottleneck (i.e. the entity is in the issues/MRs/comments band, not code). Larger batches mean fewer claim round-trips but a longer worst- case stall if a row in the batch parks. Cap at 1000 (BulkBackfillPageMax). Going below 100 is not recommended; the per-tick fixed cost (claim + Ack/Nack) starts to dominate.

  • search.poll_interval knob -- the default 5 s is rarely the binding constraint. Lowering to 1 s converts more wall-clock into outbox SELECT load on Postgres without raising effective throughput when batches are full. Raise above 30 s only if the outbox SELECT itself is showing up in slow-query logs.

For a one-shot tenant migration where backfill duration matters, the recommended fast-path is search.workers=8 for the duration of the migration, then revert to the default. This roughly quadruples the rate for issue/MR/comment/page entity types and cuts code-blob backfill in half (the OpenSearch payload-size ceiling becomes the next bottleneck above 4 workers for code).