Vetrix Docs

OpenSearch — admin recovery runbook

The admin dashboard's OpenSearch card renders a link to this runbook whenever the probe surfaces a structured cause code. Each section below maps a cause code to the diagnostic path. Use the dev stack (mydev_opensearch at 172.18.16.46:9200) for all reproductions; the production cluster is operator-only.

Cause codes

Code Card headline
startup_failed OpenSearch — startup fallback (configured but never connected)
dns_failure OpenSearch — DNS lookup failed
auth_failure OpenSearch — authentication rejected
tls_failure OpenSearch — TLS / certificate error
network OpenSearch — network unreachable
timeout OpenSearch — probe timed out
cluster_red Cluster status: red
version_mismatch OpenSearch — version mismatch
unknown OpenSearch — unreachable (verbatim error in the small-print field)

startup_failed

The API booted with search.backend = opensearch and a URL set, but the boot-time HealthCheck against cfg.Search.URL failed. The process is running with osClient = nil and every search read/write is going through the Postgres pg_trgm fallback. Code search is degraded; issue search returns the legacy results path.

The probe wired in internal/opensearch/admin_health.go attempts a late-bind reconnect on every /admin/health call. If the cluster comes back the next probe clears StartupFailed and the card flips to healthy without an API restart.

Diagnostic checklist:

  1. Confirm intent vs reality:
    cat $CONFIG_DIR/app.toml | grep -A2 '\[search\]'
curl -m 3 -sv $OPENSEARCH_URL/_cluster/health
  2. If the URL is reachable from the operator host but not from the API container, the issue is network policy / DNS scoping inside the container — not OpenSearch itself. Resolve from inside the container:
    docker exec mydev_vetrix wget -qO- "$OPENSEARCH_URL/_cluster/health"
  3. If the URL is unreachable everywhere: bring the cluster up. The API will self-heal on the next 30 s probe — no rolling restart.

dns_failure

Hostname in cfg.Search.URL cannot be resolved. Common causes:

  • Compose service name (mydev_opensearch) used outside the compose network — switch to the static IP or add the host to extra_hosts.
  • Recently-renamed cluster — the URL still points at the old name.

auth_failure

OpenSearch returned 401/403. The probe wraps the bare _cluster/health call so credentials live in the URL or Authorization header constructed in the SDK. Rotate the password via admin settings; never log the URL with embedded credentials (the probe explicitly avoids this).

tls_failure

Certificate not trusted by the API container. If the cluster runs HTTPS with a self-signed CA, mount the CA into the API container's trust store; do not flip InsecureSkipVerify permanently — surface a hardening follow-up ticket and patch via the trust store.

network

connection refused, no route to host, connection reset, or similar. Cluster process is up but something between API and cluster is not. Check Docker network policy, firewall, port exposure.

timeout

Probe context expired before the cluster answered. Probe budget is admin.healthProbeBudget = 150 ms inside CollectHealth (overrides the 1 s standalone probe budget). A wedged cluster, full GC pause, or disk-full red state will all surface as timeout. Cross-check /_cluster/health?timeout=2s from the operator host before assuming the cluster is dead — a single timeout window can be transient.

cluster_red

Cluster answered but reports status: red. Walk through:

  1. Disk pressure: _cat/allocation. Free up data nodes if any are at flood_stage (default 95 %).
  2. Shard allocation: _cluster/allocation/explain. Common causes are missing replicas after a node crash; bring the node back or reduce the replica count temporarily.
  3. Index corruption: rare, but _cat/indices?v shows red indexes alongside the cluster. Recover via snapshot restore.

version_mismatch

Server reports an OpenSearch version the SDK does not support. Pin the cluster to a supported major (>= 2.x) and retry the probe.

Reachable cluster, no code

If the card is healthy (no code field), the probe round-trip succeeded and cluster_status is green or yellow. No action required.

Push alerting

The structured-code admin card and this runbook cover the operator-watching-the-dashboard path. For on-call to be paged without watching the dashboard, the API exposes two Prometheus-text-exposition gauges on the /metrics scrape endpoint:

Series Type Meaning
opensearch_admin_health_status gauge 1 when the most recent admin probe round-trip succeeded, 0 when it failed. A red-but-reachable cluster reports 1 here.
opensearch_admin_health_consecutive_failures gauge Run-length of consecutive failed probes since the last success. Resets to 0 on the next reachable probe.

Both series are labelled instance="<hostname>" so a multi-replica deployment can scope the alert per-API-pod. The override metrics.SetOpenSearchAdminInstance is available for deployments where the Kubernetes pod name is the meaningful identifier rather than the container's internal hostname.

The consecutive_failures series is rendered as a gauge — not a counter — because it MUST decrement on success. Counter semantics break rate() and increase() if the value ever drops; the "consecutive" framing IS a gauge: the current run-length. Alert rules threshold on >= N directly without rate().

Probe cadence

The admin dashboard polls /api/v1/admin/health every 30 s (fixed cadence). Each poll runs the OpenSearch probe and updates both gauges. Two consecutive failed polls equals 60 s of unreachability — the threshold below.

Alert rule

Append to your Prometheus / alertmanager rule files:

groups:
  - name: vetrix-opensearch-admin
    rules:
      - alert: OpenSearchAdminProbeDisconnected
        expr: opensearch_admin_health_consecutive_failures >= 2
        for: 60s
        labels:
          severity: page
          service: vetrix-opensearch
        annotations:
          summary: "OpenSearch admin probe has failed {{ $value }} consecutive times on {{ $labels.instance }}"
          description: |
            The Vetrix admin probe to OpenSearch has failed for two or
            more consecutive 30s polls on instance {{ $labels.instance }}.
            Search reads are degraded (Postgres pg_trgm fallback). See
            the operator runbook for the structured cause-code matrix.
          runbook_url: "/runbook-docs/search/opensearch-recovery.md"

The for: 60s clause guards against a single 30 s probe blip caused by a transient network hiccup; the alert fires only when both probes inside the 60 s window failed (run-length stayed >= 2 for the full interval).

Verifying the metric

Reproduce the disconnect locally with the dev stack:

docker compose stop mydev_opensearch
sleep 60
curl -s http://localhost/metrics | grep opensearch_admin_health
docker compose start mydev_opensearch

Expected: after ~60 s the _consecutive_failures series is >= 2 and _status is 0. After OpenSearch comes back the next probe flips _status to 1 and resets _consecutive_failures to 0.

If the deployment has no Prometheus scraper

The metric series are still emitted on /metrics regardless of whether anyone scrapes them — alertmanager wiring is operator-side. If your deployment ships without a Prometheus pipeline, point any HTTP-scrape-capable monitor (Datadog, Vector, plain curl cron) at the endpoint and threshold on opensearch_admin_health_consecutive_failures. The series name is stable; the value semantics match across scrapers.

Cross-references

  • Health probe code: internal/admin/health_opensearch.go
  • Adapter (late-bind retry): internal/opensearch/admin_health.go
  • Frontend card: web/src/components/admin/OpenSearchHealthCard.tsx
  • Boot-time wiring: cmd/server/main.go (search for osIntent)
  • Push-alert metrics: internal/metrics/opensearch_health.go
  • Probe-side emission: internal/admin/health_opensearch.go (search for RecordOpenSearchAdminHealth)