OpenSearch index mappings and query patterns

This doc is the canonical reference the indexer and the query layer build against. Every mapping here carries the common ACL field block defined in acl-schema.md. JSON blocks are reproduced in-line so operators rebuilding indexes by hand can copy them verbatim.

1. Custom analyzers

Three analyzers are defined across the Vetrix indexes. They are not in-built on OpenSearch so they must be part of the settings.analysis.* block at index-create time.

`code_analyzer`

Splits identifiers at case boundaries and connective punctuation so that handleHTTPRequest indexes as handle, http, request (plus the full handleHTTPRequest token for exact hits).

{
  "analyzer": {
    "code_analyzer": {
      "type": "custom",
      "tokenizer": "standard",
      "filter": ["lowercase", "code_word_delimiter"]
    }
  },
  "filter": {
    "code_word_delimiter": {
      "type": "word_delimiter_graph",
      "generate_word_parts": true,
      "generate_number_parts": true,
      "catenate_words": false,
      "catenate_numbers": false,
      "catenate_all": false,
      "split_on_case_change": true,
      "preserve_original": true,
      "split_on_numerics": false,
      "stem_english_possessive": false
    }
  }
}

Sample tokenisation — input httpHandleFunc_v2:

httpHandleFunc_v2 → [httphandlefunc_v2, http, handle, func, v2]

`path_analyzer`

Emits every parent prefix of a file path so queries like internal/api match internal/api/repositories.go.

{
  "analyzer": {
    "path_analyzer": {
      "type": "custom",
      "tokenizer": "path_hierarchy_tokenizer"
    }
  },
  "tokenizer": {
    "path_hierarchy_tokenizer": {
      "type": "path_hierarchy",
      "delimiter": "/"
    }
  }
}

Sample tokenisation — input internal/api/repositories.go:

internal/api/repositories.go → [internal, internal/api, internal/api/repositories.go]

2. Index: `vetrix-code`

{
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 0,
    "analysis": { "analyzer": {"code_analyzer": "…", "path_analyzer": "…"} }
  },
  "mappings": {
    "dynamic": "strict",
    "properties": {
      "repo_id":               { "type": "keyword" },
      "is_private":            { "type": "boolean" },
      "visibility":            { "type": "keyword" },
      "owner_id":              { "type": "keyword" },
      "collaborator_user_ids": { "type": "keyword" },
      "group_ids":             { "type": "keyword" },
      "content":               { "type": "text", "analyzer": "code_analyzer" },
      "file_path":             {
        "type": "text",
        "analyzer": "path_analyzer",
        "fields": { "raw": { "type": "keyword" } }
      },
      "lang":                  { "type": "keyword" },
      "ref":                   { "type": "keyword" },
      "blob_sha":              { "type": "keyword" }
    }
  }
}

"dynamic": "strict" rejects any document with unrecognised fields, catching stray ACL leaks at ingest.

3. Index: `vetrix-symbols`

{
  "mappings": {
    "dynamic": "strict",
    "properties": {
      "repo_id":               { "type": "keyword" },
      "is_private":            { "type": "boolean" },
      "visibility":            { "type": "keyword" },
      "owner_id":              { "type": "keyword" },
      "collaborator_user_ids": { "type": "keyword" },
      "group_ids":             { "type": "keyword" },
      "symbol_name":           {
        "type": "text",
        "analyzer": "standard",
        "fields": {
          "raw":       { "type": "keyword" },
          "edge_ngram":{ "type": "text", "analyzer": "autocomplete_analyzer" }
        }
      },
      "kind":                  { "type": "keyword" },
      "file_path":             { "type": "keyword" },
      "line_number":           { "type": "integer" }
    }
  }
}

autocomplete_analyzer is an edge_ngram of size 2–12 built from the standard tokenizer; it ships in the same settings.analysis block as code_analyzer.

4. Index: `vetrix-issues` / `vetrix-mrs`

{
  "mappings": {
    "dynamic": "strict",
    "properties": {
      "repo_id":               { "type": "keyword" },
      "is_private":            { "type": "boolean" },
      "visibility":            { "type": "keyword" },
      "owner_id":              { "type": "keyword" },
      "collaborator_user_ids": { "type": "keyword" },
      "group_ids":             { "type": "keyword" },
      "title":                 { "type": "text", "analyzer": "english", "boost": 3.0 },
      "body":                  { "type": "text", "analyzer": "english" },
      "number":                { "type": "integer" },
      "state":                 { "type": "keyword" },
      "author_id":             { "type": "keyword" },
      "assignee_ids":          { "type": "keyword" },
      "labels":                { "type": "keyword" },
      "components":            { "type": "keyword" },
      "priority":              { "type": "keyword" },
      "updated_at":            { "type": "date" }
    }
  }
}

vetrix-mrs adds source_branch + target_branch + merge_state (all keyword).

updated_at is the canonical date anchor used by the ?sort=recent site-search ordering. The indexer wires it from the Postgres updated_at column on issues / merge_requests. Documents that predate the field sort to the bottom under the missing:"_last" hedge — operators upgrading an existing cluster should run scripts/opensearch/backfill-updated-at.sh once the new mapping is applied.

5. Index: `vetrix-pages`

{
  "mappings": {
    "dynamic": "strict",
    "properties": {
      "repo_id":                     { "type": "keyword" },
      "is_private":                  { "type": "boolean" },
      "visibility":                  { "type": "keyword" },
      "owner_id":                    { "type": "keyword" },
      "collaborator_user_ids":       { "type": "keyword" },
      "group_ids":                   { "type": "keyword" },
      "page_permission_user_ids":    { "type": "keyword" },
      "page_permission_group_ids":   { "type": "keyword" },
      "share_token":                 { "type": "keyword" },
      "title":                       { "type": "text", "analyzer": "english", "boost": 3.0 },
      "body":                        { "type": "text", "analyzer": "english" },
      "published_title":             { "type": "text", "analyzer": "english", "boost": 3.0 },
      "published_body":              { "type": "text", "analyzer": "english" },
      "space_id":                    { "type": "keyword" },
      "updated_at":                  { "type": "date" }
    }
  }
}

updated_at on vetrix-pages is fed from pages.updated_at by the page-indexer hook in internal/pages/indexer.go. Same ?sort=recent semantics as vetrix-issues / vetrix-mrs.

6. Shard / replica strategy

Environment	Shards per index	Replicas	Rationale
Dev	1	0	Single-node cluster; no replication possible. Keeps mapping reload cheap.
Prod small	3	1	3-node cluster, every doc survives a single-node loss.
Prod large	6	1	Split wide so reindex cascades finish within SLO.

The shard count is settable via app.toml under [search.opensearch] per-index:

[search.opensearch]
url = "https://opensearch.internal:9200"
username = "${OPENSEARCH_USER}"
password = "${OPENSEARCH_PASSWORD}"

[search.opensearch.indexes.vetrix-code]
shards = 6
replicas = 1

Defaults when the section is absent: dev table (1 shard, 0 replicas).

7. Query DSL by endpoint

Every query is wrapped in a bool { filter: [ … AuthFilter(claims), … entity-specific] } envelope. Relevance scoring runs in must — filters never contribute to _score so the ACL block cannot accidentally re-rank a hit.

`GET /api/v1/search/code` — code search

{
  "query": {
    "bool": {
      "must": [
        {
          "match": {
            "content": {
              "query": "<q>",
              "operator": "and"
            }
          }
        }
      ],
      "filter": [
        { "terms": { "repo_id": ["<repo-uuid>"] } },
        { "term":  { "lang":    "<lang>"         } },
        /* AuthFilter appends its own terms/bool here */
      ]
    }
  },
  "highlight": {
    "fields": { "content": {} }
  },
  "size": 20
}

Expected hit shape:

{
  "hits": {
    "hits": [
      {
        "_id": "<repo-uuid>:<file_path>:<ref>",
        "_score": 8.7,
        "_source": { "file_path": "…", "lang": "go", "…": "…" },
        "highlight": { "content": ["… <em>handler</em> …"] }
      }
    ]
  }
}

`GET /api/v1/repos/:owner/:repo/symbols` — symbol search

{
  "query": {
    "bool": {
      "must": [
        {
          "dis_max": {
            "queries": [
              { "prefix": { "symbol_name": "<q>" } },
              { "fuzzy":  { "symbol_name": { "value": "<q>", "fuzziness": "AUTO" } } }
            ]
          }
        }
      ],
      "filter": [
        { "term": { "repo_id": "<repo-uuid>" } },
        { "term": { "kind":    "<kind>"      } }
      ]
    }
  }
}

`GET /api/v1/repos/:owner/:repo/symbols/:sym/definition` — jump-to-definition

{
  "query": {
    "bool": {
      "filter": [
        { "term": { "symbol_name.raw": "<sym>" } },
        { "term": { "repo_id":         "<repo-uuid>" } },
        { "terms":{ "kind": ["function", "type", "const", "method"] } }
      ]
    }
  },
  "size": 5
}

`GET /api/v1/repos/:owner/:repo/issues/search` — issue search

{
  "query": {
    "bool": {
      "must": [
        {
          "multi_match": {
            "query":   "<q>",
            "fields":  ["title^3", "body"],
            "operator":"and"
          }
        }
      ],
      "filter": [
        { "term": { "repo_id": "<repo-uuid>" } },
        { "term": { "state":   "<open|closed>" } }
      ]
    }
  },
  "aggs": {
    "by_label":     { "terms": { "field": "labels"      } },
    "by_assignee":  { "terms": { "field": "assignee_ids"} },
    "by_component": { "terms": { "field": "components"  } }
  },
  "highlight": {
    "fields": { "title": {}, "body": {} }
  },
  "size": 20
}

8. Bootstrap: creating an index vs. updating its mapping

Two separate boot-time steps own index lifecycle (internal/opensearch/bootstrap.go), and they are not interchangeable:

EnsureIndices — create-only. For each index in DefaultIndexSpecs() it HEADs the index; a 404 triggers (Client).EnsureIndex, which creates the index with its full mapping (HEAD-then-PUT). A 200 is a no-op — an existing index's mapping is never rewritten by this path.
EnsureMappings — additive-only. It calls (Client).EnsureMapping per index, which issues an additive PUT _mapping. Critically, EnsureMapping HEADs the index first and returns nil (no-op) if the index does not exist — it will never create a missing index.

The consequence: if an index is missing entirely (a HEAD check on it returns 404), the additive PUT _mapping path used to roll a mapping change forward silently no-ops — it cannot recreate a missing index. The only way to get a missing index back is to restart the API process so EnsureIndices runs its HEAD-then-create path again on boot; then re-check with a HEAD request that it now returns 200. Do not try to fix a missing index with the _mapping PUT path.

9. Migration from the current ILIKE / tsvector approach

The current implementation lives in internal/search/store.go (code search via ILIKE '%q%' against code_search_index.content), internal/issues/search.go (issue search via pg_trgm against title + body), and the page search index page_search_index.tsv (page search via plainto_tsquery). The swap to OpenSearch follows a feature-flag pattern:

Dual-write phase. Every write path (indexer hooks in cicd/pipeline.go, issue hooks, page triggers) writes to both Postgres and the vetrix-* indexes. Postgres remains the authoritative store.
Shadow-query phase. The handlers run the Postgres query and the OpenSearch query in parallel; the frontend continues to see the Postgres response while permission-matrix tests assert that the OpenSearch response is a superset (or equal) with the same ACL visibility.
Cutover. The handler factory in internal/api/search.go reads settings.search_backend (new runtime setting with key search.backend, enum postgres|opensearch, default postgres) and routes accordingly. Admins flip it per-environment without a restart.
Cleanup. Once every environment has run on opensearch for 14 days, the Postgres-side trigger + migration for code_search_index / page_search_index + the internal/search/store.go ILIKE path are removed in a dedicated follow-up.