ACL-aware OpenSearch index schema

This document is the authoritative mapping schema for the Vetrix OpenSearch migration. Every Vetrix index carries a common ACL field block so that a single query-time filter (search.AuthFilter(claims)) can enforce visibility across every entity type. The per-entity field blocks are additive: a code document carries the common ACL fields plus the vetrix-code body fields; a page document carries the common ACL fields plus the page-specific permission arrays.

Common ACL field block (present on every index)

Field	Type	Required	Notes
`repo_id`	keyword	yes	Always present, even on cross-repo entities like newsfeed posts (`""` sentinel).
`is_private`	boolean	yes	Legacy. True when the owning repo is private. Retained for back-compat with older documents that have not yet been backfilled with `visibility`.
`visibility`	keyword	yes	3-tier enum `public` / `internal` / `private`. The search authfilter clauses key on this field; `is_private` remains for back-compat only.
`owner_id`	keyword	yes	Repo / space owner user UUID.
`collaborator_user_ids`	keyword (array)	yes	Flat union of direct collaborators and inherited-via-groups users.
`group_ids`	keyword (array)	yes	Groups the entity is shared with (empty array allowed; null is NOT).
`share_token`	keyword (nullable)	page only	Only populated on the `vetrix-pages` index when a share-link exists.

Mapping rule: all five common fields use explicit "type" declarations and "null_value" is forbidden on the array fields (an empty array [] is the only legal zero-case). Documents missing any required ACL field must be rejected at ingest by a DynamicTemplatesGuard rule — see index-mappings.md §3.

Per-entity additions

`vetrix-code`

Additive fields on top of the common block:

Field	Type	Notes
`content`	text (`code_analyzer`)	File body; word_delimiter_graph splits camelCase / snake_case.
`file_path`	text (`path_analyzer`) + keyword	Dual-indexed: prefix search on the `text`, exact filter on raw.
`lang`	keyword	Detected language (`go`, `ts`, `py`, …).
`ref`	keyword	Branch or tag pointer (`HEAD`, `refs/heads/main`, tags…).
`blob_sha`	keyword	Content-addressable dedupe key.

`vetrix-symbols`

Field	Type	Notes
`symbol_name`	text (`standard` + edge-ngram)	Prefix + fuzzy. `.raw` multi-field is `keyword`.
`kind`	keyword	`function`, `type`, `const`, …
`file_path`	keyword	Exact — matches the code doc's `file_path`.
`line_number`	integer	1-based line.

`vetrix-issues`

Field	Type	Notes
`title`	text (`english`) — boost 3	High relevance weighting on title hits.
`body`	text (`english`)	Markdown body, stripped to plain text by the ingest pipeline.
`number`	integer	Repo-scoped issue number.
`state`	keyword	`open` / `closed`.
`author_id`	keyword	Denormalised for author filter facet.
`assignee_ids`	keyword (array)	For the assignee facet.
`labels`	keyword (array)	Label names.
`components`	keyword (array)	Component names.
`priority`	keyword	`critical`, `high`, `medium`, `low`.

`vetrix-mrs`

Same shape as vetrix-issues with an added source_branch + target_branch (keyword each) and merge_state (keyword — draft, open, merged, closed). No new ACL fields.

`vetrix-comments`

Shared index for both issue comments and MR review comments. Carries parent_type (keyword — issue / mr) and parent_id (keyword) plus the common ACL block inherited from the parent entity (denormalised at write time — the reindex cascade keeps it in sync).

`vetrix-pages`

Additive fields on top of the common block:

Field	Type	Notes
`title`	text (`english`) — boost 3	Same weighting as issues title.
`body`	text (`english`)	Rendered markdown, plain-text stripped.
`published_title`	text (`english`) — boost 3
`published_body`	text (`english`)
`space_id`	keyword
`page_permission_user_ids`	keyword (array)	Explicit page-scoped ACL — empty array means "inherit from space".
`page_permission_group_ids`	keyword (array)
`share_token`	keyword (nullable)	When present, the share-link auth path uses it as a short-circuit grant.

Rejection rules at ingest

The OpenSearch ingest pipeline (vetrix_acl_required) must reject a document when:

Any required common ACL field is absent (all six: repo_id, is_private, visibility, owner_id, collaborator_user_ids, group_ids).
collaborator_user_ids or group_ids is JSON null (must be an empty array).
A page document lacks page_permission_user_ids or page_permission_group_ids (both are required to have an explicit empty array, even when inheritance applies).
A code or symbol document lacks file_path / lang / ref.

The Go ingest layer mirrors these rejections in internal/search/indexer.go so a broken ingest fails loud before the OpenSearch round-trip.

Null-value stance

"null_value" is disallowed on every ACL array field. Rationale: OpenSearch's keyword null_value substitutes a sentinel token at query time, which would silently satisfy a terms filter — a ghost-grant the reviewer would never see on the document. An empty array is unambiguous: the terms clause simply cannot match.

Source-of-truth table per field

Field	Source table	Query shape
`repo_id`	`repositories.id`	direct projection
`is_private`	`repositories.is_private`	direct projection
`visibility`	`repositories.visibility`	`COALESCE(visibility, CASE WHEN is_private THEN 'private' ELSE 'public' END)` to backfill rows that predate the `visibility` column
`owner_id`	`repositories.owner_id`	direct projection
`collaborator_user_ids`	`repo_collaborators.user_id`	`SELECT user_id FROM repo_collaborators WHERE repo_id = $1`
`group_ids`	`group_repo_grants.group_id`	`SELECT group_id FROM group_repo_grants WHERE repo_id = $1`
`page_permission_user_ids`	`page_permissions.user_id`	`SELECT user_id FROM page_permissions WHERE page_id = $1 AND user_id IS NOT NULL`
`page_permission_group_ids`	`page_permissions.group_id`	`SELECT group_id FROM page_permissions WHERE page_id = $1 AND group_id IS NOT NULL`
`share_token`	`page_share_links.token` (most recent non-expired)	`SELECT token FROM page_share_links WHERE page_id = $1 AND (expires_at IS NULL OR expires_at > NOW()) ORDER BY created_at DESC LIMIT 1`

Group-membership resolution for the query-time AuthFilter uses group_members.user_id joined against the denormalised group_ids — the index stores which groups can see a repo, and the filter fans out to "which groups does the caller belong to" at query time. This keeps the indexed document size small (groups, not individual group members) while still supporting large groups.

Denormalization caps + overflow strategy

Indexed arrays have a hard upper bound so a pathologically-collaborated repo cannot inflate a single document beyond a sensible size:

Field	Cap (default)	Overflow behaviour
`collaborator_user_ids`	4 096	Set a boolean `collab_overflow=true` on the doc; `AuthFilter` forces a Postgres-side `repo_collaborators` lookup for docs where `collab_overflow=true`.
`group_ids`	256	Same pattern — `group_overflow=true` gates a Postgres `group_repo_grants` fallback.
`page_permission_user_ids`	1 024	Per-page ACL — cap is per-page-document and exists to guard against a runaway `page_permissions` row count. Overflow uses the same `page_user_overflow` boolean.
`page_permission_group_ids`	256	Same pattern with `page_group_overflow`.

Defaults are operator-settable via [search.opensearch] in app.toml (e.g. collab_list_cap = 8192). The overflow booleans are currently reserved — the indexers do not yet emit them since no repo in the known fleet is near the cap; the AuthFilter builder will honour them once the indexers start tagging overflowed docs.

Staleness expectations (SLA per field)

Field / event class	Expected propagation
New collaborator added → visibility of existing search results	≤ 2 s under typical load; fail-open (caller sees results slightly late)
Collaborator removed → exclusion of results	≤ 2 s; fail-closed against the primary Postgres ACL query in `SearchHandler` so a delayed sync cannot leak
Group membership change → repos via group grants	≤ 5 s; fan-out is O(affected repos)
Group grant add/remove on a repo	≤ 2 s
Repo `is_private` toggle	≤ 2 s; intentionally-narrow stale window (see "stale-ACL window SLA" below)
New page_permissions row	≤ 2 s
Share-link revoked (`page_share_links.revoked_at`)	≤ 2 s — the cascade deletes the stale `share_token` field
Repo deleted	best-effort — `DeleteByRepo` removes every document; out-of-band any stale hit is surfaced by the permission-matrix test

All above deltas are eventually-consistent. Query-time enforcement in SearchHandler is the authoritative gate for security-critical revocations — a delayed OpenSearch sync never leaks because the Postgres allow-list is consulted on every request.

Indexer surface per index

Each indexer below MUST populate every required common ACL field and, for page documents, the three addenda.

Index	Indexer surface
vetrix-code	`NewCodeDoc(ACLFields, filePath, ref, lang, blobSHA, content)`
vetrix-symbols	`NewSymbolDoc(ACLFields, symbolName, kind, filePath, lineNumber)`
vetrix-issues	`NewIssueDoc(IssueDocInput{ACL: ACLFields, …})`
vetrix-mrs + vetrix-comments	reuses `ACLFields` — schema identical to issues + extra `source_branch`/`target_branch` on MRs
vetrix-pages	adds `page_permission_user_ids`, `page_permission_group_ids`, `share_token`

Every indexer builder in internal/opensearch/ forces ACL array fields to []string{} rather than nil so the null-forbidden rule above cannot trip even on nil input.

Design decision: groups, split across doc-side + query-side

Group-based access decomposes into two independent mutation classes that are handled asymmetrically to minimise reindex churn:

Event	Doc-side update?	Query-side behaviour
`group_members` INSERT/DELETE (membership change)	No. Docs are unchanged.	Caller's group IDs resolved per-request from `group_members` (see `GroupsResolver` + request-context cache via `WithCachedGroups`).
`group_repo_grants` INSERT/UPDATE/DELETE	Yes. `group_ids` on every doc for the repo refreshed via `_update_by_query`.	`AuthFilter` emits `terms{group_ids: caller.groups}` which evaluates against the refreshed doc-side set.

Why this asymmetry:

Membership churn is high — users join and leave groups frequently. Refreshing every doc for every repo the group has a grant on, for every member change, would generate unbounded reindex load.
Grant churn is low — operators rarely grant/revoke group access to a repo. Refreshing one repo's docs on each mutation is cheap.

The caller's group IDs are therefore looked up at query time (acl.Store.GroupsForUser hits group_members with one small query). The cache is request-scoped via authfilter.WithCachedGroups(ctx, groups) — no long-lived cache is used because membership can change between requests and a stale long-cache would re-introduce the bug this design is explicitly avoiding.

Cascade trigger points (all defined in internal/acl/cascade.go):

CascadeAfterGroupRepoGrantChange(ctx, repoID, syncer) — fires after UpsertGroupRepoGrant / DeleteGroupRepoGrant. Single repo refresh.
CascadeGroupMembershipFanOut(ctx, groupID, syncer) — only runs when the caller explicitly wants the fan-out. The design above means this is rarely the right call; it exists for admin-run reindex workflows where an operator wants to force a doc-side refresh (e.g. after bulk backfilling group_members from an LDAP sync).

Staleness expectations: grant changes ≤ 2 s; membership changes are effectively instantaneous — they take effect on the caller's very next query because the resolver pulls fresh group_members rows.

Page lifecycle hooks (write-side index)

Page create and update events are dual-written: every successful pages.Store.CreatePage / UpdatePage / PublishPage / UnpublishPage / RevertToVersion call invokes pages.Indexer.IndexPage(ctx, *Page) after the Postgres commit. Production wires pages.OpenSearchIndexer (internal/pages/indexer.go) which:

Looks up the page's repo_id (skips orphan pages with NULL repo_id).
Resolves the repo ACL block via acl.Store.ResolveRepoACL(ctx, repoID).
Pulls per-page ACL principals from page_permissions.
Pulls the most-recent non-expired token from page_share_links.
Builds opensearch.PageDoc and calls Indexer.IndexPageDocs([...]) against the deterministic PageDocID(pageID) so the bulk action acts as an upsert.

Page delete events are mirrored on the same interface: pages.Store.DeletePage invokes pages.Indexer.RemovePage(ctx, pageID) after the Postgres DELETE FROM pages commits. The OpenSearch implementation routes through the existing opensearch.Indexer.DeletePage helper, which uses _delete_by_query against the deterministic PageDocID. That call is naturally idempotent — a missing document yields a successful empty result rather than a 404 — so a second DeletePage (or a delete event arriving for a page that was never indexed) never returns an error.

Eventually-consistent semantics: both hooks are fire-and-forget. An OpenSearch outage logs and swallows the error rather than failing the page-create or page-delete transaction; the read path wraps queries with a graceful Postgres fallback. The query-time gate documented under "Stale-ACL window SLA" remains authoritative — a delayed index write never broadens visibility because SearchHandler consults Postgres on every request.

The query-time gate consumes the indexed page_permission_* arrays + the common ACL block at query time.

Page query-time ACL gate

Every page-search OpenSearch query is constructed by internal/opensearch/pages_search.go::BuildPageSearchQuery, which splices the bool.filter clauses produced by internal/search/authfilter/page.go::BuildOpenSearchPageFilter. The two call paths that exercise it today:

The unified GET /api/v1/search handler — internal/api/site_search.go resolves a *authfilter.PageFilter via authfilter.ResolvePage(ctx, claims, resolver, shareToken) and passes it through opensearch.SiteSearchParams.PageAuthFilter to the _msearch builder, which routes it to the pages sub-query.
Any future pages-only OpenSearch endpoint must reuse BuildPageSearchQuery and provide a resolved PageAuthFilter — the builder enforces the ACL clause on every emitted body.

The Postgres-backed legacy search at GET /api/v1/pages/search carries its own SQL-level permission filter in internal/pages/analytics.go::SearchPages; it is not affected by the OpenSearch page gate because it does not query OpenSearch.

Predicate

A page document satisfies the visibility predicate when at least one of:

is_private = false — the parent repo is public.
owner_id = caller.user_id.
caller.user_id ∈ collaborator_user_ids.
caller.groups ∩ group_ids ≠ ∅.
caller.user_id ∈ page_permission_user_ids — explicit page grant.
caller.groups ∩ page_permission_group_ids ≠ ∅ — group page grant.
share_token = caller.presented_share_token — short-circuit grant.

Anonymous callers (Filter.Anonymous = true) only have (1) and (7) applicable; clauses (2)–(6) are not added to the should set since there is no user_id or group-id to bind.

Admin callers (Filter == nil) bypass the predicate entirely — the filter builder returns nil and OpenSearch performs a match-all evaluation. This is the same short-circuit used by every other index's auth filter.

Wire shape

The constructed bool.filter always wraps in:

{ "bool": { "should": [...], "minimum_should_match": 1 } }

minimum_should_match: 1 is the safety latch — at least one membership clause must match, otherwise OpenSearch emits zero hits regardless of any other filter clause (e.g. a hostile space_id parameter cannot widen the candidate set past the auth predicate).

The filter is pushed entirely into OpenSearch — the handler does NOT post-filter results in Go. This keeps total.value accurate, pagination correct, and per-page latency minimal.

Test coverage

Pure clause-shape tests: internal/search/authfilter/page_matrix_test.go and internal/opensearch/pages_search_test.go (hostile-case regressions).
Live-cluster integration tests: internal/opensearch/pages_acl_search_integration_test.go (build tag integration, requires OPENSEARCH_URL). Asserts a private page is absent from a non-member's results, page-level grants override repo ACL, share-token gating works for anonymous callers, public pages are visible to everyone, and admin sees every doc regardless of grants.

Eventual-consistency note

The query-time predicate is evaluated against the indexed copy of the ACL fields. The lifecycle hook keeps those fields fresh on create/update/publish; the delete hook removes the doc when a page is deleted; the cascade entries above keep them fresh on permission/share-link mutations. The read path wraps a Postgres fallback for the OpenSearch-unavailable case (see "Page search graceful fallback" below) — the fallback honours the same predicate via internal/pages/analytics.go::SearchPages, which enforces an equivalent permission filter at the SQL layer.

Page search graceful fallback

Page search must never return 5xx to the client even when OpenSearch is unreachable. The unified GET /api/v1/search handler (internal/api/site_search.go) wraps every OpenSearch call with runSiteSearchWithFallback (internal/api/site_search_fallback.go), which produces a 200 response in every failure mode listed below. There is currently no dedicated pages-only OpenSearch endpoint; the fallback wrapper covers /api/v1/search exclusively. The legacy GET /api/v1/pages/search already runs against Postgres and is unaffected.

Trigger conditions

The fallback fires when the OpenSearch call returns any error — connection refused / network unreachable, request timeout / context.DeadlineExceeded, transport-level 5xx, and the eventually-consistent missing-index 404 (a page write before the index has been bootstrapped emits a 404 on read; treated identically to other transport failures). The OpenSearch-disabled case (svc.OpenSearchClient == nil) routes through the same wrapper so the response shape is uniform across "disabled" and "unavailable" states.

Strategy

The wrapper preserves the pages section of the response by routing through pages.Store.SearchPages, which carries its own SQL-level visibility filter:

public space  OR  space owner  OR  page-level explicit page_permissions row for the viewer

This is the SQL analogue of the OpenSearch should-set predicate documented above; the visible-to-viewer set is therefore the same in both backends. The Postgres path also has the property that it reads the canonical source of truth, so under fallback the pages section is more up-to-date than the indexed path would be — at the cost of dropping the cross-entity ranked sections (code/symbols/issues/mrs are empty under fallback, since no Postgres equivalent exists at this layer).

Response contract

Scenario	Status	`X-Search-Fallback` header	Body
OpenSearch reachable, query succeeds	200	(absent)	Normal `SiteSearchResponse` from `_msearch`
OpenSearch errors AND `pages.Store` wired AND `types` includes `pages`	200	`degraded-pages`	`pages` section populated from Postgres; other sections zero
OpenSearch errors AND `pages.Store` not wired	200	`degraded-empty`	Empty envelope (every section zero)
OpenSearch errors AND `pages.Store` returns its own error	200	`degraded-empty`	Empty envelope
OpenSearch errors AND `types` excludes `pages`	200	`degraded-empty`	Empty envelope (no Postgres analogue for non-pages sections)

The frontend treats the presence of X-Search-Fallback as the only banner trigger — absent header → normal mode, any value → degraded mode.

The pages section under fallback emits hits whose _source JSON carries fallback_source: "postgres" so the frontend can distinguish the source if it needs to render a per-section badge in addition to the banner.

Logging

Fallback events emit a structured warning via svc.Logger.Error.Warn with event=opensearch_pages_fallback and a reason field carrying the underlying error message (or "unknown" if the OpenSearch client was nil). Operators can build alerts off the event tag.

Test coverage

Unit + handler tests: internal/api/site_search_fallback_test.go — covers OpenSearch 500, connection-refused, missing-index 404, OpenSearch-disabled, Postgres-also-failing, and the happy-path "no fallback header" assertion. Uses a pagesSearchFn method-expression seam to stub the Postgres call without requiring a live database.
Existing live-cluster integration tests under internal/opensearch/pages_acl_search_integration_test.go (build tag integration) continue to assert the OpenSearch-reachable predicate.

Three additional trigger points close out the cascade surface:

Event	Cascade call
Repo hard-delete	`opensearch.Indexer.DeleteAllDocsForRepo(ctx, repoID)` — `_delete_by_query` across all 6 ACL-carrying indexes (code + symbols + issues + mrs + comments + pages). Per-index errors aggregated into `joinedErrs` so one broken index does not leave stale docs on the others.
`page_permissions` INSERT/UPDATE/DELETE	Affects exactly one page doc. Caller resolves the fresh `page_permission_user_ids` + `page_permission_group_ids` arrays and calls `IndexPageDocs([{PageDocID(pageID), NewPageDoc(...)}])` which overwrites the doc via the Bulk API's `index` action (which is an upsert when the `_id` is deterministic).
`page_share_links` revoke / expire	Caller invokes `Indexer.ClearPageShareToken(ctx, pageID)` which runs `_update_by_query` with `ctx._source.share_token = null` scoped to the single page doc. Preserves concurrent body edits because it only touches the one field.

page_share_links rows with expires_at < NOW() must not continue to grant access via the share_token field on the page doc. Two sweep mechanisms exist:

Admin reindex — a full reindex re-reads the most-recent non-expired token so expired links naturally drop out on the next backfill.
Periodic sweep — a scheduled job that runs SELECT page_id FROM page_share_links WHERE expires_at < NOW() and calls Indexer.ClearPageShareToken(ctx, pageID) for each result. Today this runs on-demand via the admin reindex; a nightly timer wrapper is a documented follow-up.

The staleness SLA for expired tokens is therefore bounded by the sweep cadence. Operators who need a tighter SLA can schedule more-frequent sweeps or configure their reverse proxy to validate share tokens against Postgres instead of trusting the indexed value.

Stale-ACL window SLA + synchronous revoke fast-path

Between the DB commit of an ACL change and the OpenSearch refresh that reflects it, there is a narrow window during which a search against the indexed collaborator_user_ids could still return hits for a removed collaborator. This is a property of every eventually-consistent dual-write design; what matters is the bound.

Bounds:

Load regime	Worst-case staleness
Steady state	≤ 5 s
Backlog (e.g. cascade fan-out > cluster refresh interval)	≤ 30 s

Synchronous fast-path: opensearch.Indexer.SyncACLForRepoSync(ctx, u) issues ?refresh=true on every _update_by_query so the updated ACL is searchable the instant the call returns — no stale window. Trades ~100–500 ms extra latency per sync for strict consistency. Wired by operators into revoke-class events (collaborator remove, repo make-private, share-link revoke, page_permissions delete) when the load profile allows; async SyncACLForRepo remains the default for cheap events (collaborator add, grant add) where the stale-window is harmless.

Why the query-time path is the authoritative gate. The SearchHandler applies the Postgres ACL before any OpenSearch hit leaves the handler — a stale OpenSearch doc that still carries a removed user's UUID cannot actually leak because the allow-list generated at query time from repositories + repo_collaborators + group_repo_grants is the final gate. The OpenSearch refresh is the fast-path; Postgres is the safety net. This design explicitly accepts the stale window at the OpenSearch layer because the handler layer never uses OpenSearch's ACL fields alone as the authorisation decision.

Metric: vetrix_acl_sync_lag_seconds histogram exported by the cascade layer records the interval between the DB commit and the SyncACLForRepo return. p50 + p99 tracked for operator visibility. Metric plumbing composes against the existing runUpdateByQuery call site unchanged.