Email-queue (AMQP) — operator guide

Vetrix can hand outbound email off to a RabbitMQ broker (the "EmailQueue" event bus) instead of sending inline. This page covers how to configure the broker, how to read the broker-health admin card and the /api/v1/admin/health rabbitmq slice, the structured log lines the AMQP path emits, and the Postgres poller that keeps the feature resilient when the broker is missing or slow.

The broker is optional. With the feature off — the default — Vetrix delivers email exactly as it always has (inline send + the Postgres retry poller) and has no broker dependency.

Configuration
The events.enabled admin setting
Broker-health admin card — the four states
The Postgres poller fallback
Reading /api/v1/admin/health.rabbitmq
Log grep cheat-sheet
Broker infrastructure hardening

Configuration

Both broker URLs are environment-only secrets. They carry their credentials in the URL userinfo, so — exactly like DATABASE_URL, REDIS_URL, and SMTP_PASSWORD — they must never be written to app.toml. A guard test (internal/admin/settings_events_amqp_url_vbe928_test.go) enforces that EVENTS_AMQP_URL can never be stored in app_settings.

`EVENTS_AMQP_URL` (required to enable the feature)

The RabbitMQ broker URI for the EmailQueue event bus.


Scheme	`amqp://` or `amqps://`
Format	`amqp://user:password@host:5672/vhost`
Credentials	Carried in the URL userinfo (`user:password@`)
Where	Env-only. Never in `app.toml`.
Effect when set	The broker is wired only when this is set and the `events.enabled` admin setting is true (see below). Otherwise the no-op bus is installed and the server boots with no broker dependency.

The host (everything after @, e.g. host:5672) is the only part of the URL that ever appears in a log line — the userinfo is stripped before any log or error crosses the package boundary, so the password never reaches error.log, the admin card, or any API response.

If the feature is enabled but the broker is unreachable at boot, the initial connection is synchronous and the server exits fast with:

events: connect amqp bus: <scrubbed dial error>

This fail-fast-on-boot behaviour is intentional: a bad broker URL fails loudly rather than silently degrading. (The right place to absorb a transient broker outage is the compose depends_on + restart: gate — see Broker infrastructure hardening.)

`EVENTS_AMQP_MANAGEMENT_URL` (optional)

The RabbitMQ management HTTP API base URL. It is used solely to source the retry_delayed_depth value shown on GET /api/v1/admin/email/queue/stats.


Scheme	`http://` or `https://`
Format	`http://user:password@host:15672`
Credentials	Carried in the URL userinfo (basic auth)
Where	Env-only. Never in `app.toml`.
Requires	The `rabbitmq_management` plugin enabled on the broker.

Why it exists: retries are held in the vetrix.events.retry x-delayed-message exchange. The delayed-message plugin keeps those messages inside the exchange, not in a classic queue, so their count is not observable via an AMQP passive queue declare. The only live source is the management API's messages_delayed gauge, read from:

GET {management-url}/api/exchanges/{vhost}/vetrix.events.retry
  → { ..., "messages_delayed": <N>, ... }

When this variable is unset, or the management API is unreachable / returns an unparseable response, retry_delayed_depth renders null (never a misleading 0). The probe carries a 3-second timeout so a slow management API degrades quickly rather than stalling the stats handler.

The `events.enabled` admin setting

events.enabled is the master switch for the EmailQueue event bus.

Default: false (off). With it off, the inline + Postgres-poller delivery path is used and there is no broker dependency.
Where to flip it: Admin → Settings (/admin/settings), in the "Email Queue" group. The toggle is labelled "Enable event bus".
Takes effect only when EVENTS_AMQP_URL is also set on the server. Flipping it on with no broker URL configured has no effect — the no-op bus stays installed.

The flag is read at server boot to decide whether to dial the broker. Turning it on (with EVENTS_AMQP_URL set) wires the AMQP bus on the next boot; with the flag off, EVENTS_AMQP_URL is ignored entirely.

Broker-health admin card — the four states

Screenshots: the admin card is rendered by the running web UI. The card labels and colours are the frontend's rendering of the backend /api/v1/admin/health rabbitmq slice. The state semantics and the JSON that drives each state are given below; treat the card-label wording as indicative rather than pixel-exact.

The card is driven entirely by the rabbitmq slice of GET /api/v1/admin/health (re-probed on every request — there is no cache). The slice has four fields: configured, reachable, latency_ms, and ping_err. The card maps them to four operator-facing states:

Card state	What it means	Backing JSON	What triggers it
Not configured	The broker is not wired. Email uses the inline/poller path.	`configured: false`	`EVENTS_AMQP_URL` is unset or `events.enabled` is off. The probe handle is nil, so no I/O runs. Identical affordance to the "Redis not configured" card.
Connecting	The broker is configured but not currently reachable from Vetrix's live connection — typically the bounded-backoff reconnect loop is between connections.	`configured: true`, `reachable: false`, `ping_err` set	The most recent `Ping` failed: the supervisor's connection handle is nil or reports `IsClosed()`. The reconnect supervisor retries with backoff (1s → 30s).
Connected	The live broker connection is up and answering.	`configured: true`, `reachable: true`, `ping_err` empty	The most recent `Ping` succeeded (the live connection exists and is not closed).
Degraded	The broker is configured but the probe is failing — Vetrix cannot confirm broker liveness.	`configured: true`, `reachable: false`, `ping_err` set	Same JSON as "Connecting": a non-nil `Ping` error. The distinction is operational, not a separate backend field — a brief, self-clearing failure during a reconnect reads as "connecting"; a persistent failure (broker down, wrong URL, network partition) reads as "degraded". In both cases `ping_err` carries the verbatim, credential-redacted error string.

Notes:

The health Ping is deliberately a connection-state check, not an AMQP round-trip: it reports the supervisor's live connection's IsClosed state under lock. A nil connection or IsClosed() == true both surface as the same "broker not connected" error.
The probe runs under a 500 ms deadline so a wedged broker cannot stall the admin overview page.
ping_err is always run through the URL-credential redactor before it reaches the card or error.log, so AMQP basic-auth credentials never leak.

A finer-grained connection state (disconnected / connected / reconnecting) is also available on the EmailQueue stats endpoint (GET /api/v1/admin/email/queue/stats) for operators who need to tell "never wired" apart from "dropped and retrying".

The Postgres poller fallback

Vetrix is resilient to a missing or delayed broker. Every outbound email is written to the email_deliveries table as a queued row before anything is published. The Postgres retry poller (internal/email/retry.go) runs as the live fallback:

It drains email_deliveries rows whose next_retry_at has come due, re-renders them, and re-drives the wire send through the same Deliverer.attempt path the first send used.
It reconciles rows the broker never delivered — broker down, a lost publish, or a consumer crash. Under a healthy broker this stays near zero (the EmailPollerReconcile metric).
It and the queue consumer share the same status-guarded claim (Store.ClaimQueued). When both race the same due row, exactly one claim matches the still-queued row and wins; the loser skips it. This is the at-least-once de-dup primitive — a row is sent exactly once whether the broker or the poller drives it.

The practical upshot: if the broker is down, queued rows simply sit in Postgres and the poller reconciles them on its next tick; nothing is lost. When a transient send fails on the consumer path with broker-retry off, the consumer ACKs and leaves the queued row for the poller rather than re-driving at the broker layer (avoids double-attempts).

The poll cadence is read live from the email.reconcile_poll_interval admin setting (falling back to the deprecated email.retry_poll_interval and then the built-in default).

Reading `/api/v1/admin/health.rabbitmq`

GET /api/v1/admin/health returns the system-health document; its rabbitmq key is the broker slice. Wire shape (internal/admin/health_rabbitmq.go, struct RabbitMQHealth):

Field	Type	Meaning
`configured`	bool	`true` only when the broker probe handle is wired (`EVENTS_AMQP_URL` set and `events.enabled` on).
`reachable`	bool	`true` when the most recent `Ping` succeeded.
`latency_ms`	int	Round-trip duration of the just-completed `Ping`, in milliseconds. `0` when `configured` is false.
`ping_err`	string	Verbatim (credential-redacted) `Ping` error. Omitted (`omitempty`) when empty.

Example payloads per state

Not configured — EVENTS_AMQP_URL unset or events.enabled off:

{
  "configured": false,
  "reachable": false,
  "latency_ms": 0
}

Connecting / Degraded — configured but the probe is failing (the JSON is the same; "connecting" vs "degraded" is the operator's read of whether it is transient or persistent):

{
  "configured": true,
  "reachable": false,
  "latency_ms": 2,
  "ping_err": "events: amqpBus not connected"
}

Connected — live connection up and answering:

{
  "configured": true,
  "reachable": true,
  "latency_ms": 3
}

The admin slice name is rabbitmq (lowercase, operator-facing). The public /api/v1/status equivalent uses the human-readable label "Message Queue"; the two surfaces are intentionally distinct.

A one-liner to pull just the slice (matches the INC-2026-06-04 probe):

curl -sS -H "Authorization: Bearer $TOK" \
  https://<host>/api/v1/admin/health \
  | python3 -c 'import json,sys; print(json.dumps(json.load(sys.stdin).get("rabbitmq"), indent=2))'

Log grep cheat-sheet

Every structured log line the AMQP / EmailQueue path emits. Messages are verbatim; the listed keys are the structured attributes carried alongside. The connection-lifecycle lines (prefix events:) identify the broker by host only — the userinfo/password is never logged. They land in the backend three-file logger (error.log for errors, the app log for info/warn).

Connection lifecycle (`internal/events/amqp.go`)

Level	Message	Keys
Warn	`events: amqp connection closed, reconnecting`	`host`, `error`
Warn	`events: amqp reconnect attempt failed`	`host`, `error`, `next_backoff`
Info	`events: amqp connection re-established`	`host`
Info	`events: pause raced reconnect establish; cancelled freshly established consumers`	`host`
Warn	`events: failed to cancel consumer after raced pause`	`event`, `error`
Warn	`events: failed to cancel consumer on pause`	`event`, `error`

Boot-time fatal (wrapped from cmd/server/main.go, printed to error.log then exit): events: connect amqp bus: <scrubbed dial error>.

Message handling / parking (`internal/events/amqp.go`)

Level	Message	Keys
Warn	`events: handler returned error, requeueing`	`event`, `id`, `error`
Error	`events: handler panicked, parking message`	`event`, `id`, `panic`
Warn	`events: parked poison message`	`event`, `id`, `reason`
Error	`events: failed to park poison message; leaving un-acked for redelivery`	`event`, `id`, `reason`, `error`

Consumer (`internal/email/consumer.go`)

Level	Message	Keys
Info	`vetrix: email queue consumer starting`	`routing_key`, `workers`, `prefetch`
Warn	`vetrix: email consumer parking undecodable message`	`message_id`, `err`
Warn	`vetrix: email consumer parking message with nil delivery id`	`message_id`
Debug	`vetrix: email consumer claim miss (no-op)`	`delivery_id`
Error	`vetrix: email consumer claim failed`	`delivery_id`, `err`
Error	`vetrix: email consumer re-publish failed; poller will reconcile`	`delivery_id`, `err`

Postgres poller / fallback (`internal/email/retry.go`)

Level	Message	Keys
Info	`vetrix: email retry worker started`	`poll_interval`
Info	`vetrix: email retry worker stopping`	—
Error	`vetrix: email retry list failed`	`err`
Error	`vetrix: email reconcile claim failed`	`id`, `err`

Quick greps

# Every broker connection-lifecycle line (host only, never credentials):
grep -E 'events: amqp (connection (closed|re-established)|reconnect attempt failed)' error.log

# Boot-time broker-unreachable (the INC-2026-06-04 symptom):
grep 'events: connect amqp bus' error.log

# Parked / poison messages (need an operator to requeue or purge):
grep -E 'events: (parked poison message|failed to park poison message|handler panicked)' error.log

# Poller fallback actually doing work (should be near-zero under a healthy broker):
grep -E 'vetrix: email (retry worker|reconcile claim failed|retry list failed)' error.log

Broker infrastructure hardening

The RabbitMQ broker container lives in the separate production-implementation repo (not the vetrix code repo). Proposed Dockerfile / docker-compose changes are collected for operator review in docker-updates.md (outside the code repos).

The incident write-up INC-2026-06-04 in docker-updates.md hardens the broker against exactly the boot-time events: connect amqp bus: dial tcp …: connection refused symptom this page describes. It applies four fixes together:

restart: unless-stopped on the broker (and on appmerc_vetrix / appmerc_vetrixworker) so a single crash no longer keeps the broker down, and a transient broker outage no longer permanently kills Vetrix — Vetrix exits, Docker restarts it, the broker is back, and the reconnect succeeds.
A healthcheck: (rabbitmq-diagnostics -q check_port_connectivity) so dependants can wait on the broker genuinely accepting AMQP connections.
A persistent volume (./site/rabbitmq/data:/var/lib/rabbitmq) so the operator-defined user / vhost / permissions survive a container recreate. (Mind the bind-mount ownership note — UID/GID 999.)
depends_on … condition: service_healthy on the broker for Vetrix and the worker, so docker compose up -d blocks them until the broker's healthcheck is green.

After applying, the acceptance check is that /api/v1/admin/health.rabbitmq reports configured: true, reachable: true — i.e. the Connected card state above. The incident explicitly does not touch Vetrix code: the fail-fast-on-AMQP behaviour is intentional, and the right layer to absorb transient broker outages is the compose depends_on + restart: gate.

See INC-2026-06-04 in docker-updates.md for the full apply order, the operator commands, and the optional declarative-definitions follow-up (C.4).