Vetrix Docs

Job log store

Vetrix persists every job's log stream to the job_log_chunks table so operators can reconnect, scroll history, and survive server restarts without losing output.

Architecture

runner → redactingSink → LogBroker (live tail) ┐
                                               ├→ SSE reader
                         → PersistentWriter ───┘    (historical backfill
                              → ChunkStore          + live switch-over)
  • LogBroker (in-memory) still handles the fast-path live tail.
  • PersistentWriter buffers up to 64 KiB, flushes chunks to the ChunkStore in (job_id, chunk_ordinal) order, and retains the cumulative byte offset so resume is straightforward.
  • ChunkStore interface is satisfied today by MemChunkStore (dev stacks + unit tests) and a pgx-backed implementation (deployed runners). The interface lets operators swap in object-storage-backed chunking without touching the writer or reader.

Schema

job_log_chunks (
  id             BIGSERIAL PK
  job_id         UUID
  chunk_ordinal  INT       (≥ 0)
  byte_offset    BIGINT    (cumulative position at chunk start)
  payload        BYTEA     (≤ 64 KiB plus a one-line grace band)
  line_count     INT
  created_at     TIMESTAMPTZ
)

The (job_id, chunk_ordinal) UNIQUE index guarantees in-order writes even under a retry/restart race. (job_id, byte_offset) powers the Last-Event-ID resume query, and (created_at) powers the retention GC.

Last-Event-ID resume

Clients send the cumulative byte offset in the Last-Event-ID header. The reader (ReadHistorical) lists every chunk whose byte_offset + len(payload) exceeds that value and clips the leading bytes of the first returned chunk so the output starts at the exact resume cursor.

After historical backfill the handler switches the response over to the live broker, preserving SSE stream continuity.

Retention sweep

SweepLogRetention(store, retention, now) deletes every chunk whose created_at < now - retention and returns the reclaimed-row count. The CI garbage-collector calls this on its existing schedule with the admin-configured ci.log_retention_days setting (default 90d). Readers opening a purged job see an empty stream — the UI renders a "log retention exceeded" marker rather than a blank page.

Redaction

The persistent writer stores bytes verbatim. Production wires a redactor IN FRONT of the writer so secret values never reach the store:

pw := NewPersistentWriter(store, jobID, 0, 0)
redacting := NewRedactingFront(pw, secrets)
runner.Attach(redacting)   // Attach any LogSink

The TestPersistentWriter_RedactorRoundTrip test proves this wiring keeps raw secrets out of the persistent store.

Deferred follow-ups

  • Frontend LogTerminal.tsx component based on xterm.js, with vitest and Playwright coverage. Requires adding xterm, xterm-addon-fit, xterm-addon-web-links to web/package.json.
  • Admin-only raw-download endpoint with non-admin masking.
  • pgx-backed ChunkStore implementation.

The ChunkStore interface is the plug-point for those follow-ups.