Operations Guide

Day-2 concerns for running Vanzik in production: log rotation, scaling, capacity limits, and restart semantics. Read docs/DEPLOYMENT.md first — this guide assumes you already have a correctly configured server.

---

Audit log rotation#

Vanzik rotates audit.jsonl by default. The size-triggered rotator is on out of the box, controlled by these vanzik server flags:

Rotated files carry a _meta.rotated_from chain pointing to the previous segment (see docs/FILE_FORMATS.md) so external consumers can stitch the timeline.

Do not stack rotators. Do NOT also configure logrotate (or any external rotator) against audit.jsonl — two rotators on one file corrupt the rotation index and breaks the _meta.rotated_from continuity. Pick one. If you need logrotate's archival semantics, set --audit-max-size-mb 0 first to disable Vanzik's own rotator.

Two follow-on consequences regardless of who rotates:

1. Disk usage grows proportional to request volume × retention. A busy deployment can produce hundreds of MB per day; the defaults bound this to ~500 MiB across 5 backups.
2. Startup replay seeds the decision counters from the replay checkpoint (<audit-log>.replay-checkpoint, which carries the lifetime tally) and re-reads only the entries written since it was last written — following the _meta.rotated_from chain through any archives rotated in between — so /metrics and /api/stats survive restarts with the totals a never-restarted process would show. The file audit logger also rewrites the checkpoint after every rotation (before --audit-max-backups prunes old archives) and on shutdown, so pruned files keep their counts. A boot with no checkpoint (the first one, or after vanzik migrate --reset-checkpoint) replays the whole live file once; a multi-GB live file delays counter accuracy until that scan completes. The --audit-backend=store logger has no checkpoint and re-queries the store on every boot.

External shipping (compatible with default rotation)#

The audit log is append-only JSON Lines with one decision per line (schema in docs/FILE_FORMATS.md). Any log shipper that handles newline-delimited JSON works — Vector, Fluent Bit, Filebeat, Datadog agent, etc. Point your shipper at audit.jsonl AND the rotated audit.jsonl.<YYYYMMDDTHHMMSSZ>[.gz] siblings (glob audit.jsonl.[0-9]*, or <audit-log>.[0-9]* for another --audit-log path; a plain audit.jsonl.* also picks up audit.jsonl.replay-checkpoint); the shipper does not need to truncate anything.

Disabling default rotation (legacy / external rotator path)#

Set --audit-max-size-mb 0. Vanzik then opens audit.jsonl with O_CREATE|O_WRONLY|O_APPEND at mode 0600 and never rotates it. Use this when you operate a managed log pipeline that handles rotation upstream (e.g., a sidecar that reads via tail -F).

If you must use logrotate against Vanzik's audit file, use the copytruncate directive AND --audit-max-size-mb 0:

/var/lib/vanzik/audit.jsonl {
    daily
    rotate 14
    compress
    copytruncate
    missingok
    notifempty
    create 0600 vanzik vanzik
}

Keep copytruncate — the default create-based rotation breaks Vanzik's open fd.

Reducing startup replay time#

If restart latency bothers you more than disk usage, archive aggressively:

• Keep audit.jsonl scoped to the last N hours of decisions.
• Ship everything older to external storage (S3, GCS, Datadog, etc.) where /v1/audit queries do not need to reach.

Historical audit queries then become a two-tier lookup: recent entries from the local file, older entries from the external store. /v1/audit's ?offset= + ?limit= support pagination within the local file; cross-tier paging is your responsibility.

---

Audit redaction#

vanzik server masks secrets in every request before it is stored or shown (--audit-redact, default true). Matches are replaced with [REDACTED] in the command, path, domain, action, URL, reason and every meta value.

What is masked: bearer tokens, Authorization: / x-api-key: / api-key: header values, AWS access keys (AKIA…), GitHub tokens (ghp_, gho_, ghs_, ghu_, ghr_, github_pat_), Slack tokens (xox?-), LLM provider keys (sk-…), Google API keys (AIza…), JWTs, PEM private keys, and secret= / token= / password= / api_key= pairs — plus the policy's notifications.redaction.extra_patterns. It is pattern-based and best effort: a secret in a shape none of these match is stored as sent.

Where: the audit file or store, the overflow spill file, GET /v1/audit, the SSE stream (so the dashboard feed), and /api/pending. The MCP gateway and LLM proxy mask their --fail-audit-log entries with the built-in patterns. Redaction runs in the audit worker goroutines and the SSE writers, not on the /v1/check request goroutine — except with --audit-buffered=false, where the audit write itself is synchronous.

What still holds the original request:

• The approval store (the approvals table in SQLite or Postgres): replaying an approval_id compares the retry to the original request field by field. Protect vanzik.db / the database credentials accordingly.
• The in-memory require_prior index, until the next restart; after a restart it is rebuilt from the redacted audit trail, so a require_prior condition that names a secret-bearing command may stop matching.

Also note: only the --policy file's extra_patterns are used (not tenant policies'); audit files written before 1.2.0 are not rewritten; notifications were already redacted and still are. --audit-redact=false restores verbatim audit content and logs WARNING: --audit-redact=false: … at startup.

Sealing and anchoring#

The file audit log can be sealed into a hash chain: vanzik server --audit-seal --audit-seal-key-file <key>. Every entry gets seq, prev_hash and hash, and each rotated file's header names the last entry of the file before it. vanzik audit verify checks all of it offline. Sealing is off by default, works with the file backend only (--audit-backend=store with --audit-seal is refused at startup), and is not encryption. The format and the hashing rule are in FILE_FORMATS. vanzik setup turns it on for new setups, with a key at <data-dir>/audit-seal.key.

The key. Any random string works; openssl rand -hex 32 > audit-seal.key && chmod 600 audit-seal.key makes a good one. The server refuses a key file that is missing, empty or readable by every user. Keep a copy somewhere other than the audit host: verify needs it, and anyone who has it can forge a valid chain. To change keys, restart the server with the new key file; the entries after the restart are sealed with it, and verify needs both (--key-file old --key-file new). Without --audit-seal-key-file the chain is plain SHA-256 and the server warns at startup: it shows naive edits, but anyone who can write the file can recompute it.

Anchors. An anchor is the head of the chain: the seq and hash of the newest sealed entry. The server publishes one every --audit-anchor-interval (default 5m, 0 turns the periodic one off; skipped when nothing new was sealed), at startup, at each rotation and at shutdown:

• a log line, audit anchor segment=<segment_id> seq=<n> hash=<hex>, on stderr, so it reaches journald, syslog or whatever collects the server's log;
• the vanzik_audit_seal_seq gauge on /metrics;
• GET /v1/audit/head (Bearer or session, like GET /v1/audit) returns the current head as JSON at any time. See API.

Ship anchors off the box: forward the server log to a SIEM, scrape the gauge into Prometheus, or poll /v1/audit/head from another machine and keep what it returns. Later, vanzik audit verify --anchor <seq>:<hash> checks that the log still contains that entry with that hash.

What verify proves. If verify passes with your key, no sealed entry was changed, removed, inserted or reordered by someone without the key, and no file between the oldest one left and the live file is missing. Archives are named in write order and pruning deletes the oldest first, so verify tells pruning apart from a removed or unlinked archive: an archive still on disk behind a missing link is reported as broken. Once a file says it is sealed, an entry without a hash is reported as inserted, and with --key-file a log that declares no seal at all, or declares it only after every entry, fails. Without a key, a log whose entries are all unsealed is reported as NOT SEALED (exit 1), never as intact. With an anchor that matches, nothing up to the anchor's seq was rewritten, even by someone with the key.

What it does not prove.

• That the newest entries are all there. Cutting entries off the end leaves a shorter chain that still verifies. An anchor taken before the cut shows it (MISSING: the log ends at seq …); so does the startup check against the replay checkpoint (WARNING audit: … possible truncation, vanzik_audit_seal_tail_mismatch_total).
• Anything about pruned files. When --audit-max-backups or --audit-max-age-days delete archives, verify says where the chain starts (The chain starts at seq N (older files were pruned)) and reports anchors in the pruned range as not checkable.
• Anything about entries written before sealing was turned on. verify counts them and says they are not protected.
• That an entry is true. The chain covers what the server wrote, after redaction.

Turning sealing off, or off and on again, is reported as broken: the entries written unsealed are not protected. A crash in the middle of a write leaves a cut line. The server ends it (on restart, or right away when a write comes up short) and writes a _meta line that continues the chain from the last whole entry; verify lists the cut line as a warning and goes on checking everything after it. A cut line that is not followed by that _meta line, with the chain continuing exactly where it stopped, is reported as broken.

Exit status of verify: 0 intact, 1 broken or an anchor that doesn't match, 2 command-line mistake (or a keyed log without --key-file), 3 a file can't be read. --output json prints the report for scripts.

Multi-instance deployments#

Replicas can share state through PostgreSQL. Which mode you are in depends on the store backend:

Shared mode — PostgreSQL (v1.0)#

Set --store-dsn postgres://… on every replica and give each a distinct --node-id (defaults to the hostname — fine for pods, wrong for two processes on one host). Replicas then converge through background reconciliation every --reconcile-interval (default 2s); the enforcement hot path never waits on the database.

What "shared" means, precisely:

• Rate-limit buckets and session costs are bounded-overshoot, not globally strict. Each node enforces against its in-memory view and learns the other nodes' consumption on the next reconcile. Worst case, admissions overshoot a limit by ≈ reconcile-interval × peak rate per additional replica; steady state converges to the global limit. If you need a hard global ceiling, backstop with an upstream rate limiter.
• Approvals are visible cluster-wide. A pending approval created on node A can be approved via node B's dashboard; resolutions propagate within one reconcile interval. Conflicting resolutions converge to DENY, a resolved approval is never un-resolved, and a one-shot ALLOW consumed on any node is spent cluster-wide once reconciled (within the staleness window a replay racing to another node can still be honored there — size --reconcile-interval accordingly if that window matters to you). Every node's own deny is enforced immediately (fail-closed); only adopting a remote decision is subject to the interval.

Per-instance mode — SQLite (the default)#

Two state stores do not propagate across replicas (persistence makes them survive a restart, but each replica keeps its own working copy):

• Rate-limit buckets (pkg/ratelimit/ratelimit.go): key "<scope>:<tenant>:<agent_id>"; an agent hitting two replicas round-robin gets up to 2 × max_requests per window. MaxBuckets = 10000 per instance.
• Session-cost accumulators (pkg/policy/engine.go sessionCosts): keyed by tenant and session_id; each replica enforces max_per_session against its local total. --session-cost-ttl evicts idle entries; it does not share state.

Mitigations if you must run multiple replicas without PostgreSQL:

1. Run one replica. Scale vertically. Vanzik is a latency-sensitive sidecar, not a throughput bottleneck — one well-sized instance usually handles thousands of agents.
2. Pin sessions. Configure your load balancer for session affinity on session_id (header or cookie). Rate-limit buckets still drift across replicas for multi-session agents, but cost accounting becomes accurate.
3. Divide limits by replica count. If you run 3 replicas and want a global cap of 30 req/min, set max_requests: 10. Works only for steady-state traffic; burst behavior remains sloppy.
4. Aggregate externally. Have each replica emit /metrics to Prometheus; alert on aggregate drift between declared and observed rates. Treat Vanzik rate limits as best-effort and backstop with an upstream rate limiter (the reverse proxy, a CDN, or an API gateway).

---

Approval queue capacity#

Behavior at capacity:

• A new approval into a full queue evicts the one oldest resolved entry (evictOldestResolvedLocked) before inserting. Unresolved pendings are never evicted.
• If every entry is unresolved, /v1/check calls that would require approval get 503 Service Unavailable with Retry-After: 30. Approving or denying never returns 503.
• Approvals survive restarts — --persist (default true) snapshots the queue to the SQLite store and rehydrates it on boot. Only with --persist=false, or for entries created inside the final ≥1 s store-sync window before a hard crash, are pending approvals lost (agents polling /v1/status/{id} then hit their SDK wall-clock timeout). See CLI.md.

Operational guidance:

• Resolve approvals fast. A backlog >1k almost always means nobody is triaging.
• Set SDK approval timeouts higher than your human SLA, but not so high that a restart silently hangs agents. A practical rule: timeout = expected_human_response_time + restart_budget.
• Avoid restarts during business hours. Use rolling restarts with drained approval queues if at all possible.
• Watch vanzik_pending_approvals (gauge in /metrics). Sustained growth = triage is not keeping up.

---

Session store limits#

At capacity, a login first sweeps expired sessions; live sessions are never evicted. If the store is still full, /auth/login returns 503 with Retry-After: 5 (under pathological login bursts, for example).

Operational guidance:

• Most deployments never approach 1024 dashboard sessions. If you do, it is usually a bot probing /auth/login — firewall it.

• MaxSessions is not yet configurable via YAML. If you legitimately need more, open an issue with your use case.

• Session TTL is configurable:

  proxy:
    session:
      ttl: "8h"   # dashboards stay logged in for an operator shift
  

---

Notifier queue capacity#

Dispatch is non-blocking: Dispatcher.Send(event) enqueues one job per configured notifier. If the queue is full (a slow Slack webhook is the usual culprit), the event is dropped and counted:

• vanzik_notify_events_dropped_total{notifier, reason} — Prometheus counter.
• notify.DroppedEvents — atomic uint64 for in-process Go consumers.
• vanzik_notify_queue_depth — current depth of the one queue all notifiers share (no notifier label).
• vanzik_notify_dispatch_duration_seconds{notifier} — histogram of per-target HTTP latency.

Tune by:

• Reducing notifier fanout (drop unused targets).
• Lowering per-target timeout so a flaky endpoint stalls fewer workers.
• Setting a lower per-target timeout on Slack while keeping webhook timeout long (use notifications.dispatch_timeout + per-target timeout).

---

Session-cost sweeper#

If your agents use long-lived session_ids, the sessionCosts map can grow unbounded. Two flags control eviction:

vanzik server \
  --session-cost-ttl 24h \
  --session-cost-sweep-interval 1h \
  ...

• --session-cost-ttl 0 (default): entries never expire.
• --session-cost-sweep-interval defaults to max(ttl/4, 1m) if unset.

Eviction logs INFO: session-cost sweeper evicted N entries (ttl=…) when non-zero.

Trade-off: a short TTL resets session totals mid-run if an agent goes idle longer than the TTL. Pick ttl > your longest expected idle gap within a session.

---

MCP Gateway and LLM API Proxy#

Both wire-level enforcement points are stateless — they fan every gated request out to the Vanzik server. Restart freely, no audit replay on boot.

• Health: the LLM API Proxy serves GET /healthz on its listen port — use it as the readiness probe. The MCP Gateway is a stdio bridge with no HTTP listener; probe the process, not a port.
• Metrics: neither proxy binary exposes /metrics — the central vanzik server does. See OBSERVABILITY.md for what is and isn't observable at the proxies.
• Shutdown: the LLM API Proxy buffers tool calls inside streaming responses; on SIGTERM it drains in-flight requests for 10 s, then closes them. Give the host a stop timeout above 10 s (the systemd and Kubernetes defaults, 90 s and 30 s, already are); a longer one doesn't extend the 10 s drain, and a hard kill truncates the client's response.
• Version skew: the /v1/* wire protocol is frozen across 1.x and 2.x, so mixed server and proxy versions interoperate. Upgrading the server first, then the proxies, is still the safe order.
• Topology: prefer per-agent sidecars — shared proxies couple every agent's lifecycle.

---

Backups#

Three things are worth backing up:

1. Policy files (configs/*.yaml) — version control them. Every change should land through a commit and, if possible, a CI vanzik validate check.
2. Audit log archive — whatever your shipper lands in S3 / object storage. The local audit.jsonl is the working copy; the archive is truth. Retain per your compliance requirements.
3. The durable store — vanzik.db plus its -wal/-shm sidecars in --data-dir. It holds pending approvals, rate-limit buckets, cost accumulators, and (with --audit-backend=store) the audit trail. Snapshot it with the process stopped, or use SQLite-aware tooling (sqlite3 .backup) on a live file — copying vanzik.db alone mid-write can produce a torn backup.

The dashboard session store is the one piece that stays ephemeral by design — don't back it up.

---

Restart checklist#

Before a scheduled restart in production:

• Pending approvals checked (curl .../api/pending) — with --persist (default) they survive the restart but sit unresolvable while the process is down; with --persist=false they are lost.
• Agents are either idle or configured with a generous wait_for_approval / waitForApproval timeout.
• Audit log recently archived, so replay latency on boot is bounded.
• You know how to roll back the binary if the new version fails (vanzik version).

Graceful shutdown handles SIGINT and SIGTERM — the process drains in-flight requests up to ShutdownTimeout before exiting.

---

Capacity sizing#

For a single replica on modern hardware:

• CPU: 1 vCPU comfortably handles 500+ RPS of /v1/check (mostly spent in YAML policy evaluation and JSON decode).
• Memory: base ~50 MB; linear growth with audit log size during startup replay only.
• Disk: audit log drives this. Size for at least 2× your retention window.

For higher throughput, benchmark with your actual policy before scaling. The response headers X-Vanzik-Policy-Ms, X-Vanzik-Audit-Ms, X-Vanzik-Total-Ms give you a per-request breakdown.

---

Related docs#

• docs/DEPLOYMENT.md — reverse proxy, TLS, CORS, initial setup.
• docs/OBSERVABILITY.md — /metrics, alerts.
• docs/FILE_FORMATS.md — audit log schema and migration format.
• docs/TROUBLESHOOTING.md — symptom-keyed diagnostics.
• docs/CONFIG.md — YAML tunables (session TTL, audit limits, body caps).