Observability

Everything Vanzik exposes for monitoring: the /metrics endpoint, /api/stats, response-header timings, and example Prometheus / alerting rules.

Source of truth: pkg/metrics/metrics.go.

---

/metrics — Prometheus text format#

• Unauthenticated, scrape-friendly.
• Always emits HELP and TYPE lines for every metric, including those with no observations yet.
• Zero external deps — lightweight hand-rolled exposition.

curl -s http://127.0.0.1:8080/metrics

Counters#

Gauges#

Histograms#

Duration bucket bounds (ms, shared across the three _ms histograms):

0.25, 0.5, 1, 2, 5, 10, 25, 50, 100, 250, 500, 1000, 2500, 5000, 10000, +Inf

Notifier histogram buckets (seconds):

0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, +Inf

Bucket boundaries are treated as a stable contract. Re-bucketing invalidates historical Prometheus data.

---

MCP Gateway / LLM API Proxy metrics (v0.5+)#

Neither proxy binary exposes /metrics. The LLM API Proxy serves only its API routes plus GET /healthz; the MCP Gateway is a stdio bridge with no HTTP surface at all. Per-request gating metrics for both flow through the central server's /metrics, distinguished in the audit stream by Entry.Transport (llm_api_proxy / mcp_gateway). See PROXY_ARCHITECTURE.md §8.

The LLM proxy additionally records these series in its own process (pkg/metrics), which serves no /metrics. Because the registry is shared code, the central server's /metrics lists all six too, but they never move there: the two unlabelled ones read 0 and the four provider counters have no samples, because the counting happens in the proxy process, not the server. Don't alert on them — such an alert never fires. They matter only to code embedding the proxy in-process (they are listed here so the names stay reserved):

The MCP gateway emits no metrics of its own; its decisions surface in the central server's audit stream and counters.

Approval-id replay defense (central server): vanzik_approval_replay_mismatch_total — non-zero is a security alert. Its companion vanzik_approval_replay_refused_total{reason="consumed"|"expired"} counts shape-matched replays refused by the approval state machine (approvals are one-shot and time-boxed by --approval-validity); a rising consumed count means something is trying to reuse one human approval for multiple executions.

---

Prometheus scrape config#

scrape_configs:
  - job_name: vanzik
    metrics_path: /metrics
    static_configs:
      - targets: ['vanzik.internal:8080']
    # /metrics is unauthenticated; gate at the network layer if needed.
    scrape_interval: 15s
    scrape_timeout: 10s

Behind a reverse proxy with path rewriting, keep /metrics un-prefixed (do not add auth headers — the endpoint is intentionally open).

---

Example alert rules#

groups:
  - name: vanzik
    rules:
      # Sudden spike of denies: either an attack or a policy that's too strict.
      - alert: VanzikDeniedSpike
        expr: rate(vanzik_denied_total[5m]) > 5 * rate(vanzik_denied_total[1h] offset 1h)
        for: 10m
        labels: { severity: warning }
        annotations:
          summary: "DENY rate 5× its 1h baseline"

      # Human approvers falling behind.
      - alert: VanzikPendingApprovalsGrowing
        expr: vanzik_pending_approvals > 100
        for: 15m
        labels: { severity: warning }
        annotations:
          summary: "Approval queue > 100 for 15m; triage is not keeping up"

      # Approval queue saturation — approaching 503s.
      - alert: VanzikApprovalQueueCritical
        expr: vanzik_pending_approvals > 9000
        for: 2m
        labels: { severity: critical }
        annotations:
          summary: "Approval queue near MaxPendingApprovals=10000"

      # Notifier events are being dropped — Slack/webhook probably slow.
      - alert: VanzikNotifierDrops
        expr: rate(vanzik_notify_events_dropped_total[5m]) > 0
        for: 10m
        labels: { severity: warning }
        annotations:
          summary: "Notifier {{ $labels.notifier }} dropping events: {{ $labels.reason }}"

      # p99 policy eval > 50 ms sustained — policy is heavy or a reload is contending.
      - alert: VanzikSlowPolicyEval
        expr: histogram_quantile(0.99, sum by (le) (rate(vanzik_policy_eval_duration_ms_bucket[5m]))) > 50
        for: 10m
        labels: { severity: warning }
        annotations:
          summary: "p99 Engine.Check > 50 ms"

      # Audit replay on startup is becoming the slow path.
      - alert: VanzikSlowAuditReplay
        expr: vanzik_audit_replay_duration_seconds > 30
        labels: { severity: warning }
        annotations:
          summary: "Startup audit replay > 30s; consider rotating the audit log"

      # Agents bypassing per-instance rate limits via multi-replica fanout.
      - alert: VanzikRateLimitBucketEviction
        expr: rate(vanzik_ratelimit_bucket_evictions_total[5m]) > 1
        for: 15m
        labels: { severity: info }
        annotations:
          summary: "Rate-limit buckets evicting in scope {{ $labels.scope }}"

---

/api/stats vs /metrics#

Example /api/stats response:

{
  "total": 12034,
  "allowed": 10998,
  "denied": 842,
  "approvals": 194
}

For the pending-approval count, use the length of /api/pending.

Do not poll /api/stats in a tight loop; use /api/stream (SSE) for real-time updates and fall back to one /api/stats on reconnect.

---

Response-header timings#

Every /v1/check response carries three timing headers (values in ms, 3-decimal precision). Useful for ad-hoc debugging without scraping /metrics.

Total - Policy - Audit approximates the framework overhead (JSON decode/encode, rate limit lookup, notify dispatch enqueue).

---

SSE event types on /api/stream#

The dashboard and any external listener can subscribe. Events are JSON objects sent as:

data: {"type":"check","timestamp":"…","request":{…},"result":{…}}\n\n

Events are unnamed SSE messages (no event: field), so an EventSource receives them through onmessage. Each carries transport (sdk, mcp_gateway, llm_api_proxy, or a coding-agent hook's claude_code, cursor, gemini_cli, codex_cli) and, for tenants other than local, tenant, alongside request and result. approval_required and denied are notifier event types (webhook / Slack / console), not stream events — see APPROVAL_WORKFLOW.md.

Slow consumers drop events; see vanzik_sse_events_dropped_total{reason="slow_consumer"}.

---

Debug-quick-loop checklist#

1. curl -s localhost:8080/health — is the server up?
2. curl -s localhost:8080/metrics | head -80 — counters growing? pending gauge reasonable?
3. curl -D- -X POST localhost:8080/v1/check -d '{"scope":"shell","command":"ls"}' — check the X-Vanzik-*-Ms headers.
4. curl -s -H "Authorization: Bearer $KEY" localhost:8080/v1/audit?limit=20 — see the last 20 decisions.
5. vanzik status — health + pending approvals CLI-style.

---

Related docs#

• docs/OPERATIONS.md — notifier queue sizing, audit rotation, capacity limits.
• docs/DEPLOYMENT.md — exposing /metrics safely behind a reverse proxy.
• docs/TROUBLESHOOTING.md — keyed by symptom; most entries reference a metric to confirm.
• docs/DEPRECATIONS.md — feature keys for vanzik_deprecations_used_total.