Skip to content
browse the docs

Troubleshooting

vanzik v2.0updated view on github

Symptom-first reference. If something is wrong, skim the headers below for the one that matches what you are seeing.

Agents get connection refused / connect: timed out#

Likely cause: --api-key is not set, so the server is bound to 127.0.0.1:<port> only. Agents running on any other host cannot reach it.

Confirm: the server log on startup will show:

INFO: binding to 127.0.0.1:8080 (localhost only) — set --api-key to listen on all interfaces

Fix: set a Bearer token.

export VANZIK_API_KEY="$(openssl rand -hex 32)"
vanzik server --api-key "$VANZIK_API_KEY" ...

Details: docs/DEPLOYMENT.md §1.

Dashboard login loops back to the login page#

Likely cause: you are running behind an HTTPS-terminating reverse proxy, but Vanzik is issuing session cookies without Secure. The browser sets them, then refuses to send them back on the next HTTPS request.

Confirm: in browser devtools (Application → Cookies), vanzik_session is present but Secure column is blank.

Fix: add --tls-terminated-upstream.

vanzik server --tls-terminated-upstream ...

Details: docs/DEPLOYMENT.md §2a.

Slack / webhook approval links point at localhost#

Likely cause: --base-url is not set, so approval_url falls back to http://localhost:<port>.

Fix:

vanzik server --base-url https://guard.example.com ...

Policy edits do not take effect#

Likely cause (a): the server loaded a different file than the one you edited. The startup log names it: Loaded policy: <name> from <path> …. Without --policy, the server picks the file itself (see CLI.md). Reloading is always on; --watch only adds a Policy reloaded: … log line per reload, which shows whether an edit was picked up:

vanzik server --policy ./my-policy.yaml --watch ...

Likely cause (b): the file's mtime did not change. The watcher reacts to fsnotify events on the file's directory, and falls back to polling os.Stat(path).ModTime() every 2s (pkg/policy/watcher.go DefaultPollInterval) only when fsnotify can't start. Either way a reload happens only when the mtime has advanced. If you edited through a tool that preserves mtime, touch the file:

touch configs/default.yaml

Likely cause (c): your edit is syntactically invalid. Parse errors are logged but do not halt the watcher — the old policy keeps serving. Check the server log for Policy reload failed: and run vanzik validate --policy configs/default.yaml to pinpoint the error.

Rate limit works locally but gets bypassed in production#

Likely cause: you are running more than one Vanzik replica on the default SQLite store. Rate-limit buckets are in-memory per instance (pkg/ratelimit/ratelimit.go); an agent can burst past the cap by hitting different pods / VMs round-robin.

Mitigations:

  • Share state across replicas: --store-dsn postgres://… plus a distinct --node-id per replica. Note the shared mode is bounded-overshoot — brief bursts can still exceed the cap by ≈ reconcile-interval × peak rate per extra replica before the nodes converge.
  • Divide per-scope max_requests by the replica count when writing the policy.
  • Pin agents to a single Vanzik instance via session affinity.
  • Keep replicas: 1 and scale vertically.

Details: docs/OPERATIONS.md.

Approvals disappeared after a restart#

Cause: the queue is persistent by default, so this should only happen if (a) you run --persist=false (in-memory only), (b) the process can't write its --data-dir (check the startup log for store errors), or (c) the entries were created inside the final ≥1 s store-sync window before a hard crash.

Mitigations:

  • Run with --persist (the default) and verify vanzik.db exists and grows in --data-dir.
  • Set SDK wait_for_approval(timeout=...) to a realistic human SLA — approvals survive a restart, but nobody can resolve them while the process is down.
  • For mission-critical flows, have the calling agent checkpoint its state so it can re-issue the check and get a fresh approval ID.

/metrics or /api/stats are slow / inaccurate right after startup#

Cause: on boot, Vanzik replays the audit log through metrics.IncDecision to seed decision counters so /metrics survives restarts with accurate totals. The replay resumes from a saved byte-offset checkpoint, so normally only entries written since the last checkpoint are read; the whole log is replayed only on first boot, after a reset, or when the checkpoint can't be read (the log says replaying full log). For a multi-GB audit.jsonl, that full scan can take seconds to minutes.

Fix:

  • Keep rotation enabled (it is on by default: --audit-max-size-mb 100, 5 backups) so the active file stays bounded.
  • Ship historical audit data to an external aggregator and truncate the local file during scheduled maintenance.

Details: docs/OPERATIONS.md.

Webhook / Slack events are missing#

Cause: the notification dispatcher has a bounded queue (DefaultQueueSize = 256). If notifiers are slow or the queue is full, Dispatcher.Send drops the event and increments vanzik_notify_events_dropped_total{notifier, reason}.

Confirm:

curl -s http://127.0.0.1:8080/metrics | grep notify_events_dropped_total

Mitigations:

  • Lower notifier timeouts in the policy file (notifications.dispatch_timeout, or a target's own timeout), so a slow or flaky endpoint holds a worker for less time. Longer timeouts keep workers busy and make drops more likely.
  • Reduce notifier fanout (remove unused targets).
  • Shard to multiple webhook endpoints if a single Slack channel is a bottleneck.
  • Watch vanzik_notify_queue_depth (one shared queue, so it has no notifier label) — steady growth means the workers cannot drain.

Python SDK always returns DENY in tests#

Cause: the Python SDK defaults to fail-closed on any network error (urllib.error.URLError, OSError, or a non-JSON response body → synthetic DENY with reason "Vanzik unreachable (deny): ..."). If your test environment cannot reach the proxy URL, every call fails closed.

Confirm: inspect result.reason — it will contain "Vanzik unreachable".

Fixes:

  • Point the test Guard(base_url=...) at a running test server, or stub the HTTP layer with the standard-library stub in SDK_PYTHON.md ("Option B"). responses can't intercept the SDK's calls: it only patches requests, and the SDK uses urllib.
  • For tests that need fail-open behavior, pass fail_mode="allow" explicitly. Do not flip this in production unless your threat model treats Vanzik as advisory — the "deny" default is what keeps agents safe when the proxy is down.

A chained or piped shell command is denied after upgrading to 1.2#

Symptom: a command that 1.1.x allowed, such as cat README.md | head -5 or ls | grep foo, is now DENY or REQUIRE_APPROVAL, and the reason names one part of it: No matching allow rule (default deny) (command 2 of 2: "head -5").

Cause: since v1.2.0, a shell command with shell syntax (;, &&, ||, |, &, a line break, $( ), backticks, a redirection) is split into the commands a shell would run, and every one of them must be allowed. In 1.1.x the whole line matched the first command's rule, which is how allow: "ls *" also allowed ls /tmp; rm -rf /. A bare command is its own segment, and ls * needs something after ls, so the ls in ls | grep foo is not allowed by ls *.

Fix: add allow rules for the commands your agents chain (head *, grep *, or ls on its own), or write an allow pattern that contains the operator, which matches command by command:

- scope: shell
  allow:
    - pattern: "head *"
    - pattern: "ls"
    - pattern: "cat * | head *"

A deny:shell:unparseable_command verdict means the command used syntax Vanzik doesn't split (a heredoc, $(( )), unbalanced quotes, or nesting deeper than 8 levels); rewrite the command without it. To see what 1.2.0 would decide for commands your agents already ran, replay them with vanzik check --batch — see MIGRATION.md.

*.foo.com does not match foo.com#

Not a bug. Vanzik uses standard glob semantics: *.foo.com requires at least one character before the dot. Note the opposite surprise for paths: a single * does cross / (api/*/users matches api/v1/internal/users, and /workspace/* matches every file under /workspace), so bound path allows with ** segments instead. See POLICY_REFERENCE § Single-star * crosses /.

Fix: add both patterns explicitly.

allow:
  - domain: "foo.com"
  - domain: "*.foo.com"

Details: docs/POLICY_REFERENCE.md.

Condition has time_window but it is being ignored#

Cause: time_window without require_prior errors out at policy load. LoadFromFile returns the validation error; the server refuses to boot until the policy is fixed. Either remove the orphan time_window or add a require_prior clause. See DEPRECATIONS.md.

Confirm: policy load fails with an error like:

rules[0]("deploy_*").conditions[0]: time_window without require_prior is rejected; pair time_window with require_prior or remove it

Fix: add a require_prior (the pattern/action that must have been allowed within the window), or remove the orphan time_window.

Details: docs/DEPRECATIONS.md and docs/POLICY_REFERENCE.md.

Dashboard shows a red LIVE badge#

Cause: the SSE connection on /api/stream dropped — either the server restarted, the network hiccupped, or your reverse proxy buffered the stream shut (nginx default is proxy_buffering on).

Fixes:

  • Wait a few seconds: the dashboard reconnects by itself, and its badge shows RECONNECTING... meanwhile. If it stays there, the server is down or something between you and it is blocking the stream.
  • If behind nginx, set proxy_buffering off on /api/stream. See docs/DEPLOYMENT.md §2c.

I see 503 Service Unavailable on /v1/check or /auth/login#

Cause: the approval queue or the dashboard session store is full. (/v1/approve and /v1/deny never return 503.)

  • Approval queue, on /v1/check: MaxPendingApprovals = 10000 (pkg/proxy/server.go). When a new approval arrives at capacity, the oldest resolved entry is evicted to make room; if every entry is still pending, the check that needed approval gets 503 with Retry-After: 30.
  • Session store, on /auth/login: MaxSessions = 1024 (hardcoded in pkg/proxy/auth.go). At capacity, expired sessions are swept first; live sessions are never evicted, so if all are live the login gets 503 with Retry-After: 5.

Mitigation: resolve outstanding approvals faster, or log out unused dashboard sessions (or wait for them to expire) to shrink the session table.

CORS preflight fails from a production frontend#

Cause: --allowed-origin is unset, so Vanzik is in permissive-localhost mode — any non-localhost Origin is rejected.

Fix: set the exact frontend origin.

vanzik server --allowed-origin https://app.example.com ...

Details: docs/DEPLOYMENT.md §3.

Docker: audit log is empty after container restart#

Cause: the audit log is written inside the container at /var/lib/vanzik/audit.jsonl. Without a volume mount, the filesystem is destroyed when the container is removed.

Fix:

docker run -d -p 8080:8080 \
  -e VANZIK_API_KEY="$VANZIK_API_KEY" \
  -v vanzik-audit:/var/lib/vanzik \
  vanzik:latest

Keep -e VANZIK_API_KEY: without an API key the server binds 127.0.0.1 inside the container and the published port refuses connections.

MCP Gateway / LLM API Proxy issues (v0.5+)#

MCP client shows zero tools after pointing at vanzik-mcp-gateway#

The gateway started but no --upstream is wired, or the upstream subprocess crashed at boot. Run the gateway in the foreground and watch for info mcpgw: startup: upstream "<name>" failed to spawn: … (a missing launcher such as npx, uvx or docker shows up here). Verify the upstream command runs standalone. Details: MCP_GATEWAY.md.

LLM API Proxy: SDK hangs / unexpected end of stream#

The proxy was killed mid-stream and the buffered tool-call payload was never flushed. On SIGTERM the proxy drains in-flight requests for 10 s (fixed, pkg/llmproxy/server.go), then closes what is left. Give the host at least that long before it kills the process (TimeoutStopSec=15s / terminationGracePeriodSeconds: 15); a longer grace period doesn't extend the 10 s drain, so a stream that runs past it is still cut.

LLM API Proxy: agent sees 502 Bad Gateway#

The proxy couldn't reach the model provider: the response body reads upstream error: … (for example connection refused). Check --upstream-openai / --upstream-anthropic and that the proxy host can reach the provider (DNS, egress firewall, corporate proxy).

An unreachable Vanzik server (--guard-url) does not cause a 502. With --fail-mode=deny (the default) the agent gets a synthetic refusal with HTTP 200 until the server comes back; curl -sf <guard-url>/health confirms reachability, and VANZIK_API_KEY must match on both sides.

Proxy rejects every incoming connection#

It bound to 127.0.0.1. If your agent runs in a separate container or host, the loopback bind blocks it. Change to --listen 0.0.0.0:<port> and set --proxy-api-key: the proxy refuses to start on a non-loopback address without it (exit code 2), and then requires every client to send the key in the X-Vanzik-Proxy-Auth header. See LLM_API_PROXY.md.

Tool calls flow through the LLM API Proxy but never reach the Vanzik server#

The client's OPENAI_BASE_URL is missing the /v1 suffix, or the request is a plain chat completion (intentionally passthrough — no tool call means nothing to gate). Check the central server's audit log (vanzik audit --transport llm_api_proxy) — if nothing lands there while tool calls run, the traffic isn't reaching the proxy. Set OPENAI_BASE_URL=http://127.0.0.1:8081/v1 (include /v1); for Anthropic use ANTHROPIC_BASE_URL=http://127.0.0.1:8081 (no /v1). Details: QUICKSTART_LLM_PROXY.md.

Still stuck?#

  • Run vanzik status — shows /health reachability and the pending queue.
  • Check /metrics on the central server — counters, gauges, histograms, dispatch drops. The LLM API Proxy and the MCP Gateway serve no /metrics; their decisions show up in the central server's metrics and audit log.
  • Response headers on /v1/check: X-Vanzik-Policy-Ms, X-Vanzik-Audit-Ms, X-Vanzik-Total-Ms (timings in ms, 3 decimals).
  • Version skew: all three binaries report their version via vanzik version / vanzik-mcp-gateway --version / vanzik-llm-proxy --version. Mismatch is the most common ops-time bug after a partial upgrade.
  • Open an issue with: versions of all binaries, sanitized policy snippet, and the relevant log lines from both the proxy and the central server. Security-sensitive reports go to hello@lictorate.com — not the issue tracker.