LLM API Proxy

The vanzik-llm-proxy binary speaks /v1/chat/completions (OpenAI) and /v1/messages (Anthropic) wire formats and forwards to the real upstream after gating any tool calls the model emits — including tool calls that arrive inside an SSE stream.

For cross-cutting concerns (binary structure, audit transport tag, deployment topologies, fail-mode flag) see docs/PROXY_ARCHITECTURE.md.

---

1. Architecture#

                ┌─────────────────────────────────────┐
                │  agent process (LangChain, raw      │
                │  OpenAI SDK, etc.)                  │
                │  OPENAI_BASE_URL=http://...:8081    │
                └──────────────────┬──────────────────┘
                                   │ HTTP (chat/completions, messages)
                                   │ Authorization: Bearer <user-token>
                                   ▼
                ┌─────────────────────────────────────┐
                │      vanzik-llm-proxy           │
                │  (HTTP server, default 127.0.0.1:8081)│
                │                                      │
                │  ┌─ /v1/chat/completions  (OpenAI)   │
                │  ├─ /v1/completions       (legacy)   │
                │  ├─ /v1/messages          (Anthropic)│
                │  ├─ /v1/embeddings        (passthru) │
                │  ├─ /v1/models            (passthru) │
                │  └─ /healthz                         │
                │                                      │
                │  per-request:                        │
                │  • parse body for tools[]            │
                │  • upstream HTTP client (passthrough │
                │    Authorization)                    │
                │  • streaming parser (OAI deltas /    │
                │    Anthropic events)                 │
                │  • tool-call accumulator → policy    │
                │    check → forward / refuse-rewrite  │
                └──────────────────┬──────────────────┘
                                   │ HTTP/HTTPS (passthrough)
                                   ▼
                ┌─────────────────────────────────────┐
                │   api.openai.com  /  api.anthropic.com │
                │   (or self-hosted OpenAI-compatible)│
                └─────────────────────────────────────┘

       ─── policy-check side channel ───
       proxy → http://127.0.0.1:8080/v1/check  (central guard)

---

2. CLI surface#

vanzik-llm-proxy \
  --listen 127.0.0.1:8081 \
  --upstream-openai https://api.openai.com \
  --upstream-anthropic https://api.anthropic.com \
  --guard-url http://127.0.0.1:8080 \
  --api-key $VANZIK_API_KEY \
  --proxy-api-key $PROXY_AUTH_TOKEN \
  --tenant-id local \
  --fail-mode deny \
  --max-buffer-bytes 1048576 \
  --log-level info

If --listen is non-loopback (0.0.0.0, [::], an external address, or an empty host such as :8081) and --proxy-api-key is unset, the proxy refuses to start — see § 8.1. --api-key is unrelated to this: it is the proxy's credential for the central server's /v1/check.

---

3. Forwarding model#

3.1 Headers#

The proxy is dumb on auth: it forwards Authorization, x-api-key (Anthropic), OpenAI-Organization, OpenAI-Project, and all other request headers to the upstream verbatim — except hop-by-hop headers, the proxy's own X-Vanzik-Proxy-Auth (§ 8.1) and X-Vanzik-Approval-ID (§ 11.3), which are dropped. The proxy never reads the user's bearer token; the upstream is responsible for auth. The Authorization header for the central guard server is a separate concern (see § 3.2) — the proxy adds it on the /v1/check side channel only, never on the upstream-forwarded request.

User-Agent is rewritten to start with vanzik-llm-proxy/<version>, followed by the client's original User-Agent when it sent one (vanzik-llm-proxy/2.0.0 openai-python/1.x), so server-side logs at OpenAI/Anthropic can identify proxied traffic. Spec-compliant: User-Agent is mutable per RFC 7231.

The Host header is rewritten to the upstream's authority. Cookie headers (rare for LLM APIs) are forwarded unchanged.

Three request headers also name who is calling. The proxy copies them into each tool call's /v1/check request (and forwards them upstream like any other header):

3.2 Body handling#

Two paths:

Non-streaming (stream: false or absent)#

1. Read full body into memory. Cap: --max-buffer-bytes (1 MiB default). Larger requests get HTTP 413.
2. Parse JSON to inspect tools (OpenAI) / tools (Anthropic) for policy-relevant context. The request itself is not gated — only the model's tool-call response is gated.
3. Forward the original bytes (not re-encoded JSON — preserves field order, whitespace, and any caller-specific quirks the upstream might key on).
4. Stream the upstream response back to the client. If the response contains tool_calls (OpenAI) or tool_use content blocks (Anthropic), gate them before forwarding the response body. See § 5.

Streaming (stream: true or Accept: text/event-stream)#

1. Forward request body (capped, but typically small).
2. Open the upstream connection with Accept: text/event-stream.
3. Read the upstream SSE response chunk-by-chunk via the streaming parser (§ 5).
4. Pause and gate any tool calls before flushing the corresponding deltas to the client.

3.3 Pass-through endpoints#

/v1/embeddings and /v1/models are forwarded as-is, with no policy check and no audit entry — there are no tool calls in the response, so nothing reaches /v1/check. This is honest about coverage: the proxy is for tool-call gating, not for blocking the agent from talking to the model.

---

4. Tool-call → scope mapping#

The mapping table is shipped with the binary and overridable via policy YAML, mirroring the MCP gateway approach (§ 4.4 of MCP_GATEWAY.md).

4.1 Built-in defaults#

DefaultLLMToolScopeMap in pkg/llmproxy/scope_map.go is a []policy.ToolScopeMapping{Pattern, Scope} list: a tool name (exact, case-sensitive, or a prefix_* glob) and the scope it gates under. It has 47 entries:

The capitalised names are Claude Code's built-in tools. A mapping carries no argument names: which arguments feed the check is fixed per scope (pkg/llmproxy/gate.go):

4.2 Policy-YAML override#

tool_scope_map is a top-level list in the policy YAML (same level as rules:) — there is no separate llm: section; the LLM proxy and the MCP gateway share the one key (see POLICY_REFERENCE.md):

tool_scope_map:
  - pattern: custom_db_query
    scope: network
  - pattern: deploy_lambda
    scope: shell
  - pattern: "*_secret"
    scope: data

Merge: policy entries before built-ins, first match wins. Same semantics as the MCP gateway.

4.3 Unmapped tools#

A tool name that matches no built-in or policy entry is dispatched as scope: "unmapped" with the tool name in command and each top-level argument as a redacted meta["arg_<key>"] entry.

Operators MUST configure an unmapped scope rule to control behaviour:

- scope: unmapped
  deny:
    - pattern: "*"     # default-deny on anything we don't recognise

If no unmapped rule exists, the engine's standard fall-through default-deny applies — fail-closed by design. A tool the policy author has not seen is, by Vanzik's policy contract, denied.

---

5. Streaming pause / resume / rewrite — the hard one#

When the response is streaming and the model is calling tools, the proxy must:

1. Read the SSE stream from upstream chunk by chunk.
2. Parse tool-call deltas (different shape per provider — see § 5.1 and § 5.2).
3. Accumulate until a tool call is complete.
4. Pause the stream to the client (do not flush partial deltas yet — buffer them).
5. Run Engine.Check on the assembled tool call.
6. ALLOW → flush buffered deltas to the client and resume the stream.
7. DENY → rewrite the buffered deltas as a synthetic tool-result message containing Vanzik's denial; resume the stream with the rewrite.
8. REQUIRE_APPROVAL → similar to DENY but the synthetic message includes the approval URL.

5.1 OpenAI streaming wire format#

Verified shape (cross-checked against the OpenAI Python SDK source tree and live captures; reference docs at https://platform.openai.com/docs/api-reference/chat/streaming and https://platform.openai.com/docs/guides/streaming-responses.)

SSE-formatted lines: data: <json>\n\n. Final delimiter: data: [DONE]\n\n.

A streaming chat-completion delta with tool calls:

data: {"id":"chatcmpl-123","object":"chat.completion.chunk","created":1730000000,"model":"gpt-4","choices":[{"index":0,"delta":{"role":"assistant","tool_calls":[{"index":0,"id":"call_abc","type":"function","function":{"name":"read_file","arguments":""}}]},"finish_reason":null}]}

data: {"choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"{\"path"}}]}}]}

data: {"choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"\":\"/tmp/x"}}]}}]}

data: {"choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"\"}"}}]}}]}

data: {"choices":[{"index":0,"delta":{},"finish_reason":"tool_calls"}]}

data: [DONE]

Critical observations:

• Multiple tool calls in one assistant turn arrive interleaved by tool_calls[i].index. The accumulator MUST key by index, not by delta arrival order.
• tool_calls[i].id may appear only on the first delta for that index. Subsequent deltas for the same index have only function fields.
• function.name may be split across deltas in principle; in practice it always arrives whole on the first delta. The accumulator handles fragmentation for both name and arguments defensively.
• arguments is a JSON-encoded string in the wire format (yes — the field is a string, not an object). Concatenating fragments and parsing the result yields the actual argument object.
• finish_reason: "tool_calls" on a delta with empty delta signals the assistant turn is complete. This is the gate trigger — every accumulated tool call must be policy-checked before we forward the next delta.

5.2 Anthropic streaming wire format#

Verified shape (against https://platform.claude.com/docs/en/api/messages, fetched 2026-05-06.)

SSE event types arrive in this order for a single message that calls a tool:

event: message_start
data: { "type": "message_start", "message": { ... } }

event: content_block_start
data: { "type": "content_block_start", "index": 0,
        "content_block": { "type": "tool_use",
                           "id": "toolu_01D7FLrfh4GYq7yT1ULFeyMV",
                           "name": "read_file",
                           "input": {} } }

event: content_block_delta
data: { "type": "content_block_delta", "index": 0,
        "delta": { "type": "input_json_delta",
                   "partial_json": "{\"path\": \"/tmp" } }

event: content_block_delta
data: { "type": "content_block_delta", "index": 0,
        "delta": { "type": "input_json_delta",
                   "partial_json": "/x\"}" } }

event: content_block_stop
data: { "type": "content_block_stop", "index": 0 }

event: message_delta
data: { "type": "message_delta", "delta": { "stop_reason": "tool_use" } }

event: message_stop
data: { "type": "message_stop" }

Critical observations:

• Each content block has its own index (0-based). Multiple tool_use blocks in one message stream get distinct indices.
• content_block_start carries id, name, and an empty input: {}.
• content_block_delta with delta.type == "input_json_delta" carries a partial_json string. Concatenate ALL partial_json for a given block index, parse the result as JSON.
• content_block_stop (with the matching index) signals the tool block is fully assembled. This is the gate trigger for that block. Anthropic gates per block, not per assistant message — this is the key shape difference vs OpenAI.
• message_delta { stop_reason: "tool_use" } is informational; gating has already happened.

5.3 Pause/resume mechanism (both providers)#

Per request, the proxy holds the tool call being assembled and the raw upstream events held back until it is gated, capped by --max-buffer-bytes. The pseudocode below shows the logic, not the code.

Algorithm (OpenAI):

for each SSE line from upstream:
  parse(line) -> delta
  append raw line bytes to bufferedBytes
  bufferedSize += len(line)
  if bufferedSize > maxBufferBytes:
    abort: write a synthetic refusal block to client and return
  for each delta.tool_calls[i]:
    accumulator[i].appendName(delta.tool_calls[i].function.name)
    accumulator[i].appendArgs(delta.tool_calls[i].function.arguments)
    accumulator[i].id = first non-empty delta id observed for this index
  if delta.finish_reason == "tool_calls":
    // GATE TRIGGER — assistant turn complete.
    decisions := for each accumulated tool_call: guard.Check(...)
    if all ALLOW:
      flush bufferedBytes to client                          // resume
      bufferedBytes = nil; bufferedSize = 0
      accumulator = {}
    else:
      // Don't flush the buffered deltas. Synthesize refusal:
      write synthetic chunk to client (see § 5.4)
      bufferedBytes = nil; bufferedSize = 0
      accumulator = {}
      // Continue reading upstream to drain it; discard remaining bytes
      // unless there are no more tool calls coming (assistant turn done).
  elif delta has no tool_calls AND no buffered tool calls:
    // Plain content delta; flush immediately to keep TTFT low.
    flush this delta to client
    bufferedBytes = ...  // reset to empty for non-tool deltas

Algorithm (Anthropic): identical structure, but:

• Gate per content_block_stop (with matching index whose content_block_start was a tool_use), not per finish_reason.
• Plain text blocks pass through immediately. Only tool_use blocks buffer.
• The buffer scope is the bytes for that one block, from content_block_start through content_block_stop. Other content blocks (text, image) flow around the buffered tool block in arrival order — but if a text block arrives after an unflushed tool_use block on the same message, we flush the tool block first (preserving order).

5.4 Synthetic refusal shape#

OpenAI#

When DENY (or REQUIRE_APPROVAL with a URL):

data: {"choices":[{"delta":{"content":"Vanzik denied this action.\n\nReason: <reason>\nRule: <rule>","role":"assistant"},"finish_reason":"stop","index":0}],"created":0,"id":"vanzik-refusal","object":"chat.completion.chunk"}

data: [DONE]

The agent SDK's normal text-completion handling engages — the model appears to have given up and returned plain text. No SDK changes required.

For REQUIRE_APPROVAL the content is:

Vanzik paused this action pending human approval.

Reason: <reason>
Rule: <rule>

Approval URL: <approval_url>
Approval ID: ap_<hex> (send it in the X-Vanzik-Approval-ID header on retry to resume after approval)

Wait for human approval, then retry the same call.

§ 11.3 covers the retry.

Design note on tool role injection. Injecting a synthetic role: "tool" message with a fake tool_call_id instead has two problems: (a) role: "tool" messages are valid in request turns, not response deltas — the OpenAI response schema only emits role: "assistant"; (b) the agent's SDK would expect a paired tool_calls[*].id it had already buffered, which the proxy did NOT flush. Synthesising assistant text is simpler and correct.

Anthropic#

For a denied tool_use block, the proxy emits a synthetic text block in place:

event: content_block_start
data: { "type": "content_block_start", "index": <same>,
        "content_block": { "type": "text", "text": "" } }

event: content_block_delta
data: { "type": "content_block_delta", "index": <same>,
        "delta": { "type": "text_delta",
                   "text": "Vanzik denied this action.\n\nReason: <reason>\nRule: <rule>" } }

event: content_block_stop
data: { "type": "content_block_stop", "index": <same> }

event: message_delta
data: { "type": "message_delta", "delta": { "stop_reason": "end_turn" },
        "usage": { "output_tokens": 0 } }

event: message_stop
data: { "type": "message_stop" }

Anthropic's content_block index is reusable — emitting a text block at the same index that the tool_use block was buffered for is wire- legal. The proxy stops reading the upstream there and closes the message itself: its message_delta sets stop_reason to end_turn (not tool_use) so the agent doesn't expect a tool result it's not getting, and carries usage, which the Anthropic SDKs' messages.stream() reads on every message_delta (without it they raise AttributeError).

Malformed tool calls fail closed (v1.0)#

A tool call that completes with arguments that are not valid JSON — routine on a max_tokens cutoff mid-arguments, and attacker-inducible — is refused with the same synthetic-refusal shapes above, under the fixed rule deny:llm_api_proxy:malformed_tool_call (stable; alert on it). The refusal is recorded in the central audit trail with the exact DENY the client received (see Audit fidelity below), the buffered malformed bytes are discarded (never forwarded), and the accumulator is reset so any later valid tool call in the same stream is still gated. Malformed non-completion deltas are dropped silently, because corruption that matters resurfaces at completion, where it denies.

Undecodable non-streaming bodies that carry a tool call fail closed#

A non-streaming response is gated by decoding it into the typed response struct and inspecting the tool calls it contains. A body that fails that decode is forwarded verbatim — Vanzik never rewrites an already-corrupt wire — with one exception.

Go's decoder is stricter than a client SDK's in one specific way: it aborts on a type-mismatched field the struct declares, such as "created": "1730000000" where the spec says integer. A Python or TypeScript SDK parsing the same bytes ignores the odd field and executes the tool_calls sitting beside it. So "Vanzik cannot decode it" does not imply "the client cannot use it".

Before falling back to passthrough the proxy asks the narrower question — does this body carry a tool call at all? — using a probe that constrains only the path to tool_calls / tool_use and nothing else. If one is present the response is refused under the fixed rule deny:llm_api_proxy:undecodable_tool_call. If none is present the body passes through unchanged.

Where to see it. The refusal is recorded in the central audit trail with the DENY the client actually received, via the same /v1/audit ingest path the malformed-tool-call refusal uses — so it arrives tagged transport: llm_api_proxy under that stable rule string, which is the signal to alert on. The proxy also increments vanzik_llmproxy_undecodable_tool_call_total{provider}, but it counts in the proxy process, which serves no /metrics (see OBSERVABILITY.md). The central server's /metrics does list all six vanzik_llmproxy_* series (the two binaries share one metrics registry), but nothing there increments them, so they always read 0. Don't alert on any of them; alert on the audit rule.

Two details worth knowing:

• The refusal is not subject to --fail-mode allow, matching the duplicate-JSON-key deny. --fail-mode governs an unreachable guard; here the guard is healthy and simply cannot see what the client will execute, so honouring allow would reopen the bypass.
• A JSON null or an unknown extra field is not a type error — Go ignores both — so neither diverts a response down this path. Only a wrong-typed value does. A provider adding fields never trips it.

A non-zero rate on that counter usually means an OpenAI-compatible shim (vLLM, Ollama, LiteLLM, a cloud gateway) is stringifying a numeric field. Worth fixing at the source: each one is a call your agent did not get to make. Streaming responses are unaffected — they are gated by the accumulator, which has its own fail-closed defences.

Ungateable streams fail closed (v1.0)#

Some streams are not corrupt so much as unpartitionable: the bytes arrive, but there is no way to bind them to a tool call the gate can evaluate. Gating requires knowing which fragments belong to which call, and when one of those relationships is missing the firewall cannot honestly say what the client is about to execute. Those streams are refused rather than forwarded, each under its own stable rule string:

All three discard the buffered bytes and stop reading upstream. None is subject to --fail-mode allow: the guard is healthy, it simply cannot see what the client will execute.

All three are recorded in the central audit trail, as are the two buffer-cap refusals (deny:llm_api_proxy:buffer_overflow, emitted both for one oversized SSE event and for cumulative arguments past --max-buffer-bytes). Each entry carries the rule, tenant, agent and provider, plus whatever tool identity the parser observed before refusing — often none, since a stream refused because its bytes cannot be bound to a call is by definition one where no call assembled.

n > 1 with tool calls is the only capability this costs, and it can't be gated correctly anyway — the merged arguments would be wrong. Every mainstream agent framework sends n = 1, which is untouched.

A stream that ends mid-tool-call is gated, not dropped#

When upstream reaches EOF — or sends [DONE] — while a tool call is still buffered, the cycle is completed, not abandoned:

• Arguments that parse are gated normally and, on ALLOW, replayed byte-identical.
• Arguments truncated by the cut do not parse, so the cycle lands on deny:llm_api_proxy:malformed_tool_call above: denied and audited.

The buffered bytes are never flushed unexamined — that would hand the client the ungated call this whole mechanism exists to gate, and discarding them would leave the client an empty 200 with no audit entry. [DONE] is treated as the end of stream in its own right, so a gateway that holds the connection open after it does not delay the verdict.

Audit fidelity#

For a proxy-manufactured refusal (the malformed-completion DENY above), the engine verdict a plain /v1/check would log is the wrong verdict — the engine might well ALLOW the garbage-projected request while the client got DENY, so the trail would read ALLOW for a refused call. To avoid that, the proxy POSTs the client-visible DENY to the central server's POST /v1/audit ingest, which appends it verbatim (no re-evaluation).

POST /v1/audit is auth-gated. If the central server runs with --api-key, give this proxy --api-key too — otherwise the audit POST returns 401. On any /v1/audit failure the proxy does not drop the entry: it best-effort falls back to the /v1/check audit path, which still writes a transport-tagged entry (carrying the engine verdict, not the DENY). So fidelity degrades to the lower-fidelity path without --api-key, but audit coverage never does — a forced refusal is always recorded one way or the other.

5.5 Byte-identity invariant on ALLOW#

Critical correctness property: in the ALLOW path, the bytes delivered to the client must be byte-identical to what the upstream sent. The proxy buffers raw bytes (not re-marshalled JSON) for exactly this reason. Re-marshalling could re-order fields, normalise whitespace, or change Unicode escape forms — any of which can break client SDKs that rely on byte-level invariants (some agents hash the delta stream for reproducibility).

CI replays captured upstream streams through the proxy with an ALLOW policy and asserts the byte diff is empty.

5.6 Pause/resume mechanism — buffer bound#

--max-buffer-bytes (default 1 MiB) caps the bytes buffered while gating a single tool call. Rationale:

• Typical tool argument JSON is 50–500 bytes.
• Pathological cases (large file contents passed as a tool argument, e.g., a write_file with a multi-megabyte payload) hit this cap.
• Above the cap, the proxy emits a synthetic refusal under the rule deny:llm_api_proxy:buffer_overflow. Its text starts "Vanzik refused this tool call: its accumulated arguments exceeded the gating buffer cap.", gives the reason "tool call arguments exceed gating buffer", and names the --max-buffer-bytes flag. The proxy counts it in vanzik_llmproxy_buffer_overflow_total (process-local, like every vanzik_llmproxy_* series; see Undecodable non-streaming bodies in § 5.4); operators can raise the limit if their use case needs it.

Why 1 MiB:

• Matches the central server's MaxRequestBodySize for /v1/check.
• Captures 99% of real-world tool calls.
• Keeps the proxy memory bounded under load (N concurrent streams × 1 MiB worst case).

---

6. Concurrency model#

• Each request has its own buffer and parser state; no mutable state is shared between requests. The upstream and /v1/check HTTP clients (and their connection pools) are shared.
• When the client closes the connection, the proxy cancels both the upstream read and any in-flight /v1/check call, then drops buffers.
• An SSE event is read up to its \n\n delimiter with no fixed line limit other than --max-buffer-bytes.

---

7. Audit shape#

One audit entry per tool call (not per stream, not per delta).

{
  "schema_version": "v1",
  "transport": "llm_api_proxy",
  "timestamp": "2026-05-06T12:34:56.789Z",
  "session_id": "<from x-session-id header; empty when absent>",
  "agent_id": "<from x-agent-id header, or 'llm-proxy'>",
  "request": {
    "scope": "filesystem",
    "parent_session_id": "<from x-vanzik-parent-session header; omitted when absent>",
    "command": "read_file /tmp/x",
    "path": "/tmp/x",
    "action": "read",
    "meta": {
      "arg_path": "/tmp/x",
      "mapped_scope": "filesystem",
      "model": "gpt-4",
      "provider": "openai",
      "stream": "true",
      "tool_call_id": "call_abc",
      "tool_name": "read_file",
      "transport": "llm_api_proxy"
    }
  },
  "result": {
    "decision": "ALLOW",
    "matched_rule": "allow:filesystem:tmp",
    "reason": "matched allow rule for /tmp/**"
  },
  "duration_ms": 4
}

Each top-level tool argument becomes a meta["arg_<key>"] entry, redacted via pkg/notify.DefaultRedactor; command is redacted the same way. stream is present only for streaming responses, and approval_id is added when the request carried X-Vanzik-Approval-ID.

For an unmapped scope: request.scope = "unmapped", request.command = "<tool_name>", and the arguments are the same redacted arg_<key> entries.

Dashboard chip: transport: "llm_api_proxy" (color: purple — distinct from mcp_gateway blue and sdk green).

---

8. Auth posture#

8.1 Inbound (agent → proxy)#

By default, the proxy enforces no auth of its own and binds to loopback (127.0.0.1). Treat this like the central guard server's no-API-key fallback: localhost is the implicit trust boundary.

If --proxy-api-key is set, every API request (everything except GET /healthz) must carry the key in the X-Vanzik-Proxy-Auth header — as <key> or Bearer <key>. The key is compared in constant time; a missing or wrong key gets 401 with {"error":{"message":"unauthorized: X-Vanzik-Proxy-Auth missing or invalid","type":"vanzik_error"}}.

It is a separate header so it never collides with the caller's own provider credential: Authorization / x-api-key (the OpenAI / Anthropic key) pass through to the upstream unchanged, while X-Vanzik-Proxy-Auth is stripped before forwarding. The proxy does not re-key calls — the upstream sees the caller's credential.

If --listen is non-loopback and --proxy-api-key is unset, the proxy refuses to start (it prints the reason to stderr and exits with status 2). Non-loopback means 0.0.0.0, [::], an external address, or an empty host (:8081), which binds every interface; only 127.x.x.x, [::1] and localhost count as loopback. Avoids accidental internet-exposed proxies.

8.2 Outbound to central guard#

The proxy uses --api-key as Authorization: Bearer <key> on every /v1/check call. Same shape as the SDKs.

8.3 Rate limiting#

The proxy itself does not rate-limit. The central guard server's existing per-scope per-agent rate limiter applies — synthetic DENY with Rule="deny:ratelimit:<scope>" flows back to the proxy via the /v1/check response and is rewritten as a synthetic refusal in the stream. Operators can tune the limits in policy YAML (rate_limit.max_requests / rate_limit.window).

---

9. Test strategy#

The tests use scripted fake upstreams (Go httptest in pkg/llmproxy, with captured streams in pkg/llmproxy/testdata; Python mocks in plugins/python/tests/integration), so CI needs no provider key.

---

10. Currently out of scope#

• Google Gemini wire format (different shape again, uses functionCall content parts in a streaming JSON-line format).
• Per-model cost gating beyond the existing cost-scope rules — the proxy uses scope: cost against the agent-supplied est_cost, not actual model token usage.
• Tool-result inspection — gating the agent's response to a tool's output (e.g., refuse to act on data containing PII patterns). The proxy gates request-side only.
• Caching of policy decisions for repeat tool calls inside one stream. The proxy makes one /v1/check call per tool call.
• The OpenAI Assistants API (different wire format, different streaming envelope). Supported endpoints are /v1/chat/completions and /v1/messages only.

---

11. Client integration#

Vanzik ships working examples for the four most common ways agents talk to OpenAI / Anthropic: the raw OpenAI and Anthropic Python SDKs, LangChain (langchain-openai), and CrewAI (LiteLLM under the hood). Each example is a runnable Python script with a paired .md walkthrough. The fastest end-to-end is docs/QUICKSTART_LLM_PROXY.md. The examples need Vanzik 2.0 or later: older proxies lack the approval retry header (X-Vanzik-Approval-ID, § 11.3), and their Anthropic refusal streams crash messages.stream().

11.1 Per-SDK base-URL summary#

11.2 Verifying the integration#

Once the proxy is running and your code points at it:

1. The proxy logs a startup line. Stderr: vanzik-llm-proxy <version> listening on 127.0.0.1:8081 (...). If you don't see this, the proxy didn't start — check --listen and policy paths.
2. The first SDK call appears on the dashboard. Send any tool-using prompt — within a second the dashboard at http://127.0.0.1:8080/dashboard shows an event with transport=llm_api_proxy (purple chip), the resolved scope, and the tool name as command. If nothing appears, the proxy isn't reaching the central server's /v1/check — check --guard-url and --api-key.
3. An ALLOW rule round-trips. With the bundled configs/default.yaml, ask the model to ls /tmp via a bash-named tool. The dashboard logs ALLOW; the SDK receives the original streamed tool-call deltas byte-identically.
4. A DENY rule rewrites the stream. The bundled policy denies dd if=/dev/* under scope: shell. Prompt the model to run dd if=/dev/zero of=/dev/sda. The dashboard logs DENY; your code receives a synthetic assistant text starting with Vanzik denied this action. instead of any tool-call deltas.

If actions never reach Vanzik, the SDK is probably still talking to the real upstream — verify the env var is set for the process that runs your script (not just the parent shell), and that the SDK constructor isn't overriding base_url from elsewhere.

11.3 Common gotchas#

Both binaries must be running. The proxy alone gates nothing — every tool call calls back to the central server's /v1/check, and --fail-mode decides what happens when that server is down (full table: PROXY_ARCHITECTURE.md § 6.1).

Three API keys, three purposes. Provider key (forwarded verbatim, never read), VANZIK_API_KEY (the /v1/check side-channel bearer), and the optional inbound --proxy-api-key (X-Vanzik-Proxy-Auth) — the header split is spelled out in § 3.1 and § 8.1.

Anthropic's path convention. ANTHROPIC_BASE_URL should be http://127.0.0.1:8081 — no /v1 suffix. The Anthropic SDK appends /v1/messages itself. Setting http://127.0.0.1:8081/v1 produces double-prefixed paths and 404s. This is the most common configuration mistake; double-check the env var when debugging.

Resuming after an approval takes the X-Vanzik-Approval-ID header. When Vanzik returns REQUIRE_APPROVAL, the proxy emits a synthetic refusal with the approval ID and URL embedded (§ 5.4). To resume:

1. Approve it on the dashboard (or POST /v1/approve/<id>).
2. Send the request again with the header X-Vanzik-Approval-ID: <id> — for the Python SDKs, extra_headers={"X-Vanzik-Approval-ID": approval_id}.
3. When the model emits the same tool call, the proxy passes the id to /v1/check as approval_id, Vanzik honors the approval, and the tool call reaches your code as ALLOW.

The server's replay checks still apply: the id unlocks only the action that was approved (same agent, scope, command, path, domain, URL and action), only once, and only within the approval-validity window. Anything else in the retried response is evaluated fresh, so a different command is paused or denied as usual. Without the header a retry gets a new REQUIRE_APPROVAL with a new id. The proxy reads the header for /v1/check only and does not forward it upstream. (The MCP Gateway carries the same id in _meta.dev.vanzik/approval_id; see docs/APPROVAL_WORKFLOW.md.)

Streaming vs non-streaming both work. Streaming goes through the pause/resume/rewrite gater (§ 5); non-streaming responses are buffered in full (capped by --max-buffer-bytes) and gated before forwarding. Concurrent calls are safe — per-request parser state is isolated (§ 6).

Custom tool names need a tool_scope_map: entry. Vanzik ships a default mapping for 47 common tool names (§ 4.1; see pkg/llmproxy/scope_map.go). Anything else dispatches as scope: unmapped, which the engine treats as default-deny unless an unmapped rule exists. For custom tools, add an entry under your policy's tool_scope_map: section (see docs/POLICY_REFERENCE.md § "LLM API Proxy tool scope mapping") — the proxy hot-reloads via the same mechanism the central server uses.

Self-hosted OpenAI-compatible upstreams. The proxy's --upstream-openai flag accepts any OpenAI-compatible base URL (LM Studio, vLLM, llama.cpp's --api-key server, Together AI, Anyscale, etc.). The streaming wire format is the same; the /v1/chat/completions parser handles them all. For Azure OpenAI, note the path convention is different (/openai/v1/... not /v1/...) — point --upstream-openai at the Azure base URL including the resource path.

---

12. References#

• OpenAI Python SDK (verified 2026-05-05):
  • https://github.com/openai/openai-python — README documents base_url= and OPENAI_BASE_URL. Latest release v2.35.1.
  • https://platform.openai.com/docs/api-reference/chat/create
  • https://platform.openai.com/docs/api-reference/chat/streaming. The streaming wire shape in § 5.1 is cross-checked against the OpenAI Python SDK source tree.
• Anthropic Python SDK (verified 2026-05-05):
  • https://github.com/anthropics/anthropic-sdk-python — README + src/anthropic/_client.py document base_url= and ANTHROPIC_BASE_URL. Default base URL is https://api.anthropic.com with no /v1 suffix; SDK appends /v1/messages etc. itself.
  • https://platform.claude.com/docs/en/api/client-sdks
  • https://platform.claude.com/docs/en/api/messages
• LangChain (verified 2026-05-05):
  • https://docs.langchain.com/oss/python/integrations/chat/openai — langchain_openai.ChatOpenAI base_url= parameter.
  • https://reference.langchain.com/python/langchain-anthropic/chat_models/ChatAnthropic — langchain_anthropic.ChatAnthropic base_url= parameter.
• CrewAI (verified 2026-05-05):
  • https://docs.crewai.com/en/learn/llm-connections — LLM(...) base_url= parameter; OPENAI_API_BASE env var honored.
• Architectural reference (no feature copying):
  • go-mitmproxy stream addons: https://github.com/lqqyt2423/go-mitmproxy
  • LiteLLM proxy structure: https://github.com/BerriAI/litellm
• Cross-cutting design: docs/PROXY_ARCHITECTURE.md.