Skip to main content

Documentation Index

Fetch the complete documentation index at: https://to11.ai/docs/llms.txt

Use this file to discover all available pages before exploring further.

​
Gateway Development

This guide covers how to run the Rust gateway on your host machine for faster development iteration, and how to work with the crate workspace.

​
Prerequisites

  • Rust stable toolchain (rustup default stable)
  • Docker and Docker Compose v2+ (for infrastructure services)
  • An API key for at least one LLM provider

​
Run the gateway on your host machine

This flow runs the Rust gateway on your host machine while keeping infrastructure services (ClickHouse, OTel Collector, Grafana, Tempo) in Docker containers. 
# 1. Start infra + generate the local bootstrap outputs
make dev-setup

# 2. Source the local environment
source .env.local

# 3. Run the gateway with cargo
make dev-gateway
The gateway reads docker/gateway/config.dev.toml. The host-run gateway itself listens on http://localhost:4000; read the current API and observability host ports for your worktree from .worktree/runtime.json. See the configuration reference for all available settings.

​
Crate workspace

The gateway is organised as a Cargo workspace with isolated responsibilities:
CratePurpose
gatewayBinary entry point, CLI, benchmarks
gateway-coreAxum router, provider adapters, SSE streaming
gateway-configTOML config loading, runtime telemetry resolution
gateway-securityDetector trait, PII + blocklist + ONNX pipeline
gateway-telemetryOTel init, middleware layers, context propagation
gateway-authAPI key and JWT validation
gateway-cacheSemantic and exact-match caching
gateway-typesShared types and error definitions
gateway-security owns the ort (ONNX Runtime) dependency so that changes to routing, config, or core never trigger ONNX recompilation.

​
Running tests

# Run all gateway tests
cargo test --workspace

# Run a specific crate's tests
cargo test -p gateway-core

# Run with output (useful for debugging)
cargo test -p gateway-core -- --nocapture

​
Building a release binary

cargo build --release -p gateway
The binary is output at target/release/gateway.

​
Environment variables

The gateway uses passthrough authentication — the caller’s Authorization header is forwarded directly to the upstream provider. The gateway itself does not read or store provider API keys. Telemetry can be enabled via environment variables that override the TOML config. See the configuration reference for the full list.