See where every LLM call spends its time.
Spanrail is a self-hosted place for traces, evaluations and prompts. Its Go gateway measures DNS, connect, TLS and first byte for every call that passes through it, and draws each one to scale.
The life of a call
One call, step by step. The bench below lays time out along a ruler: the carriage is your request, and each block is one phase of the round trip. Step through it, or switch to a reused connection to see which phases disappear.
Scale in ms. Sample data.
Your app sends the request
- What happens
- Your client talks to the gateway instead of the provider. The request is forwarded unchanged, and your provider key stays in the request; it is never stored.
- What Spanrail records
- The start time and, from the start of the request body, the model name.
- If this is slow
- Not timed: the gateway adds roughly half a millisecond on a local network (see the measurements below).
Change one line
The gateway speaks the same API as your provider, so your client needs one change: where it sends requests. Your provider key stays in your client.
from openai import OpenAI client = OpenAI(api_key="sk-your-provider-key")
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8080/v1",
api_key="sk-your-provider-key",
)To group several calls under a trace you already have, send its id in an X-Spanrail-Trace-Id header. The gateway attaches the call to that trace instead of starting a new one.
curl localhost:8080/v1/chat/completions \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "X-Spanrail-Trace-Id: my-existing-trace" \
-d '{"model":"...","messages":[]}'What you see
Each call shows up as a trace. The span list puts every step on one time axis, and the detail for a generation opens with its time budget, so the slowest phase is the first thing you read.
Sample data. Bars share one time axis: 0 to 612 ms.
Time budget
374 ms at the gatewayScale in ms| Phase | Duration | Share of the call |
|---|---|---|
| DNS lookup | 44 ms | 12% |
| TCP connect | 0.5 ms | 0% |
| TLS handshake | 21 ms | 6% |
| Upstream wait | 201 ms | 54% |
| Response transfer | 107 ms | 29% |
| Total at gateway | 374 ms |
New connection to the provider. Time to first byte 267 ms (includes the setup phases).
The gateway
A small proxy written in Go, using the standard library and one metrics dependency. It forwards your request unchanged and reports the call to Spanrail on the side.
- About half a millisecond
- That is what the gateway adds at the median, and 0.75 to 1.3 ms at the 95th percentile, against a 20 ms upstream. Measured on one Windows machine over loopback, three runs per case. A real network will differ, so measure your own.
- It never makes your call wait
- Reports go through a bounded queue and are sent in the background. If Spanrail is slow and the queue fills, events are dropped and counted. Your calls are not.
- Your key stays yours
- The gateway never forwards or reports your provider key. Request and response bodies are not captured unless you turn that on.
- It flushes on shutdown
- On SIGTERM it stops taking new calls, then sends what is queued before it exits.
What is in it
- Tracing
- Nested spans and generations with errors, cost and usage, in a tree you can search and expand.
- Evaluation
- Datasets, runs and deterministic scorers, with a comparison view to catch regressions.
- Prompts
- Versioned prompts with labels, linked to the generations that used them.
- Analytics
- Trace volume, error rate, cost and latency percentiles over the last 7, 14 or 30 days.
Questions
Do I have to change my code?
Only the base URL of an OpenAI-compatible client. Your provider key stays in the client and passes straight through the gateway.
What happens if Spanrail is down or slow?
Your calls are not affected. The gateway reports in the background and retries failed batches with backoff. If its queue fills up it drops events and counts them.
Does it work with streaming?
Yes. Streamed replies pass through without being buffered. Token usage is recorded only when the provider sends it in the stream; OpenAI-style streams do when the client asks for it with stream_options.include_usage.
Which providers does it understand?
Any OpenAI-compatible API. Streamed token usage from Anthropic's own API and from OpenAI's Responses API is not parsed yet, so those calls are traced without usage.
Where does my data live?
In your own Postgres. Spanrail is self-hosted, and the gateway does not store request or response bodies unless you enable it.
Is it ready for production?
Not yet. It is early software, and the README lists its known limitations. Run it on a trusted network until you have read them.
Run it yourself
Spanrail needs Docker. Postgres and Redis come with it, and your traces stay on your own infrastructure.
It is early software: read the known limitations in the README before exposing it beyond a trusted network.
curl -fsSL https://raw.githubusercontent.com/Veeky-kumar/SpanRail/main/deploy/install.sh | sh
Then send your first call. Run the prebuilt gateway image and point your client at it, or install the Python SDK and send traces directly. The image is published for amd64 and arm64.
docker run -p 127.0.0.1:8080:8080 \ -e UPSTREAM_URL=https://api.openai.com \ -e SPANRAIL_BASE_URL=http://host.docker.internal:3000 \ -e SPANRAIL_PUBLIC_KEY=pk-... -e SPANRAIL_SECRET_KEY=sk-... \ ghcr.io/veeky-kumar/spanrail-gateway:latest
pip install spanrail
from spanrail import Spanrail
sr = Spanrail("pk-...", "sk-...", base_url="http://localhost:3000")
trace = sr.trace("support-chat")
with trace.generation("answer", model="gpt-4o-mini") as gen:
gen.end(output="...", usage={"input": 12, "output": 7})