dorang (도랑) — the small channel beside a stream that splits, holds, and rejoins the flow.
dorang is a high-performance LLM gateway written in Go. It sits between your applications and your model providers, and it is designed to run the same way on a laptop as it does on a fleet — notebook to enterprise, one binary, no required dependencies.
한국어 문서: README.ko.md
Most gateways make you choose between capable and fast. dorang refuses the trade: every feature below is designed so the request hot path stays allocation-free and the cost of adding protocols is constant, not linear.
Real provider plans do not have one concurrency number. One account may allow 3 in-flight requests across all models, while a coding plan on another provider allows 7 in flight per model, so two models means 14. dorang models this directly:
capacity:
credential_groups:
account-a: { max_concurrency: 3 } # per account, any model
account-b: { max_concurrency: 3 }
models:
- { provider: coding-plan, model: model-x, max_concurrency: 7 } # per (provider, model)
- { provider: coding-plan, model: model-y, max_concurrency: 7 }The model axis is keyed by provider and upstream model, not by credential —
AxisModel in internal/capacity. So two models on one coding plan is 14 in
flight however many keys that plan has, while two models on one Ollama account is
still 3, because that ceiling sits on the credential axis. That difference between
the two axes is the whole point of the example, and it is measured rather than
asserted: testing/providers/ drives both shapes through one gateway at one
instant.
Five axes — route, provider group, model, credential group, key — are reserved atomically, all-or-nothing, so a waiter never holds one slot while queuing for another. Deadlock is structurally impossible.
- Rolling 5-hour / daily / weekly / monthly windows over tokens, requests, or cost.
- Polls provider-reported quota endpoints where they exist, so the number reflects all usage of that key — not just what went through the gateway.
- On exhaustion a credential steps aside and traffic moves to the next key automatically.
- Budgets reserve before spending, so concurrent requests cannot overshoot the cap.
Routing to the backend that still holds your KV cache requires knowing the conversation prefix matched in order. dorang chains the hashes:
h₀ = H(group) hᵢ = H(hᵢ₋₁ ‖ cᵢ ‖ len(cᵢ))
A match at depth i proves chunks c₁..cᵢ are byte-identical and in that order.
Reordering, insertion, or deletion all produce a different value. The length is mixed in
after the chunk, not before, because a trailing partial segment's length is not known
until it ends — that ordering is what lets the chain be computed in a single streaming
pass over the body instead of requiring it buffered.
Per-key usage, cost, latency, TTFT, token counts, and fallback paths are recorded for every request. Measurement runs off the hot path through per-CPU ring buffers, and the design target is that turning it on costs less than 5%.
OpenAI chat/completions/embeddings/responses/audio/images/moderations, Anthropic
messages, rerank, decision models (/v1/systemone), batch, files — plus a generic passthrough engine so provider-native
routes are opened by configuration rather than by writing another adapter.
| Notebook | Small team | Enterprise | |
|---|---|---|---|
| Storage | SQLite (embedded) | PostgreSQL | PostgreSQL + partitions |
| Coordination | none | none | Redis or PostgreSQL leases |
| Nodes | 1 | 1–2 | N, leader-elected maintenance |
| Telemetry | in-process, sampled | full ledger | ledger + rollups + retention |
| Dependencies | none | 1 | 2 |
The same binary and the same config file work across the whole range. You scale by changing configuration, not by changing deployment model.
Under construction, and it builds and passes. 33 packages, roughly 148,000 lines of Go,
with 1,390 tests, 60 benchmarks and 7 fuzz targets green under -race. It ships as a
static binary and as a container image.
Parts of it are not finished. The most common defect found in this codebase, by a wide margin, is a control that exists and is not reached: a limit that validates, stores, and is never consulted. Design §17.1 records the instances and the rule they produced.
- Design — architecture, configuration schema, algorithms, milestones
- Configuration — the config schema and every field, with examples
- Compatibility — wire contracts pinned as golden tests, and the error taxonomy
- Operations — running, scaling and administering a deployment
- Migration — moving from the incumbent gateway
- Surfaces — which API surface each provider actually serves, measured
Apache License 2.0. Distributions must preserve the dorang attribution in NOTICE, using the standard Apache 2.0 §4(d) NOTICE mechanism.