CASOON Open Source

Route every prompt to the cheapest model that can handle it.

llmux is a local, OpenAI-compatible proxy between your tools and your providers. It classifies each request, applies privacy, capability and budget rules, and forwards it to the cheapest viable model, provider and cost tier.

sourcecargo --git

git clone https://github.com/casoon/llmux && cd llmux && npm ci && npm run build && cargo install --path .

cargo install --git https://github.com/casoon/llmux

Apache-2.0v0.1.0 prototypeRustsingle binary

llmux --demo

$ curl -s localhost:3456/v1/chat/completions -H 'content-type: application/json' -d @architecture.json | jq '{model, usage}'
{
  "model": "echo-pro",
  "usage": {
    "completion_tokens": 72,
    "prompt_tokens": 19,
    "total_tokens": 91
  }
}

task types, from simple_text to architecture: 5
cost tiers in the model catalog: 5
read-only Stats API endpoints: 8
Rust tests passing (cargo test, 2026-09-27): 101

What it does

Intent-based, tier-aware selection
Each task type sets a quality floor (min_tier). Among the models that clear it and fit the context window, llmux estimates the cost of this request and takes the cheapest.
Privacy and capability filters first
Secrets matching block_cloud_patterns force local-only routing. Requests with tools, JSON schema or images only reach models that declare those capabilities.
Budgets that degrade instead of failing
Daily and monthly spend is summed from the SQLite log. Above configurable thresholds the tier ceiling drops; a 402 comes only when even the cheapest model would exceed the remaining budget.
Retry, fallback, cache and a log you can query
Transient errors retry with backoff, exhausted providers fall through to the next candidate, identical requests come from the cache, and every decision lands in SQLite with its policy result.

Embedded dashboard

The binary serves a read-only dashboard at /, fed by the Stats API. Captured from llmux --demo after 22 requests.

llmux Control Room: KPI tiles for requests per minute, cost today, budget used, cache hit rate and p95 latency above a live route feed listing each request's tool, task type, selected model and tier, cost, latency and policy result.

Real output

All examples →

{
  "allowed": 4,
  "cached": 1,
  "degraded": 0,
  "fallback": 0,
  "forced": 1,
  "forced_rejected": 0,
  "local_only": 1,
  "rejected": 0,
  "top_rejection_reasons": []
}

Policy counters from GET /api/stats/policy after the six showcase requests. Every example on this site was captured from llmux --demo with examples/capture.sh.

examples/stats-policy.out

Quickstart

Try the routing without any keys: demo mode answers every request with the built-in echo provider and an in-memory database.

Build and install the binary (see above), or run cargo run -- --demo in a checkout.
Start llmux --demo and open the dashboard at http://localhost:3456/.
Point any OpenAI-compatible client at http://localhost:3456/v1, or send a request with curl.

Read the documentation

demoShell

llmux --demo            # built-in echo provider: no keys, no network
# dashboard: http://localhost:3456/

curl -s localhost:3456/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{"messages":[{"role":"user","content":"explain the architecture trade-offs"}]}'

Route your first request in demo mode.

Read the quickstart

Route every prompt to the cheapest model that can handle it.

What it does

Intent-based, tier-aware selection

Privacy and capability filters first

Budgets that degrade instead of failing

Retry, fallback, cache and a log you can query

Embedded dashboard

Real output

Quickstart

Route your first request in demo mode.