llmuxv0.1.0

CASOON Open Source

Route every prompt to the cheapest model that can handle it.

llmux is a local, OpenAI-compatible proxy between your tools and your providers. It classifies each request, applies privacy, capability and budget rules, and forwards it to the cheapest viable model, provider and cost tier.

git clone https://github.com/casoon/llmux && cd llmux && npm ci && npm run build && cargo install --path .
cargo install --git https://github.com/casoon/llmux
Apache-2.0v0.1.0 prototypeRustsingle binary
llmux --demo
$ curl -s localhost:3456/v1/chat/completions -H 'content-type: application/json' -d @architecture.json | jq '{model, usage}'
{
  "model": "echo-pro",
  "usage": {
    "completion_tokens": 72,
    "prompt_tokens": 19,
    "total_tokens": 91
  }
}
task types, from simple_text to architecture
5
cost tiers in the model catalog
5
read-only Stats API endpoints
8
Rust tests passing (cargo test, 2026-09-27)
101

What it does

  1. Intent-based, tier-aware selection

    Each task type sets a quality floor (min_tier). Among the models that clear it and fit the context window, llmux estimates the cost of this request and takes the cheapest.

  2. Privacy and capability filters first

    Secrets matching block_cloud_patterns force local-only routing. Requests with tools, JSON schema or images only reach models that declare those capabilities.

  3. Budgets that degrade instead of failing

    Daily and monthly spend is summed from the SQLite log. Above configurable thresholds the tier ceiling drops; a 402 comes only when even the cheapest model would exceed the remaining budget.

  4. Retry, fallback, cache and a log you can query

    Transient errors retry with backoff, exhausted providers fall through to the next candidate, identical requests come from the cache, and every decision lands in SQLite with its policy result.

Embedded dashboard

The binary serves a read-only dashboard at /, fed by the Stats API. Captured from llmux --demo after 22 requests.

llmux Control Room: KPI tiles for requests per minute, cost today, budget used, cache hit rate and p95 latency above a live route feed listing each request's tool, task type, selected model and tier, cost, latency and policy result.

Real output

All examples →
{
  "allowed": 4,
  "cached": 1,
  "degraded": 0,
  "fallback": 0,
  "forced": 1,
  "forced_rejected": 0,
  "local_only": 1,
  "rejected": 0,
  "top_rejection_reasons": []
}

Policy counters from GET /api/stats/policy after the six showcase requests. Every example on this site was captured from llmux --demo with examples/capture.sh.

examples/stats-policy.out

Quickstart

Try the routing without any keys: demo mode answers every request with the built-in echo provider and an in-memory database.

  1. Build and install the binary (see above), or run cargo run -- --demo in a checkout.
  2. Start llmux --demo and open the dashboard at http://localhost:3456/.
  3. Point any OpenAI-compatible client at http://localhost:3456/v1, or send a request with curl.
Read the documentation
demoShell
llmux --demo            # built-in echo provider: no keys, no network
# dashboard: http://localhost:3456/

curl -s localhost:3456/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{"messages":[{"role":"user","content":"explain the architecture trade-offs"}]}'