How routing works

A createLCR model wraps an ordered, cheapest-first list of providers per logical model. It serves from the first healthy one, switches to the next on a retryable failure, and periodically snaps back to the cheapest. Here's each part.

Cheapest-first ordering

Each logical model maps to a list of provider entries. List order is priority order — the router tries them top to bottom.

models: {
  smart: [
    { model: tokenmart("gemini-2.5-flash"), label: "tokenmart", cost: { input: 0.3, output: 1.2 } },
    { model: openrouter("google/gemini-2.5-flash"), label: "openrouter", cost: { input: 0.3, output: 2.5 } },
  ],
}

Set autoSort: true and the router sorts each list cheapest-first by cost for you, so you can list providers in any order. Without autoSort, you control priority by hand — useful when you'd rather rank by reliability than price.

Skip the hand-typed prices (autoPrice)

For a vendor's own API the cost numbers are just the public list price. Set autoPrice: true and the router fills any entry that has no explicit cost from a bundled price table (MODEL_PRICES) — first-party rates for the native makers (OpenAI, Anthropic, Google, xAI, Mistral, plus the open-weights labs DeepSeek, Qwen, Kimi, MiniMax, GLM), keyed by the bare model id you already pass to the provider. A flat-rate reseller just adds discount:

createLCR({
  autoPrice: true,
  autoSort: true,
  models: {
    "claude-sonnet": [
      { model: anthropic("claude-sonnet-4-6"), label: "anthropic" },           // priced from the table
      { model: kunavo("claude-sonnet-4-6"), label: "kunavo", discount: 0.2 },   // 20% off the list price
    ],
  },
});

It's off by default, an explicit cost always wins, and unknown models stay unpriced — so turning it on never silently re-prices a route you priced yourself. Look a price up directly with getModelPrice("claude-sonnet-4-6"). The table carries native makers only; price open-weights hosts (DeepInfra) and breadth aggregators (OpenRouter) explicitly.

Health-based fallback (streaming-safe)

The router serves from the first healthy provider. On a retryable error (rate limit, overload, 5xx, connection reset, timeout) it switches to the next provider in the list — even mid-stream, before any tokens have been committed — so a provider dropping out doesn't fail the request.

A non-retryable error (a 400-class client mistake) is not masked — it surfaces immediately, because retrying it elsewhere would just fail again.

Automatic recovery

After a failover the router is "parked" on a more expensive provider. Left alone, a brief outage would cost you the pricey fallback forever. So every resetIntervalMs (default 60s) it re-probes the cheapest provider and snaps back when it's healthy again — under load too, not only on the next idle call.

createLCR({ models, autoSort: true, resetIntervalMs: 30_000 }); // re-probe every 30s

Circuit breaker (cooldown)

The recovery timer above re-probes the cheapest provider on a schedule. That's the right move for a brief outage — but for a provider that's persistently down, the timer keeps re-probing it, so every window pays one wasted failed attempt before falling over. This is exactly where most fallback libraries stop: they fail over every time, but never stop hitting the dead endpoint.

Turn on cooldown and the router does what a phone network does with a bad trunk — benches it. A provider that fails maxFailures times within windowMs is skipped entirely for cooldownMs, instead of being tried on every request; a single success clears it.

createLCR({
  models,
  autoSort: true,
  cooldown: true, // defaults: 3 failures / 60s → 60s benched
  // or tune: cooldown: { maxFailures: 5, windowMs: 30_000, cooldownMs: 120_000 }
});

It can only ever make routing safer: the breaker only reorders each request's attempt list (benched providers move to the back), so if every provider is cooling, a request still tries them all rather than failing outright. It's off by default — unset, no provider is ever skipped and routing is unchanged.

Caching

Two different "caches", both off by default, both pure config flags — no service to run.

cache — skip the call entirely. An exact-match request replays the stored response and calls no provider at all: zero latency, costUsd: 0. This is the layer Vercel AI Gateway doesn't offer; ai-lcr does it in-process.

createLCR({ models, cache: true }); // in-memory store, zero dependencies

cache: true is process-local memory — real on a long-running server, but it does not survive across serverless requests (instances don't share memory). For cross-request hits there, bring your own shared store: cache: myStore (any { get, set }, e.g. Upstash Redis / Vercel KV). ai-lcr runs no service of its own. Use cache: { store, ttlMs } to set a lifetime. A hit settles a CallRecord with cacheHit: true and the avoided cost on cacheHitSavingUsd; empty results are never cached. Note caching makes identical requests return identical responses — ideal for temperature: 0, a behavior change for sampled calls.

promptCache — pay less for the call. The model still runs, but the static prompt head (your system prompt) bills at the provider's cache-read rate (~0.1× input) on repeats. Inserts an Anthropic cache_control marker on the last system message for you; OpenAI / Gemini / DeepSeek cache the prefix automatically and ignore the marker, so it's safe on a mixed chain.

createLCR({ models, promptCache: true }); // 5-min window; { ttl: "1h" } for longer

It steps aside if you set cacheControl yourself. Savings surface via cachedInputTokens / cachedSavingUsd on the CallRecord.

Which errors mean what

Every failed attempt is tagged with an ErrorKind, surfaced via onError and on each RouteAttempt. Use it to tell self-healing failures apart from problems you should be paged on:

auth and billing are the dangerous ones: a misconfigured key makes a provider look "unhealthy," silently shifting all traffic onto the pricey fallback while everything still works — exactly the thing to catch early.

Next

• Cost tracking — measure what routing actually saved.
• Providers — wire each provider into a list.