Why multi model routing exists

The simplest possible architecture picks one model and sends everything to it. It is easy to reason about and it is what most implementations start with. It is also, past a certain volume, an unnecessarily expensive way to answer "what are your opening hours".

Routing means choosing per query. Creobot is designed for multi model routing so teams can route conversations by cost, quality and task type.

The three axes

Cost

The spread between a small model and a frontier model is large, often an order of magnitude or more per token. On a marketing site the majority of questions are factual lookups against retrieved content, which is a task small models handle well. Sending those to a frontier model is spending capability on a problem that does not need it.

Latency

Smaller models respond faster. In a chat interface this is felt directly. A visitor waiting eight seconds for an answer that took two seconds to retrieve has a worse experience than the same answer delivered in three, even if the slower answer is marginally better written.

Task fit

Different tasks have genuinely different difficulty. Extracting a fact from a supplied passage is easy. Comparing two plans against a stated need, or reasoning about whether a described situation falls under a policy, is not. The second category benefits from a stronger model in a way the first does not.

How routing decisions get made

Three approaches, in increasing order of complexity:

**Rule based.** Classify by pattern. Short factual questions go to the small model, comparison and reasoning questions to the large one. Crude, transparent, and captures most of the available saving. Start here.

**Confidence based.** Try the small model first. If retrieval confidence is low or the model's answer is hedged, escalate to the larger one. Costs an extra call on escalated queries but routes on actual difficulty rather than surface features.

**Learned.** Train a classifier on which queries needed the larger model. Better in principle, requires labelled data you probably do not have yet, and is not worth building before the rule based version has told you the shape of your traffic.

The failure mode to design for

Routing introduces a new class of problem: inconsistency. The same question asked twice can be answered by different models and produce noticeably different responses. Visitors notice this, and it reads as unreliability.

Two mitigations. Route deterministically on question features so the same question routes the same way. And constrain output format tightly enough that model differences show up as phrasing rather than as structure or substance.

Fallback is the other reason to build this

Routing infrastructure gives you provider failover almost for free. If your primary provider returns errors or times out, the routing layer can send the request elsewhere rather than failing the conversation.

This is worth having independently of the cost argument. Single provider dependency means your assistant's availability is capped by theirs, and provider incidents happen. Design the routing layer so a fallback is a configuration change rather than a code change.

What not to claim

Routing does not make answers better in general. It makes the cost and latency profile better while holding quality roughly constant, and it adds resilience. Those are real benefits and they are not the same as improved accuracy. Accuracy comes from retrieval.