Reliability when a model provider goes down

Every model provider has incidents. Rate limits tighten under load, latency spikes, endpoints return errors. If your assistant calls one provider with no alternative path, its availability is bounded by that provider's, and you have no lever.

The four failure shapes

They need different handling, and treating them identically is the usual mistake:

• Hard errors. A 500 or a connection failure. Fast to detect, and retryable.
• Rate limiting. A 429. Retrying immediately makes it worse. Needs backoff.
• Latency degradation. Requests succeed but slowly. The hardest to detect because nothing is technically failing.
• Quality degradation. Responses return normally and are worse. Effectively undetectable in real time.

Most retry logic handles the first and mishandles the second. Backoff with jitter is not optional on a 429, because synchronised retries from many clients are what turn a brief limit into a sustained one.

Fallback ordering

A sensible degradation ladder, in order:

1. Retry the same provider with backoff. Handles transient errors.
2. Route to a secondary provider. Handles sustained provider issues.
3. Route to a smaller or cheaper model on any available provider. Degraded but functional.
4. Return retrieved content without generation. The visitor sees the relevant passage rather than a synthesised answer.
5. Honest failure with a handoff offer.

Step four is underused and it is genuinely valuable. Retrieval usually still works when generation does not, because it depends on different infrastructure. Showing the relevant passage from your own documentation is a real answer, just an unpolished one.

Timeouts have to be set explicitly

Default HTTP timeouts are typically far longer than a chat interaction can tolerate. A visitor is not waiting sixty seconds. Set the timeout to what the experience can bear, which is usually well under ten, and treat exceeding it as a failure to be handled rather than a request to keep waiting for.

Circuit breaking

If a provider is failing, continuing to send it traffic wastes time on every request and delays every visitor. A circuit breaker that trips after a threshold of failures and routes elsewhere for a cooldown period converts a slow degraded experience into a fast degraded one, which is better.

The parameters matter less than having one at all. Trip after a handful of consecutive failures, cool down for a minute, then probe with a single request before restoring.

What to tell the visitor

During degradation, say something true and specific. "We are having trouble reaching our AI service. You can leave a message and we will follow up, or try again in a few minutes." That is better than a generic error and much better than a spinner that never resolves.

Do not claim the answer is being generated when no request is in flight. Visitors sometimes wait for several minutes on the strength of an indicator that is lying to them.

What to measure

Error rate and latency per provider, tracked separately, with the slow tail rather than the average. And fallback activation rate, because if you are silently falling back constantly, your primary configuration is wrong and nobody would otherwise notice.