Rate limits, abuse and cost control

A website assistant is a public interface that triggers paid API calls. That combination means the cost is controlled by whoever visits, which is everyone, including automated traffic that has no interest in your product.

This is not a hypothetical risk. It is the predictable consequence of the architecture, and it should be handled at design time.

Layered limits

One limit is not enough because the abuse patterns differ. Four layers, each catching something the others miss:

• Per session. Caps a single conversation. Catches accidental loops and one determined person.
• Per IP per window. Catches scripted abuse from a single origin.
• Global per hour. The backstop. Catches distributed abuse that slips the first two.
• Per account monthly. The billing boundary, which is what the credit model in most products expresses.

The global limit is the one most often omitted and the only one that guarantees a bounded bill. Without it, a distributed source defeats the per IP limit and the cost is unbounded until someone notices.

What happens at the limit

This is a product decision that gets made by default. The options: stop and say so, degrade to a cheaper model, queue, or continue and bill overage.

For a free tier, stopping with a clear message is the honest behaviour. Silent degradation makes the product look bad without explaining why. Whatever you choose, say it on the pricing page, because "what happens when I hit the limit" is one of the most reliably asked pricing questions.

Input length is a cost lever

Cost scales with tokens, and a visitor can paste a very large amount of text into a chat input. Cap input length client side and validate server side, because client side alone is not a control.

Also cap the number of retrieved chunks passed as context. More context is not monotonically better for answer quality and it is monotonically worse for cost and latency. There is a point past which additional chunks add noise, and it is usually lower than people set it.

Caching the repeated questions

On a marketing site, question distribution has a heavy head. The same handful of questions arrive constantly. Caching answers to normalised repeat questions cuts both cost and latency meaningfully.

Two cautions. Normalise carefully, because two questions that look similar can differ in a way that changes the answer. And expire the cache when sources change, or you will serve last month's pricing from cache after publishing an update.

Alerting on spend

Set an alert at a fraction of your acceptable monthly spend, not at the limit. An alert that fires when you have already exceeded the budget is a report, not a control.

Alert on rate of change as well as absolute value. A tenfold increase in hourly requests is worth knowing about immediately, even when the monthly total is still comfortable, because by the time the monthly total is uncomfortable the incident has been running for days.