Latency budgets for a chat widget

When a visitor sends a message, several things happen in sequence, and the total wait is the sum. Knowing the breakdown tells you which part is worth attention, because they are not equally expensive to improve.

Where the time goes

• Embedding the query: usually tens of milliseconds. Not your problem.
• Vector search: single digit to low tens of milliseconds for a small index. Not your problem.
• Model time to first token: typically the largest single component, and the one that varies most by model.
• Token generation: proportional to answer length.
• Network and rendering: small, but it is where layout jank shows up.

The pattern is clear. Retrieval is fast and model time dominates. Optimising your vector index when the model accounts for most of the wall clock is misplaced effort.

Streaming changes the perceived number, not the real one

Streaming tokens as they generate does not reduce total time. It reduces time to first visible output, which is what the visitor actually experiences as responsiveness.

The difference is substantial in perception. A four second response that starts appearing after 700 milliseconds feels responsive. The same four seconds spent on a blank screen feels broken. If you build only one latency improvement, build streaming.

Answer length is a latency decision

Generation time scales with output length, so a verbose system prompt that produces long answers is also a slow system. On a website assistant, shorter answers are usually better anyway, so this is a case where the quality goal and the speed goal point the same direction.

Constraining answer length in the prompt is a legitimate latency optimisation and it costs nothing.

What to do while waiting

The interface during the wait matters more than most teams treat it. Three rules:

• Show something immediately on send, so the message is visibly received.
• Use a calm indeterminate indicator. Do not show a progress bar for something whose duration is unknown, because a bar that stalls reads worse than no bar.
• Never block the rest of the page. The visitor should be able to keep reading while the answer generates.

Timeouts and honest failure

Set a timeout and decide what happens at it. A request that hangs indefinitely is the worst outcome, because the visitor does not know whether to wait or leave.

At the timeout, say plainly that the response is taking longer than expected and offer the alternative, whether that is retrying or reaching a person. An honest failure preserves more trust than an indefinite spinner.

Measure at the edge, not at the server

Server side timing measures your infrastructure. It does not include network time to the visitor, which on mobile can dominate everything else. If you only measure server side you will conclude the system is fast while a meaningful share of your traffic experiences it as slow.

Measure from the client, and look at the slow tail rather than the average. The average visitor is not the one who leaves.