Where the milliseconds go: a latency budget for RAG
A RAG request feels slow long before it is slow. Knowing which of the five stages is responsible is the difference between a targeted fix and three weeks of replacing the wrong component.
What happened
Users report that the assistant is slow. The team profiles the model, finds generation is the bottleneck, and spends a sprint optimising it. The perceived latency barely moves, because the real problem was a sequential chain: retrieval waited on query rewriting, reranking waited on retrieval, and none of it overlapped.
Why it matters for deployment teams
Latency is a budget, and like any budget the first requirement is knowing where it goes. A typical RAG request has five stages — query understanding, retrieval, reranking, packing, generation — and in a healthy system they are roughly 20 to 60 milliseconds, 20 to 60, 40 to 150, negligible, and 300 to 900 respectively. Two numbers matter more than the total: time to first token, which is what users actually feel, and the p99, which is what they remember.
What to do about it
Instrument each stage separately before optimising anything; most teams are wrong about which stage dominates. Then look for work happening sequentially that could overlap — query rewriting and embedding can run in parallel with metadata lookups. Reduce the candidate set sent to the reranker before optimising the reranker itself. Cache aggressively: retrieval and reranking are deterministic for a given corpus version, and support questions repeat far more than teams expect.
The cheap win
Stream. Time to first token is a product decision as much as a technical one, and streaming a partial answer at 400 milliseconds feels materially faster than a complete answer at 1,400 milliseconds. This is usually the highest perceived-performance change available and it costs an afternoon.