Why latency is a product problem, not just an engineering one
A feature that takes eight seconds to respond gets abandoned, no matter how accurate the output. We treat latency budgets as a product requirement from day one of any AI engagement, not an optimization pass at the end.
1. Stream everything you can
Token streaming turns a perceived eight-second wait into a perceived one-second wait, even when total generation time is unchanged. Any user-facing generation should stream by default.
2. Cache aggressively at multiple layers
Semantic caching of embeddings, prompt-response caching for repeated queries, and retrieval result caching all compound. We typically see 30-50% of production traffic served from cache in mature systems.
3. Right-size the model to the task
Not every call needs your largest model. Routing simple classification or extraction tasks to smaller, faster models — reserving the frontier model for genuinely hard reasoning — cuts both latency and cost significantly.
4. Parallelize independent steps
Retrieval, tool calls, and guardrail checks are often run sequentially by default. Restructuring pipelines to execute independent steps concurrently frequently removes hundreds of milliseconds with no accuracy tradeoff.
5. Measure p95, not average
Average latency hides the failures that actually erode trust. We instrument every production AI pipeline with p50/p95/p99 latency tracking from day one, and treat p95 regressions as release blockers.
Priya Nandakumar
Chief Technology Officer