The LLM as a System Component — Part 2: Latency — TTFT, Throughput & Streaming
maang.io Agentic System Design Series
🎬 Prefer to watch first? This lesson has a video walkthrough on YouTube: LLM Latency: TTFT, Streaming & Why Agents Are Slow. Watch it, then read on for the worked numbers.
Back to our consultant on the meter. In Part 1 we worked out what goes into the briefing pack you read down the phone line: the system prompt, the tool definitions and the whole history so far, all resent on every call because the consultant remembers nothing. Now listen to the call from the other end. When you finish reading the pack, there's a silence while they take it in. Then they start talking, at a steady pace, one word after another, until they're done. Reading the pack before speaking is prefill, and it sets your time to first token. Speaking at a steady pace is decode, and it sets everything after. The two phases have different physics, different causes and different fixes.
This part puts seconds on each half, shows why streaming changes how a wait feels without shortening it, adds the silence a reasoning model spends thinking before it speaks, and then times an agent that chains ten calls.
The one-line idea: a call's latency is time to first token plus output tokens divided by decode speed, and an agent pays that sum once per sequential call. The number of calls on the critical path and the length of each output matter far more than how fast any single call looks on a dashboard.
⚠️ Every speed and latency here is an illustrative, round figure chosen to keep the arithmetic easy. Real numbers vary by model, provider, region, prompt length and time of day. Check your provider's current figures, and measure your own traffic.
The two halves of every call: prefill and decode
Prefill. The model reads every input token (system prompt, tools, history, the user's message) and nothing comes back yet. The wait here is time to first token (TTFT), and it has three parts: the network round trip, time queueing behind other customers' requests, and the prefill compute itself.
Decode. The model produces output one token at a time. Each new token depends on all the tokens before it, so it can't skip ahead. The speed here is tokens per second (equivalently, time per output token), roughly steady for the whole answer.
Why are the halves so different? Because of what the hardware does in each:
| Prefill (reading the pack) | Decode (speaking) | |
|---|---|---|
| What it processes | All input tokens at once | One output token per step |
| Parallelism | Highly parallel: every input token is already known | Sequential: token n needs token n − 1 first |
| What limits it | Raw compute | Memory bandwidth: each step is a full forward pass over the model's weights for a single token |
| Grows with | Input length and provider load | Output length |
| Illustrative frontier tier | TTFT ≈ 0.5 s p50 (≈ 1.5 s p95) for a ~2,000-token prompt | ≈ 60 tokens/s |
| Illustrative small tier | TTFT ≈ 0.3 s | ≈ 150 tokens/s |
Prefill is fast per token because the accelerator chews through thousands of known tokens in parallel. Decode is slow per token because it's inherently a loop: produce a token, append it, run the whole model again. The consultant can skim the pack quickly, but can only speak one word at a time.
That gives you the formula for the rest of this topic:
call latency ≈ TTFT + (output tokens ÷ decode tokens per second)
Two caveats keep it honest. TTFT is not a constant: it rises with input length (more pack to read) and with provider load (a longer queue), and a very long prompt can take seconds before the first token. And the p50/p95 split is real: the typical call waits about half a second, but one in twenty waits around three times that, and nothing in your code caused it.
💡 How engines make prefill and decode fast (the KV cache, batching many users onto one accelerator, scheduling) is the 201 Inference Engines sub-topic. For design you need only the durable principle: input is cheap in time, output is expensive in time, and the queue in front of both is shared with strangers.
Here's one call, end to end:
Worked example: 2,000 tokens in, 500 tokens out
📱 This section in 61 seconds: LLM output is the slow half: 2,000 tokens read in half a second, 500 written in over eight, a YouTube Short.
Put the formula to work on one typical request: a 2,000-token prompt, a 500-token answer, a frontier-tier model decoding at 60 tokens/s.
latency ≈ 0.5 s + 500 ÷ 60 = 0.5 + 8.3 ≈ 8.8 seconds
Sit with that split. Of the 8.8 seconds, 0.5 is prefill and 8.3 is decode. The 2,000 input tokens, four times as many as the output, cost half a second. The 500 output tokens cost over eight.
Output length is your main latency knob
This is the most actionable fact in the part. Decode runs at a fixed pace, so every output token costs about 1/60 of a second on our frontier tier, every time. When a call is slow, look first at how much it says, not how much it reads:
- Ask for less. "Answer in two sentences" or "return only the tool call" moves latency more than almost any infrastructure work.
- Cap it. Set a maximum output length on every call. It bounds latency and, as Part 3 will show, cost. An uncapped call that rambles is a latency incident waiting to happen.
- Don't make the model write what code can write. If a template can produce the boilerplate, let the model produce only the variable part.
Size timeouts for decode, not for REST
⚠️ A classic production bug: a team wraps the model call in the same HTTP client they use for every internal service, with the same short timeout. A perfectly healthy 500-token answer, which legitimately takes about 9 seconds, gets killed mid-sentence. The retry starts from scratch, pays prefill and decode again, and gets killed again. The system can never answer long questions, and every attempt is still on the meter.
Size timeouts to the shape of the call instead:
- A first-token timeout, anchored to TTFT with headroom for the p95, catches a stuck queue or dead connection quickly.
- An idle timeout between tokens catches a stream that stalls halfway.
- An overall deadline scaled to the maximum output you allowed: TTFT plus max tokens ÷ decode speed, plus headroom.
A timed-out call is not a free retry, either. You pay for the whole call again. And if a timeout makes you re-run a whole step or task after a write tool already ran (like the refund after Priya's call 7), that tool needs idempotency to be safe; the Tool Use topic covers how.
Quick check before moving on:
Streaming: making the wait feel shorter
Without streaming, the user in our 8.8-second example stares at a spinner and then the whole answer lands. With streaming, the first words appear at 0.5 seconds and the rest flow in as they're produced. Same time, same tokens, same cost; a completely different experience.
Watch both on the same clock, at real speed. Keep your eye on when the first words show up, and on where both lanes finish:
It works so well because the model talks faster than people read. A typical reader manages about 250 words per minute, roughly 4.2 words per second, which at ~0.75 words per token is about 5.6 tokens per second. Decode at 60 tokens/s is about 10× faster than a person reads. Once streaming starts, the text stays ahead of the reader's eyes, so the only wait they feel is TTFT.
💡 Streaming doesn't make the work faster. It moves the moment the user first sees progress from "end of decode" to "end of prefill". For anything a human reads, that's usually the biggest perceived-latency win available.
How the tokens travel
| Server-Sent Events (SSE) | WebSockets | |
|---|---|---|
| Direction | One-way, server to client | Bidirectional |
| Runs over | A plain HTTP response held open, sending events | An upgraded, persistent connection |
| Fits | The common case: stream tokens to a browser or app | The client must talk during the response: interrupting, barging in on a voice agent |
| Operational feel | Works through most HTTP infrastructure; reconnects are simple, but a dropped token stream usually means re-running (and re-paying for) the call | More moving parts: connection state, heartbeats, reconnects |
Reach for WebSockets when the client genuinely needs to speak mid-answer. The transport mechanics are the HTTP & HTTPS and WebSockets topics in the System Design course. One gotcha worth naming: anything in the path that buffers responses or enforces idle timeouts (a proxy, an API gateway, a load balancer) can silently collect the stream and deliver it all at the end, undoing the benefit. Test streaming through your real network path, not just against the provider.
What streaming makes harder
- Structured output arrives as partial JSON. Mid-stream,
{"tool": "create_refund", "arguments": {"order_id": 44isn't valid JSON. You can't act on a tool call until it's complete, so streaming helps human-facing text far more than machine-facing output. (Validating structured output is the Tool Schemas & Structured Output lesson in Tool Use.) - Moderating partial text. Check only the finished answer and the user has already read the problem. Hold text back until a check passes and you give back the latency you streamed to win. Checking chunks as they arrive is the usual middle ground; the trade-offs belong to Guardrails & Safety.
- Cancellation. If the user navigates away, close the stream so the provider stops decoding. Tokens produced for nobody are still on the meter.
Quick check before moving on:
Reasoning models: thinking before the first visible token
Some models can be told to think before they answer. A reasoning model (also called a thinking model; many APIs expose it as an extended thinking or reasoning effort setting) produces a stretch of working-out tokens, step-by-step reasoning, before it writes the answer you see. On genuinely hard problems that working-out measurably raises accuracy: judgments with many constraints, maths, multi-step plans, tricky code. It's like asking the consultant to "talk it through before you give me your answer", except that you usually hear only a summary of the talking-through, or nothing at all, while the meter runs.
The physics doesn't change. Thinking tokens are output tokens, produced by the same one-at-a-time decode. So the formula gains a term:
call latency ≈ TTFT + (thinking tokens + answer tokens) ÷ decode tokens per second
and so does the moment the user first sees something:
first visible token ≈ TTFT + thinking tokens ÷ decode tokens per second
The thinking comes before the answer, so streaming can't hide it. Turn reasoning on and the wait the user feels is TTFT plus all of the thinking.
Most APIs let you set how much a call may think, as an effort level or a token budget. Illustrative levels at our frontier tier's 60 tokens/s:
| Effort | Thinking tokens (illustrative) | Added before the first visible token |
|---|---|---|
| Off | 0 | 0 s |
| Low | 500 | ≈ 8.3 s |
| Medium | 2,000 | ≈ 33 s |
| High | 8,000 | ≈ 133 s, over two minutes |
Real reasoning models often decode faster than 60 tokens/s, and the amount of thinking varies from call to call with how hard the model judges the problem, even at a fixed setting. The shape holds anyway: every thinking token is a decode token on the critical path.
Now switch it on for Larkspur. Low effort on all ten of Priya's calls adds about 10 × 8.3 ≈ 83 s: the run goes from 26.8 s to about 110 s, and Priya's first visible word from 21.8 s to about 105 s. And it buys almost nothing, because nine of those calls are mechanical tool picks whose answer is obvious from the context.
So reasoning is a per-step decision, like the choice of model tier. Spend it on a step that's genuinely hard and single-shot (planning a messy multi-order ticket, judging a disputed damage claim) and leave it off everywhere else. Sometimes the trade is excellent. A few seconds of thinking on one hard step can beat adding agent steps, each of which pays its own TTFT and grows the history every later call re-reads. Whether it pays for your step is an evals question, not a hunch.
Four practical consequences:
- Size timeouts for the thinking. A first-token timeout tuned for a 0.5 s TTFT kills every reasoning call. Some APIs stream the thinking, or a summary of it, which keeps a stall detector fed; otherwise the line is silent until the answer starts. Set the overall deadline from TTFT + (thinking budget + max answer) ÷ decode speed.
- Fill the silence for users. Show a reasoning summary if the provider offers one, or the step-by-step progress messages from the levers below.
- Cap it, and check the stop reason. On many APIs the thinking budget counts toward the maximum output tokens. A cap that's too tight can be spent entirely on thinking, leaving a cut-off answer or none.
- Expect a fatter tail. Thinking length varies call to call, so reasoning steps widen exactly the latency tail the next sections are about.
Part 3 puts thinking tokens on the meter. The short version: they're billed as output.
Quick check before moving on:
Why five sequential calls blow a 3-second budget
Agents don't make one call. They make a chain, and each call needs the previous result before it can start. That chain is the critical path: steps that can't overlap, whose latencies simply add.
Say you have a 3-second budget for a feature that needs five sequential model calls, each with a short 50-token output, and a 0.2-second tool call between each pair. Per call:
0.5 s TTFT + 50 ÷ 60 ≈ 0.5 + 0.83 = 1.33 s
| Component | Arithmetic | Time |
|---|---|---|
| Five model calls | 5 × 1.33 s | 6.7 s |
| Four tool calls between them | 4 × 0.2 s | 0.8 s |
| Total | ≈ 7.5 s, about 2.5× the budget |
Now the sobering part. Imagine the five calls produced zero output tokens. You'd still pay 5 × 0.5 s = 2.5 s in TTFT alone. 2.5 seconds of the 3-second budget are gone before the model has said a word or a single tool has run.
The durable lesson: TTFT is a fixed tax per call, and sequential calls pay it again and again. No prompt tuning fits a five-call chain into 3 seconds. You have to change the shape: fewer calls, faster models, or calls that run side by side.
💡 This is why interactive voice is the hardest latency problem in the course. A spoken conversation feels broken after a pause far shorter than these budgets, so every sequential call is a luxury. The Real-Time Voice Agent design problem is this section taken to its extreme.
The running example: Priya's 26.8 seconds
Now time the agent we've been following. Priya writes in about their cracked lamp from order #4417, and the Larkspur Home support agent resolves it in 10 sequential model calls: nine that each choose a tool, then a final streamed reply. All frontier tier, illustrative figures:
| Segment | Arithmetic | Time | Visible to Priya? |
|---|---|---|---|
| Nine tool-choosing calls (~100 output tokens each) | 9 × (0.5 + 100 ÷ 60 = 2.17 s) | 19.5 s | No |
| Nine tool executions | 9 × 0.2 s | 1.8 s | No |
| Final reply (~300 tokens) | 0.5 + 300 ÷ 60 | 5.5 s | Only after its TTFT |
| End to end | ≈ 26.8 s | ||
| Priya's first visible token | 19.5 + 1.8 + 0.5 | ≈ 21.8 s |
Follow the yellow token through the run. Every step it passes before the first visible token is time Priya spends looking at nothing:
Look where streaming helps: only the final call. The nine intermediate calls produce tool calls for your code, not text for Priya, so there's nothing to show. Streaming makes the last 5.5 seconds pleasant and does nothing about the silence before it.
⚠️ The ledger is generous, too: it holds TTFT flat at 0.5 s. But as Part 1 showed, the input grows from 2,100 tokens on call 1 to 6,600 on call 10 because the whole history is resent, and TTFT rises with input length. The real number drifts up from 26.8 s, not down.
The tail: why ten calls make slow runs routine
Per-call dashboards mislead you about agents. Say each call has a 5% chance of landing in its own slow tail, where TTFT is around 1.5 s rather than 0.5 s because the provider is busy. For one call, that's rare. For Priya's ten-call run:
chance at least one call lands in the tail = 1 − 0.95¹⁰ ≈ 40%
Four runs in ten hit at least one slow call, so the slow tail stops being rare and becomes routine. In our ledger a slow call costs about a second, and about one run in twelve hits two or more, so the run's p95 sits near 28.8 s even before TTFT creeps up as prompts grow. Retries, 429 backoffs and slow tools make real tails much fatter, and your per-call dashboard shows none of it.
It's the same compounding that turns 95% per-step reliability into about 60% end to end, the subject of the Compounding Error & Reliability Math lesson in The Agent Loop. What follows:
- Measure end to end. Track per-task latency percentiles, not just per-call, and trace every call in a task under one ID so you can see which step blew up. That's the Agent Observability topic.
- Budget for the tail, not the median. A design that fits only when every call hits p50 doesn't fit.
- Fewer calls shrink the tail too. Every call removed is one fewer roll of the dice.
Quick check before moving on:
The levers: getting back under budget
26.8 seconds is a long wait for a refund. Here's the toolkit, roughly in order.
1. A smaller, faster tier for the intermediate steps. The nine tool-choosing calls are short, structured decisions. Route them to the small tier (TTFT ≈ 0.3 s, ≈ 150 tokens/s) and keep the frontier model for the reply Priya reads:
9 × (0.3 + 100 ÷ 150) = 9 × 0.97 ≈ 8.7 s, plus tools 1.8 s, plus the frontier final call 5.5 s → ≈ 16.0 s, down from 26.8 s
That holds only if the small model picks tools as well as the big one did, a question you answer with the Evals topic, not with hope. (Part 3 shows the same move is also a big cost win.)
2. Fewer calls. Each call removed takes its TTFT tax and its tail risk with it. Could a deterministic rule check eligibility without asking the model? The best call is the one never made.
3. Parallel independent work. Priya's calls 1–4 look up the order, the shipment, the policy and the customer's history, and none depends on the others. Many model APIs let one step request several tool calls at once (see Tool Schemas & Structured Output in Tool Use), and your code runs them side by side: one round trip instead of four. Broader fan-out is the Multi-Agent Patterns topic.
4. Shorter outputs. Decode is the expensive half. A tighter reply and a terser tool-call format shave every step.
5. Async plus progress. If the work genuinely needs 16 seconds, don't leave the user staring at a blank box. Show each step as it happens ("Found your order", "Checking the refund policy"), or acknowledge immediately and notify when done. It doesn't cut real latency, but it changes how the wait feels.
6. A lean prompt. Shorter input means faster prefill. Prefix caching can cut prefill time as well as cost; its mechanics are the Prompt Caching & Prefix Stability lesson in Context Engineering.
Key takeaways
📱 The whole lesson in 61 seconds: LLM latency in four facts, a YouTube Short to share or revisit.
- Every call has two halves. Prefill reads all input in parallel and sets TTFT (network + queueing + prefill, rising with input length and load); decode speaks one token at a time. Latency ≈ TTFT + output tokens ÷ tokens per second, so output length is your main latency knob: 8.3 of an 8.8-second call is decode.
- Streaming changes the felt wait, not the real one. First words at 0.5 s instead of 8.8 s, and 60 tok/s outpaces a ~5.6 tok/s reader by about 10×. SSE for one-way token streams, WebSockets when the client must talk back; expect partial JSON, partial-text moderation and buffering proxies to complicate it.
- Reasoning models think in decode time, before the first visible token. Thinking tokens are output tokens, so latency ≈ TTFT + (thinking + answer) ÷ tokens per second, and streaming can't hide them. Low effort on all ten of Priya's calls adds ≈ 83 s; spend thinking per step where evals show it pays, and size timeouts and output caps for it.
- Sequential calls add, and TTFT taxes each one. Five calls take about 7.5 s against a 3-second budget, and TTFT alone eats 2.5 s. Priya's ten-call agent takes ≈ 26.8 s, first visible token at ≈ 21.8 s, because streaming only helps the final call.
- Agents amplify the tail. At a 5% slow-tail chance per call, ≈ 40% of ten-call runs hit at least one. Measure per-task percentiles end to end, and size timeouts for decode length, not REST defaults.
- Change the shape, not just the prompt. Small tier for hidden steps (26.8 s → 16.0 s, if evals agree), fewer calls, parallel independent tools, shorter outputs, and async progress when the work is genuinely long.
Test yourself on this part's key ideas:
Every one of those tokens and seconds is also on the meter. Next up, Part 3 prices them: what one call, one task, one user and a day of Priya-style conversations cost, why output is billed higher than input, and how model tiers, cascades and tokens-per-minute limits shape the bill.
