The LLM as a System Component — Part 3: Token Economics & Model Tiers
maang.io Agentic System Design Series
🎬 Prefer to watch first? This lesson has a video walkthrough on YouTube: Why 88% of an AI Agent's Bill Is Input Tokens. Watch it, then read on for the worked numbers.
Let's pick the consultant on the meter back up. In Part 1 we packed their briefing pack and found out how fast it fills. In Part 2 we timed the call: they read the pack before saying a word, then talk at a steady pace. Now the invoice arrives, and it has a few line items you may not have expected. The consultant bills for every word they read and every word they say, and saying costs more than reading. The firm has partners, associates and interns, each at a different rate. And the firm caps how many words per minute your whole company may use, not just how many calls you can place.
Those three facts are this part. You'll price Priya's refund at Larkspur Home down to the tenth of a cent, scale it to a monthly bill, cut it by more than half with tiers and cascades, price the thinking a reasoning model does out of sight, and spot the launch-day rate limit before it pages you. Then you'll settle the question every LLM bill eventually raises: keep renting the model, or run and train your own? Throughout, a model call is a metered, slow, sometimes-wrong remote dependency, never a free function call.
⚠️ Every price, speed and limit in this part is an illustrative assumption: round figures picked so the arithmetic is easy to follow. Real prices vary and change often, so check your provider's current pricing. What lasts is the method.
Two meters: input tokens and output tokens
A model provider bills you on two meters, both counted in tokens (Part 1's unit, roughly 750 words per 1,000 tokens of English). The input meter counts everything you send: the system prompt, the tool definitions, the conversation so far, every tool result. The output meter counts everything the model generates. The two are priced differently, and output is typically the expensive one.
Here are the illustrative prices we'll use for the rest of the topic, per million tokens (MTok):
| Tier | Input $/MTok | Output $/MTok |
|---|---|---|
| Frontier | 3.00 | 15.00 |
| Mid | 1.00 | 5.00 |
| Small | 0.25 | 1.25 |
(Illustrative assumptions. Check your provider's current pricing.)
Output is priced about 5× input at every tier, and Part 2 already told you why. Input tokens go through prefill: the whole prompt is processed in parallel, so each input token costs the accelerator very little time. Output tokens come out of decode: one at a time, each needing its own full pass through the model, holding expensive hardware the whole time the model talks. The price passes that cost on to you. The exact ratio differs between providers and models, but the direction is durable: reading is cheap per token, speaking is not.
So the instinct is "watch your output length": right for latency, as Part 2 showed, but usually the wrong meter for cost. Take the first call of Priya's refund on the frontier tier. From Part 1, call 1 sends a 2,000-token fixed prefix (system prompt plus nine tool definitions) and Priya's message plus the app's framing (100 tokens), so 2,100 input tokens, and the model replies with a ~100-token tool call:
call 1 = 2,100 × $3/M + 100 × $15/M = $0.0063 + $0.0015 = $0.0078
Even on this first call, input is most of the bill: you send twenty-one times more tokens than you get back. And call 1 is the cheapest call the task will make.
Pricing Priya's refund, call by call
Recall Part 1's statelessness: the consultant remembers nothing, so you read them the entire briefing pack every time. Each step adds about 500 tokens of history, and every later call carries all of it, so the per-call price climbs: $0.0078 for call 1, ≈ $0.0108 for call 3, ≈ $0.0243 for call 10 with its 6,600 input tokens.
From Part 1: 43,500 input and 1,200 output tokens per task, 97% of them input. Now apply the frontier prices:
input: 43,500 × $3/M = $0.1305
output: 1,200 × $15/M = $0.0180
total = $0.1485 ≈ $0.15 per task
Input is 88% of the cost, even though each input token is a fifth of the price of an output token. The cause is the one we keep running into: the whole history is resent on every call. And it gets worse with length: each new step makes every later call bigger, so total input grows roughly with the square of the step count. When someone proposes "just add a few more verification steps", hear the input meter speed up.
💡 The rule of thumb: in a multi-step agent, the input meter is the bill. Output length is your main latency lever (Part 2). Input size × number of calls is your main cost lever.
Here is the whole cost model as one picture. Note the label on the first box: tokens per call is not a constant in an agent.
Cost per successful task
One more line separates a real estimate from a napkin: you pay for every attempt, not every success. As The Agent Loop's Compounding Error & Reliability Math lesson shows, ten steps at 95% each finish correctly only about 60% of the time. If failed attempts aren't automatically retried and simply hand off to a human, the 40% that fail still spent their tokens:
cost per resolved task ≈ $0.1485 ÷ 0.6 ≈ $0.25
That line is illustrative (the true figure depends on where failures happen), but the framing is the point: the unit that matters to the business is cost per resolved task, not cost per attempt. Anything that raises the success rate also lowers this number, which is why reliability work and cost work are often the same work.
Quick check before moving on:
The cost ladder: from one call to the monthly bill
A cent or two per call sounds like nothing, but finance sees a monthly invoice. Climb the ladder out loud, one rung at a time, so nobody is surprised where cents turn into six figures.
Assume Larkspur Home handles 50,000 support tasks a day, and an average customer opens 4 tasks a month. All on the frontier tier:
| Rung | Arithmetic | Frontier cost |
|---|---|---|
| Per call (call 1) | 2,100 × $3/M + 100 × $15/M | $0.0078 |
| Per task | 43,500 input × $3/M + 1,200 output × $15/M | $0.1485 ≈ $0.15 |
| Per user per month | 4 tasks × $0.1485 | ≈ $0.59 |
| Per day | 50,000 tasks × $0.1485 | $7,425 |
| Per month | $7,425 × 30 days | $222,750 ≈ $223K |
(Illustrative prices and volumes. Check your provider's current pricing.)
Each rung answers a different stakeholder. Per task is the engineering number: it moves when you edit a prompt, add a tool or switch a model. Per user is the product number: set $0.59 a month against what a customer is worth and you learn whether the feature can be free, bundled or capped. Per month is the finance number, and $223K for one support agent gets a design reviewed again.
⚠️ Gotcha: the easy mistake is to estimate per call and multiply by volume, forgetting that a task is ten calls whose input grows. Call 1's $0.0078 × 10 calls is $0.078, about half the real $0.1485. Price the task as a whole, using the growing input, before you multiply by anything.
The cheapest lever you'll pull first: don't pay full price for the same prefix
Look back at what you resend. Calls 2 through 10 each carry the same 2,000-token prefix: the identical system prompt and tool definitions. Many providers offer prefix caching, which bills a repeated prefix at a steep discount. With an illustrative 90% discount (cached input billed at 10% of the input price), caching just that prefix on calls 2–10 saves:
9 × 2,000 × $3/M × 0.9 = $0.0486 per task → ≈ $0.10 per task
That's a third off without changing what the agent does, and caching the growing history too pushes it lower. How caching works and what breaks a hit is Context Engineering's Prompt Caching & Prefix Stability lesson; here, just note that it works on the dominant meter.
🔧 Try it: follow the prompt caching docs (or OpenAI's equivalent) to cache a prefix the size of Larkspur's 2,000 tokens (check the page's minimum cacheable length for your model first; on some models it's larger), send two calls, and find the cached input in the second call's usage figures.
The firm's staff: frontier, mid and small tiers
So far we've hired a partner for every call. The firm also has associates and interns: smaller, cheaper, faster, less capable models. Choosing a model tier is the second big lever: instead of trimming tokens, it changes the price of every one.
| Tier | Analogy | Input / output $/MTok | Priya's task costs | Speed (illustrative) | Typically good enough for |
|---|---|---|---|---|---|
| Frontier | Partner | 3.00 / 15.00 | $0.1485 | TTFT ≈ 0.5 s, ≈ 60 tok/s | Multi-step judgment, ambiguous cases, customer-facing writing |
| Mid | Associate | 1.00 / 5.00 | $0.0495 | between the two | Most routine reasoning and drafting |
| Small | Intern | 0.25 / 1.25 | $0.0124 ≈ $0.012 | TTFT ≈ 0.3 s, ≈ 150 tok/s | Classification, extraction, routing, picking the obvious next tool |
(Illustrative assumptions. Check your provider's current pricing and speeds.)
The task arithmetic is the same 43,500 input and 1,200 output tokens at different prices:
mid: 43,500 × $1/M + 1,200 × $5/M = $0.0435 + $0.0060 = $0.0495
small: 43,500 × $0.25/M + 1,200 × $1.25/M = $0.010875 + $0.0015 = $0.0124
The small tier runs Priya's task about 12× cheaper than frontier, and decodes faster. Why not use it everywhere? Because capability, price and speed come as a bundle: the small model is likelier to pick the wrong tool, misread the policy, or word the reply to Priya badly. And in a ten-step chain, the compounding math from The Agent Loop topic punishes every point of per-step accuracy you give up.
The durable principle is tier per step, not per product. Look at Priya's ten calls again. "Look up order 4417" and "update ticket T-5120" are mechanical: the next tool is obvious from the context. "How do we explain this to Priya?" needs judgment — and that's the final reply. Nothing forces every call onto the same model. Part 2 showed the latency side: nine intermediate steps on the small tier, the final reply on frontier, and the critical path drops from 26.8 s to about 16.0 s. The same split moves the cost meter too:
calls 1–9 on small: 36,900 input × $0.25/M + 900 output × $1.25/M ≈ $0.0092 + $0.0011 = $0.0104
call 10 on frontier: 6,600 × $3/M + 300 × $15/M = $0.0198 + $0.0045 = $0.0243
tiered total ≈ $0.035 per task
That's ≈ 77% cheaper than all-frontier, about $1,730 a day instead of $7,425, and the customer-facing reply still comes from the partner.
💡 Tier names drift. Today's frontier capability becomes next year's mid-tier price. Don't hard-code "the big model" into your architecture. Hard-code the decision: which steps need which level of capability, as measured by your evals. Then re-run that decision whenever the model menu changes.
Thinking tokens: the meter you can't read
Part 2 introduced reasoning models, which produce working-out tokens before they answer. On the invoice, thinking tokens are output tokens, billed at the output rate whether or not you ever see them. Many APIs show you only a summary of the reasoning, or none of it. The usage figures in the response still report every thinking token, and you pay for every one.
Price it on Priya's task at the frontier tier's $15 per million output tokens:
| Thinking setting | Thinking tokens per task | Added per task | Task total | Per day (50,000 tasks) |
|---|---|---|---|---|
| Off | 0 | — | $0.1485 | $7,425 |
| Low on all 10 calls (500 each) | 5,000 | $0.075 | $0.2235 (+51%) | $11,175 |
| Medium on all 10 calls (2,000 each) | 20,000 | $0.30 | $0.4485 (≈ 3×) | $22,425 |
| Medium on one hard step only | 2,000 | $0.03 | $0.1785 (+20%) | $8,925 |
(Illustrative assumptions. Check your provider's current pricing and how it reports reasoning usage.)
💡 Subtle point: "the input meter is the bill" holds only while the model is quiet. With medium thinking everywhere, the model writes 20,000 tokens nobody reads, about 17× the 1,200 visible ones, and output jumps from 12% of the cost to about 71% ($0.318 of $0.4485).
Three more things to know before you flip the switch:
- Earlier thinking may be resent as input. Some providers drop previous calls' thinking from the context; some keep it inside a tool-use loop. If it's kept, it also feeds the growing input, the meter that was already the bill. Check your provider's behaviour, then confirm it in the usage figures.
- It lands on output rate limits. At Larkspur's peak of 104 tasks a minute, medium thinking everywhere needs 104 × 21,200 ≈ 2.2M output tokens per minute, about 18× the 125K it needs without. Output limits are often the smaller ones.
- It's a third dial, beside tier and cascade. Thinking costs whatever the tier's output rate is, so 2,000 thinking tokens cost $0.03 on frontier but $0.01 on mid. A mid-tier model that thinks on a hard step can match a frontier model that doesn't, for less money but more seconds. Which combination wins is a per-step question for your evals.
💡 Same rule as tiering: think per step, not per product. Turn thinking on where a step is genuinely hard and single-shot, leave it off for mechanical tool picks, cap the budget on every call, and log thinking tokens as their own line. Then Agent Observability's cost attribution can tell a thoughtful model from a chatty one.
Quick check before moving on:
Cascading: ask the intern first, escalate when unsure
Per-step tiering decides the model before the call. A cascade decides after: give the task to the cheap model first, check the result, and send it to the expensive model only if the check fails. The intern drafts; only hard cases reach the partner's desk.
Let's price it. Every task tries the small tier first. Say 30% get escalated (flagged as low-confidence or failing a check) and re-run on frontier. The escalated ones pay for both runs:
cascade = $0.0124 + 0.30 × $0.1485 = $0.0124 + $0.0446 = $0.057 per task
That's ≈ 62% cheaper than all-frontier: about $2,850 a day instead of $7,425. A big number hiding a big assumption.
Notice it's dearer than per-step tiering's $0.035. So why would you ever pick a cascade? Tiering needs you to know in advance which steps are easy; a cascade is for when you can't tell until you've seen the answer.
The whole saving depends on the escalation check being good. A cascade can fail in two directions, and they hurt in different ways:
- False accepts (the check is too lenient). The small model gets something wrong, the check passes it, and the wrong answer ships. The bill looks great; Priya got the wrong refund amount, or a confident reply citing a policy that doesn't exist. Note that call 7 is a write:
create_refundmoves money. Many teams pin writes and other high-blast-radius steps to a stronger model or a human approval no matter what the cascade says. (Approval and idempotency for writes belong to the Tool Use topic.) - Over-escalation (the check is too strict). Every escalation pays twice, in money and in latency, because the user waits for the small attempt and the frontier attempt. As the escalation rate climbs, the 62% saving shrinks, and every escalated task ends up slower than if you had called frontier directly.
What makes a good check? Signals that cost far less than the call they're guarding: the output fails schema validation, a cheap verifier disagrees, a business rule fires ("refunds over a threshold always escalate"), or a required tool result is missing. A model's own "I'm 90% sure" is a weak signal by itself, because models are often confidently wrong.
⚠️ Gotcha: the 30% escalation rate and "the accepted 70% are fine" are exactly the two numbers you must measure, not assume. Each extra 10 points of escalation adds about $0.015 to every task, and each false accept ships a wrong answer at small-tier prices. Prove both by running the cascade over a labelled dataset against the frontier baseline.
Measuring them is the Evals topic's job. How routers and cascades are actually built (confidence signals, learned routers, multi-stage chains) is the 201 Model Routing & Cascades sub-topic.
Quick check before moving on:
Rate limits: the firm caps words per minute, not calls
Now the line item that bites on launch day. Providers set rate limits on your account, and the one people forget is the one that matters. Let's use illustrative account limits of 4,000 requests per minute (RPM) and 2,000,000 input tokens per minute (TPM).
Engineers who have run APIs think in requests. Check Larkspur Home's peak against both:
average load = 50,000 tasks ÷ 86,400 s ≈ 0.58 tasks/s
peak (3× average) ≈ 1.74 tasks/s ≈ 104 tasks/min
Requests first:
peak RPM = 104 tasks × 10 calls = ~1,040 requests/min, well under the 4,000 limit
Looks fine. Now tokens:
peak input TPM = 104 tasks × 43,500 tokens = ≈ 4.5M tokens/min, 2.3× the 2M limit
TPM binds, not RPM, because of the same resent history that dominated the bill: each agent request is a large bundle of tokens that grows every step. The ceiling this account can sustain is:
max sustainable = 2,000,000 ÷ 43,500 ≈ 46 tasks/min
That's under half the peak. Output adds pressure too: 104 × 1,200 ≈ 125K tokens/min, often metered against a separate limit.
Here is that peak minute as it happens: watch the token gauge reach its limit while the request gauge has barely moved.
What does going over look like? The provider starts returning 429 Too Many Requests. In an agent that's nastier than in a plain API, because a 429 on call 6 stalls a task that has already paid for calls 1 through 5, and naive retries from many stalled tasks pile onto the same limit. You'll often get more out of admitting work at the task level (only start a task when there's token budget to finish it) than out of throttling individual calls.
The levers, from the cheapest to apply to the most involved:
- Shrink tokens per task. Prefix caching, shorter tool results, trimmed history. Every token you stop resending raises the 46-tasks/min ceiling. (Whether cached tokens count fully against TPM varies by provider, so check.)
- Spread across tiers. Limits are often set per model, so moving the intermediate steps to the small tier also moves them onto a different quota.
- Request a higher limit. Often available, often slow; never your launch-day plan.
- Queue and admission control. Accept the task, tell Priya it's in progress, and drain the backlog at a sustainable rate. Fine for tickets, not for live chat.
🔧 Try it: open a provider's rate-limits page (Anthropic, OpenAI) and find the request and token limits Larkspur would hit, and whether input and output tokens are counted separately or together. Then divide the token limit by Priya's task (43,500 input tokens, or 44,700 if one limit counts both) to get the tasks a minute it could sustain.
💡 Capacity planning for LLM features is capacity planning in tokens. Size peak demand in tokens per minute, compare it with every limit the provider enforces, and know which one binds before you launch.
The general algorithms (token buckets, sliding windows) are covered in the Rate Limiting topic in the System Design course. Designing a limiter that meters tokens instead of requests, including reserving budget for output you haven't seen yet, is the 301 problem Token-Aware LLM Rate Limiter and the 201 LLM Gateway sub-topic.
Rent or own: APIs, open-weight models and fine-tuning
The rate limits, the prices and the model's next version all belong to someone else. Sooner or later someone in the room asks: "Shouldn't we run our own model? Or train one on our tickets?" Those are two separate decisions, and an interviewer will expect a number behind each.
Renting tokens or owning GPUs
A hosted API charges per token. You pay for exactly what you use, and the provider absorbs idle hardware, upgrades and capacity planning. Self-hosting an open-weight model (one whose weights you can download and run yourself) charges per GPU-hour. You pay for the hardware whether it's busy or not, and in return you control the model version, the data path and the capacity.
Price Larkspur both ways. Its traffic is 50,000 tasks × 44,700 tokens ≈ 2.2 billion tokens a day: about 26,000 tokens per second on average and 78,000 at the 3× peak. Assume an illustrative GPU server that costs $20 an hour and sustains 10,000 tokens per second of this input-heavy traffic, running a mid-sized open-weight model at acceptable latency:
| Option | Arithmetic | Per day |
|---|---|---|
| API, all small tier | 50,000 × $0.0124 | $620 |
| API, tiered per step (this part's split) | 50,000 × $0.035 | $1,730 |
| API, all mid tier | 50,000 × $0.0495 | $2,475 |
| API, all frontier | 50,000 × $0.1485 | $7,425 |
| Self-hosted, provisioned for peak | 8 servers × 24 h × $20 | $3,840, plus the team |
| Self-hosted, perfectly autoscaled | ≈ 2.6 servers on average × 24 h × $20 | ≈ $1,250, plus the team |
(Illustrative assumptions. Real throughput depends on the model, the hardware, the engine and your latency target.)
Provisioned for peak, the servers sit about two-thirds idle, because support traffic averages a third of its peak. Perfect autoscaling would close most of that gap, but GPU capacity is slow to add and slow to warm up, so nobody gets it. Then add the people: even two engineers at an illustrative fully loaded $250K a year is about $1,400 a day, plus an on-call rotation for a system that used to be someone else's pager.
At Larkspur's scale and shape, renting wins on price. The API's per-token rate includes someone else's high utilization, pooled across thousands of customers. Self-hosting starts to pay with high, steady volume that keeps GPUs busy around the clock, not with peaky support traffic. Teams that self-host usually do it for control, not cost:
- Data that can't leave your network. Though check first: many providers offer zero-retention terms and regional endpoints that may already meet the requirement.
- Capacity you own. No shared tokens-per-minute limit and no noisy neighbours. You get your own queue instead.
- Version pinning. The model changes only when you change it.
- Custom or narrow models at volume. Fine-tuned weights, classifiers and embedding models that run millions of times a day.
What you take on is the whole of the 201 Inference Engines and GPU Fleet Scheduling & Autoscaling sub-topics: batching, KV cache, cold starts, upgrades, and evals for every new open model you adopt.
Prompt, retrieve or fine-tune
The other question, "should we train the model on our data?", has a durable order of escalation:
| The problem | Reach for | Why |
|---|---|---|
| The model doesn't behave the way you want | Better instructions and examples in the prompt | Cheapest, instant to change, versioned with your code |
| The model doesn't know your facts (policies, orders, documents), or they change | Retrieval: fetch the facts into the context | Fresh, citable and permission-aware. Fine-tuning bakes facts into weights, where they go stale |
| The prompt works but is long, slow or costly, or a behaviour (a format, a tone, a narrow classification) won't stick from instructions | Fine-tuning | Moves behaviour from the prompt into the weights: shorter prompts, a smaller tier |
The most common winning fine-tune is distillation: record the frontier model's good trajectories, fine-tune a small model on them, and move steps that needed frontier onto a tier that's about 12× cheaper, if your evals agree. The costs are real, though: a labelled dataset, an eval set to prove it worked, retraining whenever the base model is retired, and sometimes a higher per-token price or dedicated hosting for the custom weights.
💡 Subtle point: fine-tuning changes behaviour, not knowledge. It won't reliably teach Larkspur's agent next Tuesday's refund policy, and retrieval will. Why retrieval beats fine-tuning for facts is the opening section of Retrieval-Augmented Generation's Embeddings, Chunking & Indexing lesson.
Quick check before moving on:
Good practices and don'ts
✅ Good practices
- Report cost per resolved task next to cost per attempt. Priya's $0.1485 attempt is ≈ $0.25 per resolved task at a 60% success rate, and that's the number that shows reliability work paying for itself.
- Cache the stable prefix before anything cleverer. The same 2,000-token prefix on calls 2–10 saves $0.0486 a task, about a third, without changing what the agent does.
- Choose a tier for each step, and let your evals choose it. Nine mechanical steps on small and the reply on frontier is ≈ $0.035 a task, 77% cheaper, and it cuts the critical path from 26.8 s to about 16.0 s.
- Read thinking tokens from the usage figures before you trust a cost estimate. Many APIs show only a summary of the reasoning, or none, yet bill every thinking token at the output rate: medium thinking on all ten calls adds 20,000 unseen tokens and $0.30 to Priya's task.
- Move the mechanical steps onto another model's quota. Limits are often set per model, so putting Priya's nine intermediate calls on the small tier takes them off the frontier model's 2M tokens a minute as well as cutting the bill.
- Admit work per task, not per call. A 429 on call 6 stalls a task that already paid for calls 1–5, and naive retries from many stalled tasks pile onto the same limit. Start a task only when there's token budget to finish it.
🚫 Don'ts
- Don't use the model's own confidence as the escalation check. "I'm 90% sure" is weak because models are often confidently wrong. Use signals that cost far less than the call: schema validation, a cheap verifier, a business rule, a missing tool result.
- Don't let a cascade's cheap path decide money-moving writes. Call 7,
create_refund, moves real money: pin it to a stronger model or a human approval, whatever the check says. - Don't self-host to save money on peaky traffic. Provisioned for peak, Larkspur's servers cost ≈ $3,840 a day, sit two-thirds idle and need ≈ $1,400 a day of engineers, against $1,730 for the tiered API.
- Don't make a quota increase your launch-day plan. Higher limits are often available and often slow. Shrink tokens per task first: every token you stop resending raises the 46-tasks-a-minute ceiling.
Tools and references
Open these to see this part's ideas for yourself (links checked November 2026):
- Prompt caching (official docs): how Anthropic bills a repeated prompt prefix; OpenAI documents its own in its prompt caching guide. Cache a prefix the size of Larkspur's 2,000 tokens (check the page's minimum cacheable length for your model first; on some models it's larger), send two calls, and find the cached input in the second call's usage figures.
- Rate limits (official docs): Anthropic's request and token limits; compare OpenAI's rate limits guide. Find which limits are counted separately for input and output tokens, and whether cached tokens count against the input limit, the question this part leaves to your provider.
- Reasoning models (official docs): OpenAI's guide to reasoning effort and reasoning tokens; Anthropic covers the same in extended thinking. Find where the usage figures report thinking tokens and how to cap the budget, then redo this part's arithmetic: 2,000 thinking tokens a call × 10 calls at the output rate.
- FrugalGPT (paper): a research paper that studies LLM cascades, among other strategies, for cutting cost while keeping quality. Look at how it decides whether to accept the cheap model's answer, and compare that with this part's escalation checks.
- RouteLLM (open source): a framework for serving and evaluating routers that send each query to a strong or a weak model before the call. Set its routing decision beside this part's two options: tiering decided in advance, and a cascade decided after seeing the answer.
- vLLM (open source): an engine for serving open-weight models yourself. If you have a GPU, serve a mid-sized open-weight model and measure its tokens per second against this part's assumed 10,000 per server.
Key takeaways
- Two meters, unequal prices. Output tokens cost more than input (5× in our illustrative prices) because decode is sequential and holds the hardware per token. But in a multi-step agent the input meter dominates: Priya's refund is 97% input tokens and 88% input cost, because the stateless model gets the whole history resent on every call, and total input grows roughly with the square of the step count.
- Price the task, then climb the ladder. $0.0078 per call → $0.1485 per task → $0.59 per user per month → $7,425 per day → ≈ $223K per month (all illustrative). Frame it as cost per resolved task: at a 60% success rate that's ≈ $0.25.
- Tier per step, not per product. The small tier runs the same task about 12× cheaper and faster, but it's less capable. Match each step to the cheapest tier your evals say is good enough, keep judgment-heavy and customer-facing steps on frontier, and cut the shared prefix with caching (≈ $0.10/task).
- Thinking tokens are billed output you never see. Medium reasoning on all ten calls roughly triples Priya's task to ≈ $0.45, makes output ≈ 71% of the cost and needs ≈ 18× the output tokens per minute; on one hard step it adds just $0.03. Think per step, cap the budget and log thinking as its own line.
- Cascades save money only if escalation is honest. Small first, with 30% escalated, is ≈ $0.057/task (≈ 62% cheaper). That holds only when the escalation check is measured with evals. False accepts ship wrong answers cheaply, and over-escalation pays twice in money and latency.
- Tokens per minute is the limit you hit first. At peak, Larkspur's agent needs ≈ 4.5M input TPM against a 2M limit while using only ~1,040 of 4,000 RPM. Plan capacity in tokens, admit work per task, and shrink tokens per task before you beg for quota.
- Rent tokens until volume or control says otherwise; fine-tune for behaviour, never for facts. Self-hosting Larkspur costs ≈ $3,840 a day provisioned for peak, plus the team, against $1,730 for the tiered API, because peaky traffic leaves owned GPUs idle. Self-host for data control, owned capacity, version pinning or high steady volume. Put changing facts in retrieval, and use fine-tuning (often distillation into a small model) to move behaviour and cost.
Test yourself on this part's key ideas:
You can now size an LLM feature, time it and price it. In Part 4 we take all of it into the interview room, where you'll defend those numbers against an interviewer who wants to know what happens when the provider goes down, the model is wrong, and the bill doubles.
