Most AI API budgets fail for a simple reason: the final number looks precise, but the assumptions underneath are incomplete. Teams forecast visible user requests, then forget output growth, retries, evals, batch jobs, agent steps, cache misses, and tail usage.
If you already built an AI API cost forecast or a budget spreadsheet, use this guide as a diagnostic checklist before the next bill arrives.
Forecasting Mistakes Are Usually Assumption Mistakes
A forecast is not wrong because the spreadsheet formula is ugly. It is usually wrong because one input was too optimistic.
| Forecast input | Common mistake | What happens |
|---|---|---|
| Request volume | Use launch week traffic only | Month-two spend surprises the team |
| Token size | Use average tokens only | Long-tail requests dominate cost |
| Cache hit rate | Assume savings before logs prove it | Forecast is too low |
| Retry rate | Ignore failed calls | Billable calls exceed visible tasks |
| Agent steps | Treat one task as one model call | Cost per successful task is understated |
| Evals | Forget offline tests | Engineering traffic appears as surprise spend |
| Batch jobs | Hide background jobs inside user traffic | Product growth and automation cost blur together |
Good forecasting is not about predicting perfectly. It is about making these assumptions visible early.
Mistake 1: Using Average Request Cost Only
Average request cost hides expensive tail cases.
A support chatbot may look cheap on average, but a few requests can include long conversation history, retrieved context, large tool schemas, or verbose answers. The average does not show how much P95 or P99 usage contributes.
Track:
| Metric | Why it matters |
|---|---|
| average input tokens | Baseline context size |
| P95 input tokens | Long prompts, RAG chunks, tool schemas |
| average output tokens | Normal answer length |
| P95 output tokens | Long reports, verbose JSON, repeated answers |
| cost per successful task | More useful than cost per raw call |
If P95 cost is much higher than average cost, the fix is usually not a cheaper model first. Start with input caps, output limits, chunk controls, and alerts.
Mistake 2: Ignoring Output Token Growth
Teams often estimate input tokens carefully but underestimate output tokens. This is risky because output can grow quietly when prompts change.
Common causes:
- a short answer prompt becomes a “detailed report” prompt;
- JSON output includes too many fields;
- summaries include full quotes instead of compact bullets;
- the model is asked to include reasoning-like explanations;
- users request regenerated or expanded answers.
Add output scenarios to the forecast:
| Scenario | Output assumption |
|---|---|
| Baseline | Current average output length |
| Growth | 25-50% longer answers after launch |
| Stress | P95 output length or maximum allowed length |
Then compare actual output tokens weekly.
Mistake 3: Assuming Cache Savings Before Logs Prove It
Prompt caching can reduce cost, but only when the stable prefix is actually stable.
Do not forecast an optimistic cache hit rate just because the system uses caching. Check whether the prompt includes dynamic timestamps, user IDs, randomized context order, changing tool descriptions, or per-request policy text.
Use three scenarios:
| Cache scenario | When to use |
|---|---|
| 0% hit rate | Conservative pre-launch forecast |
| Low hit rate | Early production before logs stabilize |
| Proven hit rate | Only after observing real cache metrics |
If you need to design the assumptions, connect this with prompt caching budget planning.
Mistake 4: Forgetting Retries and Fallback
User-visible tasks are not the same as billable calls.
A single successful task may include:
- first model call;
- schema correction retry;
- timeout retry;
- fallback to a more expensive model;
- final formatting call;
- logging or evaluation call.
Forecast retries separately:
billable calls = visible tasks × (1 + retry rate + fallback rate)This is not perfect, but it is better than assuming every task succeeds once.
Mistake 5: Mixing Evals and Batch Jobs With User Traffic
Evaluation and batch workloads should not be hidden inside user traffic.
| Workload | Why it should be separate |
|---|---|
| Prompt evals | Can increase when quality work starts |
| Regression tests | Run even without user growth |
| Nightly summaries | Often tied to data volume, not sessions |
| Content classification | Can rerun after rule changes |
| Log replay | Can spike during debugging |
If these are mixed into user traffic, the team may think product demand is growing when automation cost is actually growing.
Mistake 6: Forecasting Agents as One Request
Agent workflows should be forecast per successful task, not per request.
Track:
| Agent metric | Why it matters |
|---|---|
| steps per task | Model turns multiply cost |
| tool calls per task | Tools often cause more model observations |
| repeated tool calls | Detects loops and wasted context |
| failure recovery calls | Adds hidden spend |
| human confirmation rate | Shows high-risk tasks that should stop |
Use:
agent task cost = average step cost × steps per successful task + tool-related model calls + retry costFor deeper planning, use agent tool call cost planning.
Mistake 7: Not Comparing Forecast to Actuals Weekly
A forecast that is never compared to actuals becomes a guess.
Each week, compare:
| Assumption | Actual metric | Action |
|---|---|---|
| request volume | request_count | Update traffic scenario |
| average input tokens | avg_input_tokens | Update context assumptions |
| P95 input tokens | p95_input_tokens | Add input controls |
| output length | avg / P95 output_tokens | Adjust answer format |
| retry rate | retry_count share | Fix errors or schema |
| cache hit rate | cache_hit_tokens share | Stabilize cache prefix |
| agent steps | steps_per_task | Add max steps or loop detection |
| eval calls | eval_calls | Put evals in a separate budget |
Do not only adjust the final budget number. Adjust the broken assumption.
Forecast Review Checklist
Before approving a forecast, ask:
- Does it include P95 token scenarios?
- Does it separate input and output tokens?
- Does it include retries and fallback?
- Does it separate evals from user traffic?
- Does it separate batch jobs from product usage?
- Does it forecast agent workflows per successful task?
- Does cache savings come from logs, not hope?
- Does every feature have an owner?
- Is there an alert threshold before budget is exhausted?
- Is there a weekly actual-vs-forecast review?
If the answer is no, the forecast is probably too optimistic.
FAQ
How much safety margin should an AI API forecast include?
It depends on launch risk and traffic volatility. Instead of one fixed margin, build baseline, growth, and stress scenarios. The stress scenario should cover long-tail tokens, lower cache hit rate, more retries, and agent loops.
Should I forecast by provider or by feature?
Forecast by feature first. Provider totals matter for finance, but product teams need to know which feature is driving spend.
What if I do not have token logs yet?
Start with measured samples from typical requests and add conservative P95 estimates. Then make token logging a priority before scaling.
Is a monthly review enough?
For stable products, maybe. For new features, model changes, agent workflows, and launches, weekly review is safer until usage stabilizes.
Final Take
AI API forecasting fails when teams treat visible requests as the whole cost. Add token tails, output growth, cache uncertainty, retries, fallback, evals, batch jobs, and agent steps. Then compare actuals every week. The forecast will still be imperfect, but it will be useful enough to prevent the biggest surprises.