Most teams don’t notice they have a cost problem until the monthly bill arrives and someone asks why it tripled.
But AI API cost runaway is never sudden. There are always warning signals 2-3 weeks in advance — if you know what to look for.
This article covers the 7 early warning signals we see most often across 100+ AI projects. Track these, and you will catch cost issues before they become budget crises.
1. Average Tokens per Request Growing 20%+ per Week
This is the #1 predictor of cost explosions — and almost no one tracks it.
Your average tokens per API call should be relatively stable after launch. If it is growing 20%+ week over week, something is changing in your prompt engineering:
- Someone is appending more context to “improve quality”
- Chat history is not being truncated properly
- System prompts got longer after a “prompt tuning” session
The fix is simple: Track average tokens per request as a core metric. Put it on a dashboard. Alert when it grows more than 10% in a week.
2. Retry Rate Above 15%
Every retry is money spent twice (or three times, or five times) for the same result.
We have seen production systems with 40% retry rates — meaning almost half the API spending was pure waste.
Common retry causes:
- Rate limit throttling
- Timeouts on long outputs
- Format errors when the LLM returns malformed JSON
- Temporary 5xx errors
Worse, exponential backoff means each failed attempt costs more time and may trigger more retries downstream.
Set an alert when your retry rate exceeds 15%. Anything above that means your error handling or prompt format needs work.
3. Top 10% of Requests Consume 70% of Tokens
Token usage follows a power law distribution in almost every AI system — a small number of requests eat most of the budget.
If your P90/P99 token count is 10x higher than your median, you have “super request” outliers:
- Long document summaries
- Multi-step agent reasoning loops
- Batch processing jobs that should use batch endpoints
The fix is to route these high-token requests to cheaper models. GPT-4o for simple chat is fine, but a 100k token summarization? Use GPT-4o mini or Claude Haiku.
4. Cache Hit Rate Below 30%
Prompt caching (available on both Claude and OpenAI) cuts input token costs by 50-90% for repeated context.
If your cache hit rate is below 30%, you are leaving free money on the table.
Common reasons for low cache hit rates:
- Dynamic timestamps or random data in prompts
- User input placed at the beginning of prompts instead of the end
- No prompt versioning (small changes break cache)
Moving dynamic content to the end of the prompt can often double your cache hit rate in five minutes.
5. Model Upgraded But Budget Thresholds Did Not
This is a classic and completely avoidable mistake.
The team switches from Claude Haiku to Sonnet for better quality. The per-token price triples. But no one updates the budget alert thresholds.
Three weeks later: “Wait, why is our bill $15k instead of $5k?”
Rule: Every time you change your default model, update your monitoring thresholds at the same time. It takes 60 seconds and prevents six-figure surprises.
6. “Temporary” Code That Never Gets Fixed
The most expensive words in AI development:
“This is just temporary — we will optimize it later.”
Temporary code becomes permanent code.
- A loop that calls the LLM on every item instead of batching
- A naive approach that was “good enough for MVP”
- No caching because “we will add it before launch”
The problem is that AI costs scale with usage. A loop that costs $1 per day in testing costs $30 per month in production. With 1000 users, it costs $3000 per month.
Add a comment with a future date to every “temporary” piece of code, then schedule a review. If the date passes and it is still there, pay down the technical debt.
7. Only Monthly Bill Reports, No Real-Time Alerts
Waiting for the end-of-month cloud bill is how companies wake up to $50k overages.
You need at least three alerts:
- Daily spend > threshold (e.g., $500/day)
- Hourly spend spikes > 2x average
- Per-user token limit exceeded
And you need a kill switch: a single flag that caps total API spending at the account level. Most API providers support this. Do not go to production without it.
Quick Self-Check
Count how many of these apply to your system right now:
- No dashboard for average tokens per request
- No monitoring for retry rates
- Never measured P90 vs median token usage
- Cache hit rate unknown or below 30%
- Budget thresholds not updated after model changes
- “Temporary” prompt code still running in production
- No real-time cost alerts, only monthly bill reports
0-1: Good shape, stay vigilant 2-3: You have hidden cost problems waiting to surface 4+: You are already overpaying by 2x-5x. Fix this week.
Summary
AI API cost runaway is not an accident. It is a predictable outcome of not tracking the right operational metrics.
The 7 signals to monitor weekly:
- Average tokens per request growth
- Retry rate above 15%
- Extreme outlier token usage (P99 / P50 ratio)
- Low prompt caching hit rates
- Unchanged budget thresholds after model upgrades
- Permanent “temporary” code
- No real-time spending alerts
Track these, and you will never be surprised by an AI bill again.